For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Use this matrix to maintain model and provider audit evidence for NemoClaw agent behavior.
Use it to determine whether a supported model works as an agent model, not only whether it can answer a one-shot chat prompt.
Do not mark a row as completed without committed evidence or a stable CI link.
Rows seeded from source inventory start as not-yet-run until a maintainer imports or records evidence.
Result States
Every audit row must use one of these states.
State
Use when
pass
The row completes required scenarios without model-specific changes.
pass-with-affordance
The row completes required scenarios with a documented model or provider affordance.
degraded
The row is usable but has documented limits, retries, latency risk, or partial surface coverage.
blocked
The row cannot complete required scenarios and needs a linked follow-up issue or PR.
unsupported
The model, provider, or surface is intentionally unsupported.
not-yet-run
The row is in scope but has no completed evidence yet.
Required Row Schema
Use these fields for every completed row.
If a field is not applicable, write n/a and explain why in the evidence notes.
Field
Required content
Model ID
Exact model identifier used by onboarding or runtime config.
Provider path
Provider class and route, such as NVIDIA Endpoints, OpenAI, Anthropic, Gemini, Local Ollama, Local vLLM, or another compatible endpoint.
Agent surface
Exact agent path, such as OpenClaw primary agent, OpenClaw CLI prompt path, OpenClaw browser or gateway path, OpenClaw sub-agent delegation, Hermes sandbox API, Deep Agents interactive dcode, Deep Agents headless dcode -n, or auxiliary model path.
NemoClaw commit SHA
Full commit SHA for the repo state used during validation.
Runtime versions
OpenShell, OpenClaw, Hermes, Deep Agents Code, provider server, and local serving versions when available.
Endpoint/API path selected
Concrete API path, base URL class, and provider key selected by NemoClaw.
Workflow used
Exact command sequence or CI workflow used to run the scenario.
State
One result state from this page.
Evidence
Trajectory file path, session log path, request dump path, or CI artifact link.
Observed tool-call count
Count and names of structured tool calls observed in the scenario.
Final-response behavior
Whether the assistant produced a final response after tool results, stopped empty, stopped reasoning-only, or emitted raw tool text.
Multi-turn behavior
Whether turn 2 used turn 1 tool results without re-running unrelated tools.
Latency and timeout notes
Validation time, first token or first event time when available, total duration, retries, and timeout budget used.
Required affordance
Model-specific setup, provider-class transport behavior, request mutation, API path forcing, streaming requirement, or none.
Follow-up
Linked issue, PR, or registry decision when remediation or setup work is needed.
Required Scenario Coverage
Completed rows should state which required scenarios were exercised.
Rows can remain degraded, blocked, or not-yet-run when a scenario cannot be exercised yet.
Scenario
Required checks
Baseline chat
Deterministic response works, provider validation is actionable, and credentials do not leak into sandbox-visible files, logs, or prompts.
Shell tool loop
Separate structured hostname, date, and uptime tool calls are emitted, persisted, correlated with tool results, and followed by a final assistant response.
Multi-turn continuation
Turn 2 uses a tool result from turn 1 and does not ask the user to continue after a complete tool result.
Sub-agent delegation
The primary agent emits a structured sessions_spawn request, the sub-agent receives the intended task and workspace, and the primary agent consumes the result.
The row records validation duration, first event timing when available, retry behavior, timeout budget, streaming requirement, request mutation requirement, API path forcing, and cold-start differences.
Hermes Path
Validate the Hermes path with the following checks.
Start Hermes with the selected provider and model.
Confirm that it returns the expected OpenAI-compatible response shape.
Confirm that core tools remain direct.
Exercise a deferred tool through the native structured tool_search -> tool_describe -> tool_call path.
Keep Hermes tools.tool_search.enabled: on evidence separate from OpenClaw tools.toolSearch.mode: tools evidence.
Deep Agents Path
Validate the Deep Agents paths independently with the following checks.
Confirm that dcode status reports the managed route.
Confirm that interactive dcode completes a terminal task while preserving approval prompts.
Confirm that headless dcode -n completes a bounded non-shell task while preserving the managed Chat Completions route.
When testing optional thread-opt-in, record the configured host mode separately from the active TUI thread state.
Prove explicit activation and thread-boundary reset behavior.
Rerun the policy and credential boundary checks.
Keep default interactive, opted-in interactive, and headless evidence separate.
Deep Agents Approval Evidence
Use separate evidence for the default interactive posture, the optional thread-scoped capability, and headless automation.
Do not infer the active state of a TUI thread from the host-side configured mode.
Evidence case
Required checks
Default-disabled interactive
Host status reports disabled, the TUI auto-approval choice and dcode -y fail closed, and ordinary interactive tool requests still prompt.
Capability configuration
A named rebuild with --dcode-auto-approval thread-opt-in succeeds, host status reports thread-opt-in, and the operation records the exact NemoClaw commit and Deep Agents Code version.
Explicit thread activation
The operator selects Auto-approve for this thread or launches dcode -y, the TUI shows the active-state indicator and warning, and more than one tool call completes without another prompt in that same thread.
Thread reset
A new process, /clear, /force-clear, a thread switch or resume, and an agent switch each return to manual approval before another explicit opt-in.
Residual controls
Enabled-mode evidence repeats network-policy denial and credential non-disclosure checks and confirms that managed inference, MCP, filesystem, and process restrictions remain active.
Headless separation
dcode -n evidence remains in its own row and records non-shell auto-approval plus disabled shell execution without attributing that behavior to thread-opt-in.
Audit Matrix
These seed rows come from current repo source files, not from live benchmark claims.
Keep them as not-yet-run until the row has evidence that satisfies the schema above.
When importing a completed row from an issue comment, preserve the exact commit SHA, workflow, evidence paths, and observed behavior.
Add trajectory and session evidence before changing state.
Existing setup keeps Tool Search disabled and preserves direct structured tool calls, overriding the generated tools.toolSearch.mode: tools default for this route.
Verify tool_search, tool_describe, tool_call, and final execution before replacing the safeguard.
User-selected custom-model or another configured model id.
Managed inference.local route to the compatible endpoint.
not-yet-run
Add endpoint class and trajectory evidence before changing state.
Record endpoint API path forcing and store/streaming assumptions.
Add one row per endpoint class that is validated.
src/lib/inference/config.ts.
OpenClaw primary agent
Other Anthropic-compatible endpoint
User-selected custom-anthropic-model or another configured model id.
anthropic route when supported, otherwise managed compatible route.
not-yet-run
Add endpoint class and trajectory evidence before changing state.
Record native Anthropic Messages or compatible-route transport behavior.
Add one row per endpoint class that is validated.
src/lib/inference/config.ts.
Hermes sandbox API
Hermes Provider
Default moonshotai/kimi-k2.6 or any model from HERMES_PROVIDER_MODEL_OPTIONS.
Hermes Provider route through NemoClaw managed inference.
not-yet-run
Add Hermes session, request dump, logs, and local API evidence before changing state.
Generated config uses native tools.tool_search.enabled: on with snake-case 5/20 limits; core tools stay direct while deferred MCP and non-core plugin tools use structured search, describe, and call.
Verify a deferred-tool trajectory and keep it separate from OpenClaw mode: tools evidence.
Managed inference.local OpenAI-compatible Chat Completions with use_responses_api = false.
not-yet-run
Add separate default-disabled and thread-opt-in terminal transcripts, host status output, reset evidence, and route evidence before changing state.
The managed Ultra profile preserves required nonempty tool-call content and rejects the observed literal [content] execute placeholder before shell dispatch; optional thread-opt-in remains a per-thread approval affordance.
Verify a terminal task with approval prompts intact, then separately verify explicit thread activation, reset behavior, policy enforcement, placeholder rejection, and no provider credential in sandbox-visible files.
Copy this template when adding evidence for a specific model/provider/agent combination.
Do not leave placeholder text in a completed row.
Field
Value
Model ID
<provider/model-id>.
Provider path
<provider class and route>.
Agent surface
<OpenClaw primary agent, OpenClaw CLI prompt path, OpenClaw browser or gateway path, OpenClaw sub-agent delegation, Hermes sandbox API, Deep Agents interactive dcode, Deep Agents headless dcode -n, or auxiliary model path>.
NemoClaw commit SHA
<full SHA>.
Runtime versions
<OpenShell version, OpenClaw version, Hermes version, Deep Agents Code version, local server version, or n/a>.
Endpoint/API path selected
<provider key, base URL class, API mode, and endpoint path>.
Workflow used
<exact commands or CI workflow>.
State
<pass, pass-with-affordance, degraded, blocked, unsupported, or not-yet-run>.
Evidence
<trajectory, session log, request dump, CI artifact, or n/a>.
Observed tool-call count
<count, names, and shape>.
Final-response behavior
<final answer, empty stop, reasoning-only stop, raw tool text, or other behavior>.
Multi-turn behavior
<turn 1 and turn 2 behavior>.
Latency and timeout notes
<validation time, first event timing, total duration, retry behavior, timeout budget, and streaming notes>.
Required affordance
<none, setup manifest, request mutation, parser flag, API path forcing, streaming requirement, or transport policy>.
Follow-up
<issue, PR, registry decision, or n/a>.
Related Artifacts
nemoclaw-blueprint/model-specific-setup/README.md documents where model-specific setup belongs once an intervention is justified.
docs/reference/troubleshooting#tool-calls-appear-as-assistant-text explains the local inference tool-call failure mode that audit rows should classify separately from provider connectivity.
