Skip to navigation
Vision AgentAgent Harnesses

NemoClaw

View as Markdown

NemoClaw is an open-source reference stack for running always-on AI agents more safely inside NVIDIA OpenShell sandboxes. Integrated with the Video Search and Summarization (VSS) blueprint, it onboards an agent harness such as OpenClaw or Hermes—which provides the chat UI for controlling VSS deployments through natural language—backed by the repository’s VSS skills, workspace bootstrap docs, a VSS policy preset, and a host-side orchestrator MCP server.

What NemoClaw brings to VSS

Running VSS through NemoClaw turns a multi-service Docker Compose deployment into a chat session:

  • Natural-language operations. Prompts like “Deploy the VSS alerts profile in verification mode” or “Show me the logs from vss-alert-bridge” are mapped to the right orchestrator tools in the right order, without manual .env editing or compose flag juggling.
  • Guided host setup. Two bundled Jupyter notebooks walk through the Brev/host prerequisites: credentials, Docker version pinning, nvcr.io login, the VSS agent Python environment, and the host-side MCP server.
  • Policy-controlled skills. The setup applies NemoClaw’s VSS policy specified in a yaml file. The sandbox image is built from the repository’s harness Dockerfile: .openclaw/Dockerfile extends NemoClaw’s managed OpenClaw runtime with the VSS OpenClaw plugin (the vss CLI as an agent tool, the operation skills activated per deployment from what vss configure records, and the workspace bootstrap docs); .hermes/Dockerfile extends the managed Hermes runtime with the same skills (activated per deployment by vss-hermes-sync), docs and vss CLI.
  • Operational visibility. The VSS Orchestrator MCP exposes deployment status and container logs to the agent so it can inspect and remediate a deployment from chat.

Summaries, reports, and follow-up prompts

With NemoClaw’s OpenClaw runtime, video summaries and reports appear directly in the OpenClaw chat UI. Human in the Loop (HITL) interaction happens in that chat: when the agent needs a scenario, events, objects of interest, or other report details, it asks inline and you reply there. An external VSS UI connected to NemoClaw does not support the VSS Agent’s structured HITL dialogs; follow-up questions remain ordinary chat messages.

The in-stack VSS Agent UI uses HITL dialogs for these prompts and provides generated reports as PDF downloads. Screenshots of dialogs and PDF report links in the quickstart and Video Summarization workflow show that VSS Agent experience.

How to run NemoClaw with VSS

The NemoClaw notebooks are released as an Early Access feature.

This is the standalone, notebook-driven setup route. If you are following the Build Vision AI quickstart, let the Skill provision NemoClaw when prompted and do not also run these notebooks for the same deployment.

There are two host-side pieces working together:

  1. Two setup notebooks that run in order on the Brev instance:

    • deploy_nemoclaw.ipynb chooses the agent chat model provider, pins Docker to the tested package versions, installs the pinned NemoClaw release, creates the sandbox from the repository’s harness Dockerfile, applies the VSS policy and deployment origin, optionally registers an HTTPS VSS Orchestrator MCP path, configures optional webhooks, and opens the Agent UI. The harness image already contains the VSS skills, workspace docs, and vss CLI.
    • deploy_vss_orchestrator.ipynb prepares the host for local NIM-backed profiles, starts the MCP server, deploys and manages VSS from the Agent UI, and optionally verifies sandbox-to-host reachability.

    See Setup VSS and Skills.

  2. The VSS Orchestrator MCP server - a host process started by deploy_vss_orchestrator.ipynb on port 9988. It exposes nine vss_orchestrator__* tools that the NemoClaw agent calls to:

    • Generate compose artifacts.
    • Run docker compose operations.
    • Inspect containers.
    • Fetch logs.

    See VSS Orchestrator MCP.

To route NemoClaw’s agent model through Switchyard, set ROUTE_NEMOCLAW = "true" in deploy_vss_switchyard.ipynb, run that notebook through Section 6, and leave the router running. Then run deploy_nemoclaw.ipynb, select option (a), and use the router URL and route ID as the model, followed by deploy_vss_orchestrator.ipynb.

For per-variable details, provider choices, and optional verification steps, see Additional Configuration.