Cosmos3 NIM#

Note

The commands and Python examples on these pages run from the .agents/skills/cookbooks/cosmos3/nim directory of the NVIDIA/cosmos-framework GitHub repository. Clone the repository and change to that directory before you run commands such as uv run python examples/t2i.py.

Deploy Cosmos3 models for world generation and multimodal reasoning. One image contains two runtimes, but each container runs only one:

Generator serves image/video generation, Action, and Transfer through POST /v1/infer.
Reasoner serves OpenAI-compatible image/video understanding through Chat Completions and, when enabled, the Responses API.

Deployment commands pin the current release image, nvcr.io/nim/nvidia/cosmos3:2.1.0. Compatible configurations are in the Support Matrix.

Architecture#

Cosmos3 is a Mixture-of-Transformers (MoT) world foundation model with two jointly trained towers that share latent representations. The Reasoner (vision-language) tower grounds the prompt. The Generator (world simulator) tower produces the output, such as video. The Generator runtime uses both towers, and the Reasoner runtime serves understanding. For details, refer to the Cosmos3 technical report.

Choose What to Run#

Choose a runtime, a model variant (NIM_MODEL_VARIANT), optionally a precision, and for Generator a latency or throughput objective. The NIM selects a compatible profile, as described in How Selection Works.

Choose a Task#

Task	Runtime	Input	Output	Guide
Text-to-image	Generator	Prompt	JPEG image	Generation
Text-to-video	Generator	Prompt	MP4 video	Generation
Image-to-video	Generator	Prompt + image	MP4 video	Generation
Video-to-video	Generator	Prompt + video	MP4 video	Generation
Forward dynamics	Generator	Image + action trajectory	Rollout video	Action
Policy	Generator	Image + task/state	Video + predicted action	Action
Nano-DROID policy	Generator	Image + task/current state	Predicted action	Action
Edge-DROID policy (experimental)	Generator	Composed camera views + task/current state	Predicted action	Action
Inverse dynamics	Generator	Video	Video + predicted action	Action
Video Transfer	Generator	Prompt + spatial control	Controlled MP4 video	Transfer
Image/video reasoning	Reasoner	Media + text	Text or structured result	Reasoning
Streaming video reasoning	Reasoner	Sequential image frames	Per-frame text with retained context	Streaming
Responses API	Reasoner	Responses input	Response object/text	Reasoning

Local examples publish either runtime at http://localhost:8000. Remove the active example container before launching the other runtime on the same port.

Get Started#

Use an Existing NIM Endpoint#

If an administrator already deployed the NIM, install and initialize the Client Tooling, then set the service URL:

cd .agents/skills/cookbooks/cosmos3/nim
export NIM_URL=${NIM_URL:-http://localhost:8000}
curl -fsS "$NIM_URL/v1/health/ready"
curl -fsS "$NIM_URL/v1/metadata" | python3 -m json.tool

Confirm that metadata identifies the intended runtime and primary endpoint. Generator uses /v1/infer, and Reasoner uses /v1/chat/completions.

Run an example that matches the active runtime and model. For a general-purpose Generator, start with an image request:

uv run python examples/t2i.py

For a Reasoner:

uv run python examples/reasoner.py --case image_caption

Deploy the NIM Yourself#

Verify the GPU host against Prerequisites.
Choose a compatible model and hardware configuration from the Support Matrix.
Follow Deployment to authenticate, pull the image, run the preflight, and launch Generator or Reasoner. On NVIDIA DGX Spark or Jetson AGX Thor, the NIM applies a 70% unified-memory share automatically.
Return to Use an Existing NIM Endpoint to verify the service and run the first request.

Specialist variants accept only their task: run t2i_4step.py, i2v_4step.py, or policy_droid.py only against the matching variant. Examples save outputs under the ignored examples/outputs/ directory.

Validate the selected configuration with representative requests before production use.

Safety, License, and Notices#

Generator guardrails are enabled by default. Disabling them can remove content-policy and face-privacy protections. Refer to Guardrails.

Refer to License and Notices for the terms. This cookbook is licensed under the repository LICENSE. The running NIM serves its license at /v1/license. The image also contains product terms, notices, and component licenses. Python package licenses are listed in Acknowledgements.

The NIM is distributed as a container image through the NVIDIA NGC Catalog. The catalog publishes a security scan report for each image, with a security rating, CVE severity by package, and links to CVE details.

NVIDIA Developer Program#

Want to learn more about NIMs? Join the NVIDIA Developer Program to get free access to self-hosting NVIDIA NIMs and microservices on up to 16 GPUs on any infrastructure: cloud, data center, or personal workstation.

After you join the free NVIDIA Developer Program, you can access NIMs through the NVIDIA API Catalog at any time. For enterprise-grade security, support, and API stability, select the option to access NIM through our free 90-day NVIDIA AI Enterprise Trial with a business email address.

Refer to the NVIDIA NIM FAQ for additional information.