NVIDIA Cosmos

Overview

• Introduction
  • Cosmos-Predict2
  • Cosmos-Predict1
    • Examples
      • Cosmos-Predict1-7B-Text2World-Multiview
      • Cosmos-Predict1-5B-Video2World
    • Getting Started Workflow
  • Cosmos-Transfer1
    • Examples
      • Cosmos-Transfer1-7B
      • Cosmos-Transfer1-7B [LiDAR|HDMap]
  • Cosmos-Reason1
    • Additional Resources
    • License and Contact
• Release Notes
  • 1.2
  • 1.1
  • 1.0
• Prerequisites
  • Hardware
    • Predict2
    • Predict1
    • Transfer1
    • Reason1
  • Software
• NVIDIA NIM for Cosmos

Cosmos-Predict2

• Model Matrix
  • Performance Benchmarks
  • Video2World Sparse Attention Variants
  • Model Selection Guide
• Installation
  • Option 1: Conda Environment Setup
  • Option 2: Docker Container Setup
  • Next Steps
• Quickstart Guide
  • Set up the Video2World Model
  • Generate a Video from Text Input
  • Troubleshooting
  • Next Steps
• Video2World Post-Training Guides
  • Post-training with a Custom Dataset
    • Set up the Video2World Model
      • Prepare the Dataset
      • Create Configs for Training
      • Configure the System
        • Model Configurations (defaults/model.py)
        • Optimizer Configurations (defaults/optimizer.py)
        • Scheduler Configurations (defaults/scheduler.py)
        • Data Configurations (defaults/data.py)
        • Checkpoint Configurations (defaults/checkpoint.py)
        • Callback Configurations (defaults/callbacks.py)
      • Run a Training Job
      • Perform Inference on Post-trained Checkpoints
      • Cosmos-Predict2-2B-Video2World
      • Cosmos-Predict2-14B-Video2World
  • Post-training for Action-Conditioned Video Prediction
    • Set up the Video2World Model
    • Prepare the Data
      • Download Bridge Training Dataset
    • Post-training
      • 2.1. Cosmos-Predict2-2B-Video2World
        • Checkpoint Output Structure
        • Configuration Snippet
    • Inference
      • Cosmos-Predict2-2B-Video2World
  • Post-training with Cosmos-NeMo-Assets
    • Set up the Video2World Model
    • Prepare the Dataset
      • Download Cosmos-NeMo-Assets
      • Preprocess the Data
    • Post-train the Video2World Model
      • Cosmos-Predict2-2B-Video2World
      • Cosmos-Predict2-14B-Video2World
    • Run Inference on Post-Trained Checkpoints
      • Cosmos-Predict2-2B-Video2World
      • Cosmos-Predict2-14B-Video2World
  • Post-training with AgiBotWorld-Alpha
    • Set up the Video2World Model
    • Prepare the Dataset
      • Download and Pre-Process the AgiBotWorld-Alpha Dataset
      • Preprocess the Data
    • Post-train the Video2World Model
      • Cosmos-Predict2-2B-Video2World
        • Resolution and FPS Variants
      • Cosmos-Predict2-14B-Video2World
        • Resolution and FPS Variants
    • Post-training Performance
    • Perform inference with the Post-trained Checkpoint
      • Cosmos-Predict2-2B-Video2World
      • Cosmos-Predict2-14B-Video2World
  • Post-training with GR00T-Dreams-GR1 and GR00T-Dreams-DR0ID Datasets
    • Set up the Video2World Model
    • Prepare the Dataset
      • Download the DreamGen Bench Training Dataset
      • 1.2 Preprocessing the Data
    • Post-train the Video2World Model
      • Cosmos-Predict2-2B-Video2World
      • Cosmos-Predict2-14B-Video2World
    • Perform Inference for GR00T Dreams Checkpoints
      • Inference with GR1 Checkpoint
      • Inference with DROID Checkpoint
    • Perform Inference with DreamGen Benchmark
      • Download the DreamGen Benchmark Dataset
    • Perform Inference with Cosmos-Reason1 Rejection Sampling
• Text2Image Post-Training Guides
  • Post-training with a Custom Dataset
    • Set up the Text2Image Model
    • Preparing the Dataset
    • Create Configs for Training
      • Configure the System
        • Model Configurations (defaults/model.py)
        • Optimizer Configurations (defaults/optimizer.py)
        • Scheduler Configurations (defaults/scheduler.py)
        • Data Configurations (defaults/data.py)
        • Checkpoint Configurations (defaults/checkpoint.py)
        • Callback Configurations (defaults/callbacks.py)
    • Run a Training Job
    • Run Inference on Post-Trained Checkpoints
      • Cosmos-Predict2-2B-Text2Image
      • Cosmos-Predict2-14B-Text2Image
  • Post-training with Cosmos-NeMo-Assets
    • Set up the Text2Image Model
    • Preparing the Dataset
      • Downloading Cosmos-NeMo-Assets
      • Preprocess the Data
    • Post-train the Text2Image Model
      • Cosmos-Predict2-2B-Text2Image
      • Cosmos-Predict2-14B-Text2Image
    • Run Inference on Post-Trained Checkpoints
      • Cosmos-Predict2-2B-Text2Image
      • Cosmos-Predict2-14B-Text2Image
• Model Reference
  • Model Types
  • Text2Image
    • Download Commands
    • Examples
      • Single Image Generation
      • Batch Image Generation
    • API Documentation
    • Prompt Engineering Tips
  • Video2World
    • Download Commands
    • Examples
      • Single Video Generation
      • Batch Video Generation
      • Multi-Frame Video Conditioning
      • Using the Prompt Refiner
      • Multi-GPU Inference
    • Rejection Sampling for Quality Improvement
    • Long Video Generation
    • Faster inference with Sparse Attention
    • API Documentation
    • Specialized Scripts
    • Prompt Engineering Tips
  • Text2World
    • Examples
      • Single Video Generation
    • Batch Video Generation
      • Multi-GPU Inference
    • API Documentation
    • Prompt Engineering Tips

Cosmos-Predict1

• Model Matrix
  • World Foundation Models
  • Tokenizers
• Installation
  • Cosmos-Predict1 Setup
  • Next Steps
• Diffusion
  • Diffusion Quickstart Guide
    • Set up the Diffusion Model
    • Generate a Video from Text Input
    • Next Steps
  • Diffusion Model Post-Training Guide
    • Setup
    • Preparing a Dataset
    • Post-Training
      • Text2World
        • Download the Sample Dataset
        • Preprocess the Data
        • Post-Train the Model
        • Test Inference
      • Video2World
        • Download the Sample Dataset
        • Preprocess the Data
        • Post-train the Model
        • Test Inference
      • Text2World-Multiview
        • Post-Train the Model
        • Test Inference
      • Video2World-Multiview
        • Post-Train the Model
        • Test Inference
  • Diffusion Model Reference
    • Model Types
    • Sample Commands
      • Text2World
        • Downloading the Model Weights
        • Single Generation
        • Single Generation with Model Offloading
        • Single Generation with Multi-GPU Inference
        • Batch Generation
      • Video2World
        • Downloading the Model Weights
        • Single Generation
        • Single Generation with Model Offloading
        • Single Generation with Multi-GPU Inference
        • Batch Generation
        • Batch Generation without Prompt Upsampling
      • Text2World-Multiview
        • Downloading the Model Weights
        • Multiview Prompts
        • Single Generation
        • Single Generation with Model Offloading
        • Single Generation with Multi-GPU Inference
        • Batch Generation
      • Video2World-Multiview
        • Downloading the Model Weights
        • Single Generation
        • Single Generation with Model Offloading
        • Single Generation with Multi-GPU Inference
        • Batch Generation
    • Arguments
      • Common Parameters
      • Video2World-Specific Parameters
    • Prompting Guidelines
      • Prompt Example
      • Cosmos-Predict1-Prompt-Upsampler
      • Text2World
      • Video2World
    • Safety Features
• Autoregressive
  • Autoregressive Quickstart Guide
    • Set up the Autoregressive Model
    • Generate a Video using Video Input
    • Next Steps
  • Autoregressive Post-Training Guide
    • Supported Models
    • Setup
    • Preparing a Dataset
    • Post-Training
      • Download the Sample Dataset
      • Post-Train the Model
        • Post-Train with a Single GPU (TP=1)
        • Post-train with 4 GPUs (TP = 4)
      • Test Inference
  • Autoregressive Model Reference
    • Model Types
    • Single and Batch Generation
      • Single Video
      • Batch Video
    • Text Prompt
    • Sample Commands
      • Base
        • Downloading the Model Weights
        • Single Generation
        • Single Generation with Multi-GPU Inference
        • Batch Generation
      • Video2World
        • Downloading the Model Weights
        • Single Generation
        • Single Generation with Multi-GPU Inference
        • Batch Generation
    • Arguments
      • Common Parameters
        • Base Specific Parameters
        • Video2World Specific Parameters
    • Safety Features
• Tokenizer
  • Tokenizer Inference Guide
    • Set up the Tokenizer
    • Tokenizer Overview
    • Examples
  • Tokenizer Post-Training Guide
    • Set up the Tokenizer
    • Post-training a Tokenizer
      • Download a Dataset
      • Post-train the Video Tokenizers
      • Inference with the Post-trained Model Checkpoint

Cosmos-Transfer1

• Installation
  • Cosmos-Transfer1 Setup
  • Next Steps
• Transfer Quickstart Guide
  • Set up Cosmos Transfer1
  • Generate a Visual Simulation from Source Video
• Transfer Model Reference
  • Sample Commands
    • Edge Detection ControlNet
      • Multi-GPU Inference
      • Inference with Prompt Upsampling
      • Batch Inference
    • Multimodal Control
      • Multimodal control with Spatiotemporal Control Map
    • Autonomous Vehicle Transfer
      • Multi-GPU Inference
      • Additional AV Toolkits
    • 4K Upscaling
      • Multi-GPU Inference
  • Arguments
  • ControlNet Specification
  • Prompting Guidelines
  • Safety Features

Cosmos-Reason1

• Installation
  • Next Steps
• Quickstart Guide
  • Basic Example
  • Complete Examples
  • Configuring Inference
  • Next Steps
• Using Cosmos-Reason1 for Rejection Sampling
  • Generating Multiple Rollouts
  • Evaluating Physical Accuracy Using Cosmos-Reason1
  • Example
• Post-Training Guide
  • Setup
  • Monitoring
  • Training
    • Supervised Fine-Tuning (SFT)
      • Configuration
      • Training
    • Reinforcement Learning (RL)
      • Configuration
      • Training
  • Next Steps
• Benchmark Example
  • Setup
  • Prepare the Dataset
  • Run Evaluation
  • Calculate Accuracy

More Information

• Frequently Asked Questions
• Feedback and Support
• Cosmos Guardrail
  • Overview
    • Pre-Guard
    • Post-Guard
• License