NVIDIA Cosmos

Overview

• Introduction
  • Cosmos-Predict2
  • Cosmos-Predict1
    • Examples
      • Cosmos-Predict1-7B-Text2World-Multiview
      • Cosmos-Predict1-5B-Video2World
    • Getting Started Workflow
  • Cosmos-Transfer1
    • Examples
      • Cosmos-Transfer1-7B
      • Cosmos-Transfer1-7B [LiDAR|HDMap]
  • Cosmos-Reason1
• Release Notes
  • 1.1
  • 1.0
• Prerequisites
  • Hardware
    • Predict2
    • Predict1
    • Transfer1
    • Reason1
  • Software
• NVIDIA NIM for Cosmos

Cosmos-Predict2

• Model Matrix
  • Performance Benchmarks
  • Model Selection Guide
• Installation
  • Option 1: Conda Environment Setup
  • Option 2: Docker Container Setup
  • Next Steps
• Predict2 Quickstart Guide
  • Set up the Video2World Model
  • Generate a Video from Text Input
  • Next Steps
• Predict2 Post-Training Guide
  • Set up the Video2World Model
  • Post-training with a Custom Dataset
    • 1. Preparing Data
    • 2. Creating Configs for Training
    • 2.1. Config System
      • Model Configurations (defaults/model.py)
      • Optimizer Configurations (defaults/optimizer.py)
      • Scheduler Configurations (defaults/scheduler.py)
      • Data Configurations (defaults/data.py)
      • Checkpoint Configurations (defaults/checkpoint.py)
      • Callback Configurations (defaults/callbacks.py)
    • 3. Run a Training Job.
    • 4. Run Inference on Post-trained Checkpoints
      • Cosmos-Predict2-2B-Video2World
      • Cosmos-Predict2-14B-Video2World
  • Post-Training from an Example Pre-trained Video2World Checkpoint
    • 1. Preparing Data
      • 1.1 Download Bridge training dataset
    • 2. Post-training
      • 2.1. Cosmos-Predict2-2B-Video2World
        • Checkpoint Output Structure
        • Configuration Snippet
    • 3. Inference for Bridge
      • 3.1. Cosmos-Predict2-2B-Video2World
  • Post-Training with Cosmos-NeMo-Assets
    • 1. Preparing Data
      • 1.1 Downloading Cosmos-NeMo-Assets
      • 1.2 Preprocessing the Data
    • 2. Post-training
      • 2.1. Post-training on Cosmos-NeMo-Assets dataset
        • Cosmos-Predict2-2B-Video2World
        • Cosmos-Predict2-14B-Video2World
    • 3. Inference with the Post-trained checkpoint
      • 3.1 Inference
        • Cosmos-Predict2-2B-Video2World
        • Cosmos-Predict2-14B-Video2World
• Predict2 Model Reference
  • Model Types
  • Text2Image
    • Download Commands
    • Examples
      • Single Image Generation
      • Batch Image Generation
  • API Documentation
    • Prompt Engineering Tips
  • Video2World
    • Examples
      • Single Video Generation
      • Batch Video Generation
      • Multi-Frame Video Conditioning
      • Using the Prompt Refiner
      • Multi-GPU Inference
    • Rejection Sampling for Quality Improvement
    • API Documentation
  • Specialized Scripts
    • Rejection Sampling (video2world_bestofn.py)
    • Prompt Engineering Tips
  • Text2World
    • Examples
      • Single Video Generation
    • Batch Video Generation
      • Multi-GPU Inference
    • API Documentation
    • Prompt Engineering Tips

Cosmos-Predict1

• Model Matrix
  • World Foundation Models
  • Tokenizers
• Installation
  • Cosmos-Predict1 Setup
  • Next Steps
• Diffusion
  • Diffusion Quickstart Guide
    • Set up the Diffusion Model
    • Generate a Video from Text Input
    • Next Steps
  • Diffusion Model Post-Training Guide
    • Setup
    • Preparing a Dataset
    • Post-Training
      • Text2World
        • Download the Sample Dataset
        • Preprocess the Data
        • Post-Train the Model
        • Test Inference
      • Video2World
        • Download the Sample Dataset
        • Preprocess the Data
        • Post-train the Model
        • Test Inference
      • Text2World-Multiview
        • Post-Train the Model
        • Test Inference
      • Video2World-Multiview
        • Post-Train the Model
        • Test Inference
  • Diffusion Model Reference
    • Model Types
    • Sample Commands
      • Text2World
        • Downloading the Model Weights
        • Single Generation
        • Single Generation with Model Offloading
        • Single Generation with Multi-GPU Inference
        • Batch Generation
      • Video2World
        • Downloading the Model Weights
        • Single Generation
        • Single Generation with Model Offloading
        • Single Generation with Multi-GPU Inference
        • Batch Generation
        • Batch Generation without Prompt Upsampling
      • Text2World-Multiview
        • Downloading the Model Weights
        • Multiview Prompts
        • Single Generation
        • Single Generation with Model Offloading
        • Single Generation with Multi-GPU Inference
        • Batch Generation
      • Video2World-Multiview
        • Downloading the Model Weights
        • Single Generation
        • Single Generation with Model Offloading
        • Single Generation with Multi-GPU Inference
        • Batch Generation
    • Arguments
      • Common Parameters
      • Video2World-Specific Parameters
    • Prompting Guidelines
      • Prompt Example
      • Cosmos-Predict1-Prompt-Upsampler
      • Text2World
      • Video2World
    • Safety Features
• Autoregressive
  • Autoregressive Quickstart Guide
    • Set up the Autoregressive Model
    • Generate a Video using Video Input
    • Next Steps
  • Autoregressive Post-Training Guide
    • Supported Models
    • Setup
    • Preparing a Dataset
    • Post-Training
      • Download the Sample Dataset
      • Post-Train the Model
        • Post-Train with a Single GPU (TP=1)
        • Post-train with 4 GPUs (TP = 4)
      • Test Inference
  • Autoregressive Model Reference
    • Model Types
    • Single and Batch Generation
      • Single Video
      • Batch Video
    • Text Prompt
    • Sample Commands
      • Base
        • Downloading the Model Weights
        • Single Generation
        • Single Generation with Multi-GPU Inference
        • Batch Generation
      • Video2World
        • Downloading the Model Weights
        • Single Generation
        • Single Generation with Multi-GPU Inference
        • Batch Generation
    • Arguments
      • Common Parameters
        • Base Specific Parameters
        • Video2World Specific Parameters
    • Safety Features
• Tokenizer
  • Tokenizer Inference Guide
    • Set up the Tokenizer
    • Tokenizer Overview
    • Examples
  • Tokenizer Post-Training Guide
    • Set up the Tokenizer
    • Post-training a Tokenizer
      • Download a Dataset
      • Post-train the Video Tokenizers
      • Inference with the Post-trained Model Checkpoint

Cosmos-Transfer1

• Installation
  • Cosmos-Transfer1 Setup
  • Next Steps
• Transfer Quickstart Guide
  • Set up Cosmos Transfer1
  • Generate a Visual Simulation from Source Video
• Transfer Model Reference
  • Sample Commands
    • Edge Detection ControlNet
      • Multi-GPU Inference
      • Inference with Prompt Upsampling
      • Batch Inference
    • Multimodal Control
      • Multimodal control with Spatiotemporal Control Map
    • Autonomous Vehicle Transfer
      • Multi-GPU Inference
      • Additional AV Toolkits
    • 4K Upscaling
      • Multi-GPU Inference
  • Arguments
  • ControlNet Specification
  • Prompting Guidelines
  • Safety Features

Cosmos-Reason1

• Installation
  • Cosmos-Reason1 Setup
  • Monitoring Setup
  • Next Steps
• Reason1 Training Guide
  • Training Configuration
  • Job recipes
  • GPU Requirements
  • Supervised Fine-Tuning (SFT)
  • Reinforcement Learning (RL)
  • Inference
  • Next Steps
• Reason1 Evaluation Guide
  • Download Benchmark Assets
  • Download Additional Video Clips (Optional)
  • Run Evaluation on Benchmarks
    • Configure Evaluation
    • Run Evaluation
  • Benchmark Scoring
    • About Evaluation
    • Usage

More Information

• Frequently Asked Questions
• Feedback and Support
• Cosmos Guardrail
  • Overview
    • Pre-Guard
    • Post-Guard
• License