Quickstart Guide#

This guide provides instructions on running inference with the Cosmos-Transfer2.5/general model.

Note

Ensure you have completed the steps in the Transfer2.5 Installation Guide before running inference.

Example Inference Command#

Individual control variants can be run on a single GPU:

python examples/inference.py --params_file assets/robot_example/depth/robot_depth_spec.json

For multi-GPU inference on a single control, or to run multiple control variants, use torchrun:

export NUM_GPUS=8
torchrun --nproc_per_node=$NUM_GPUS --master_port=12341 -m examples.inference --params_file assets/robot_example/vis/robot_vis_spec.json --num_gpus=$NUM_GPUS

Example Parameter Files#

An example parameter file for each individual control variant is provided, along with a multi-control variant:

Variant

Parameter File

Depth

assets/robot_example/depth/robot_depth_spec.json

Edge

assets/robot_example/edge/robot_edge_spec.json

Segmentation

assets/robot_example/seg/robot_seg_spec.json

Blur

assets/robot_example/vis/robot_vis_spec.json

Multi-control

assets/robot_example/multicontrol/robot_multicontrol_spec.json

Parameters can be specified as follows:

{
    // Path to the prompt file, use "prompt" to directly specify the prompt
    "prompt_path": "assets/robot_example/robot_prompt.json",

    // Directory to save the generated video
    "output_dir": "outputs/robot_multicontrol",

    // Path to the input video
    "video_path": "assets/robot_example/robot_input.mp4",

    // Inference settings
    "guidance": 3,

    // Depth control settings
    "depth": {
        // Path to the control video
        // For "vis" and "edge", if a control is not provided, it will be computed on the fly.
        "control_path": "assets/robot_example/depth/robot_depth.mp4",

        // Control weight for the depth control
        "control_weight": 0.5
    },

    // Edge control settings
    "edge": {
        // Path to the control video
        "control_path": "assets/robot_example/edge/robot_edge.mp4",
        // Default control weight of 1.0 for edge control
    },

    // Seg control settings
    "seg": {
        // Path to the control video
        "control_path": "assets/robot_example/seg/robot_seg.mp4",

        // Control weight for the seg control
        "control_weight": 1.0
    },

    // Blur control settings
    "vis":{
        // Control video computed on the fly
        "control_weight": 0.5
    }
}

Example Output#

The following video shows output from a multiple control variant:

Next Steps#

Refer to the :ref:Transfer2.5 Model Reference <transfer2.5-model-reference> page for more information on running inference with the Auto Multiview model. If you’re ready to start post-training, refer to the :ref:Transfer2.5 Post-Training Guides <transfer2.5-post-training-guides> page.