Captions and Preview

Prepare inputs, generate window-level captions, optionally refine or enhance them, and produce preview images.

Choose a captioning model

Use the same model_variant in CaptionPreparationStage and CaptionGenerationStage. The preparation stage formats prompts and video frames for that exact checkpoint; the generation stage looks up the prepared input by the variant name.

All checkpoints are downloaded from Hugging Face on each node when absent from model_dir. Caption generation requires vLLM and reserves one GPU per worker. For Qwen, use the fp8 parameter or --captioning-use-fp8-weights; for Nemotron Nano 12B v2, select the precision-specific variant instead. Nemotron 3 Nano Omni uses one video per prompt, a 131,072-token model context, and vLLM compatibility patches that NeMo Curator applies automatically to the downloaded checkpoint.

Quickstart

1
from nemo_curator.pipeline import Pipeline
2
from nemo_curator.stages.video.caption.caption_generation import CaptionGenerationStage
3
from nemo_curator.stages.video.caption.caption_preparation import CaptionPreparationStage
4
from nemo_curator.stages.video.preview.preview import PreviewStage
5
6
model_variant = "qwen2.5"
7
8
pipe = Pipeline(name="captions_preview")
9
pipe.add_stage(
10
    CaptionPreparationStage(
11
        model_variant=model_variant,
12
        prompt_variant="default",
13
        sampling_fps=2.0,
14
        window_size=256,
15
        remainder_threshold=128,
16
        preprocess_dtype="float16",
17
        generate_previews=True,
18
    )
19
)
20
pipe.add_stage(PreviewStage(target_fps=1.0, target_height=240))
21
pipe.add_stage(
22
    CaptionGenerationStage(
23
        model_dir="/models",
24
        model_variant=model_variant,
25
        caption_batch_size=8,
26
        max_output_tokens=512,
27
    )
28
)
29
pipe.run()

Prepare caption inputs

CaptionPreparationStage splits each clip into windows, samples frames, and writes the model-ready value to window.llm_inputs[model_variant]. If generate_previews=True, it also keeps MP4 bytes for PreviewStage.

1
from nemo_curator.stages.video.caption.caption_preparation import CaptionPreparationStage
2
3
prep = CaptionPreparationStage(
4
    model_variant="qwen3",
5
    prompt_variant="default",
6
    sampling_fps=2.0,
7
    window_size=256,
8
    remainder_threshold=128,
9
    preprocess_dtype="float16",
10
    generate_previews=True,
11
)

Generate captions

CaptionGenerationStage consumes window.llm_inputs[model_variant] and writes its result to window.caption[model_variant]. After generation it removes that prepared input and the window MP4 bytes.

1
from nemo_curator.stages.video.caption.caption_generation import CaptionGenerationStage
2
3
gen = CaptionGenerationStage(
4
    model_dir="/models",
5
    model_variant="qwen3",
6
    caption_batch_size=8,
7
    fp8=False,
8
    max_output_tokens=512,
9
    disable_mmcache=False,
10
    generate_stage2_caption=False,
11
)

Refine or enhance captions

NeMo Curator provides two distinct second-pass workflows:

• Same-VLM refinement reruns the selected captioning model with its first caption. Set generate_stage2_caption=True, or pass --captioning-stage2-caption. The refined text replaces the first-pass value in window.caption[model_variant].
• Text-only enhancement runs a separate Qwen language model after caption generation. Add CaptionEnhancementStage, or pass --enhance-captions. The result is stored in window.enhanced_caption["qwen_lm"] and the original caption remains available.

1
from nemo_curator.stages.video.caption.caption_enhancement import CaptionEnhancementStage
2
3
enhance = CaptionEnhancementStage(
4
    model_dir="/models",
5
    model_variant="qwen3",
6
    captioning_model_variant="qwen3",
7
    prompt_variant="default",
8
    model_batch_size=128,
9
    fp8=False,
10
    max_output_tokens=512,
11
)

The enhancement model can be qwen2.5 (Qwen/Qwen2.5-14B-Instruct) or qwen3 (Qwen/Qwen3-14B). captioning_model_variant must identify the caption key produced by the earlier generation stage; it does not have to match the enhancement model.

In the example script, --enhance-captions-algorithm selects qwen2.5 or qwen3. The separate --enhanced-caption-models output selector currently accepts only qwen_lm.

$
python tutorials/video/getting-started/video_split_clip_example.py \
>
  ... \
>
  --generate-captions \
>
  --captioning-algorithm nemotron-fp8 \
>
  --enhance-captions \
>
  --enhance-captions-algorithm qwen3 \
>
  --enhanced-caption-models qwen_lm

Migrate from the former qwen variant

The unversioned qwen identifier is no longer accepted. Replace it with qwen2.5 to retain the previous Qwen2.5-VL behavior, or choose qwen3 explicitly. Update both preparation and generation so their keys continue to match.

1
# Before
2
CaptionPreparationStage(model_variant="qwen")
3
CaptionGenerationStage(model_variant="qwen")
4
5
# After: equivalent Qwen2.5 behavior
6
CaptionPreparationStage(model_variant="qwen2.5")
7
CaptionGenerationStage(model_variant="qwen2.5")

For the example script, change --captioning-algorithm qwen to --captioning-algorithm qwen2.5. If you enhance captions programmatically, also set captioning_model_variant="qwen2.5". Qwen video preprocessing is now always handled by the Hugging Face/vLLM processor; remove the former model_does_preprocess argument rather than replacing it.

Preview Generation

Generate lightweight .webp previews for each caption window to support review and QA workflows. A dedicated PreviewStage reads per-window mp4 bytes and encodes WebP using ffmpeg.

Preview Parameters

• target_fps (default 1.0): Target frames per second for preview generation.
• target_height (default 240): Output height. Width auto-scales to preserve aspect ratio.
• compression_level (range 0–6, default 6): WebP compression level. 0 is lossless; higher values reduce size with lower quality.
• quality (range 0–100, default 50): WebP quality. Higher values increase quality and size.
• num_cpus_per_worker (default 4.0): Number of CPU threads mapped to ffmpeg -threads.
• verbose (default False): Emit more logs.

Behavior notes:

• If the input frame rate is lower than target_fps or the input height is lower than target_height, the stage logs a warning and preview quality can degrade.
• If ffmpeg fails, the stage logs the error and skips assigning preview bytes for that window.

Example: Configure PreviewStage

1
from nemo_curator.stages.video.preview.preview import PreviewStage
2
3
preview = PreviewStage(
4
    target_fps=1.0,
5
    target_height=240,
6
    compression_level=6,
7
    quality=50,
8
    num_cpus_per_worker=4.0,
9
    verbose=False,
10
)

Outputs

The stage writes .webp files under the previews/ directory that ClipWriterStage manages. Use the helper to resolve the path:

1
from nemo_curator.stages.video.io.clip_writer import ClipWriterStage
2
previews_dir = ClipWriterStage.get_output_path_previews("/outputs")

Refer to Save & Export for directory structure and file locations: Save & Export.

Requirements and Troubleshooting

• ffmpeg with WebP (libwebp) support must be available in the environment.
• If you observe warnings about low frame rate or height, consider lowering target_fps or target_height to better match inputs.
• On encoding errors, check logs for the ffmpeg command and output to diagnose missing encoders.