> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server. > Generate clip captions with Qwen-VL or Nemotron VLMs and optional preview images # Captions and Preview Prepare inputs, generate window-level captions, optionally refine or enhance them, and produce preview images. ## Choose a captioning model Use the same `model_variant` in `CaptionPreparationStage` and `CaptionGenerationStage`. The preparation stage formats prompts and video frames for that exact checkpoint; the generation stage looks up the prepared input by the variant name. | Variant | Hugging Face model | Precision and notes | | --------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------ | | `qwen2.5` | `Qwen/Qwen2.5-VL-7B-Instruct` | Default; BF16 unless `fp8=True` or `--captioning-use-fp8-weights` is set | | `qwen3` | `Qwen/Qwen3-VL-8B-Instruct` | Qwen3-VL; BF16 unless FP8 is enabled | | `nemotron`, `nemotron-bf16` | `nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16` | `nemotron` is an alias for the BF16 checkpoint | | `nemotron-fp8` | `nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-FP8` | Pre-quantized FP8 checkpoint | | `nemotron-nvfp4` | `nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD` | NVFP4 quantization-aware-distilled checkpoint | | `nemotron-3-nano-omni` | `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning` | Nemotron 3 Nano Omni; currently initialized with tensor parallel size 1 | All checkpoints are downloaded from Hugging Face on each node when absent from `model_dir`. Caption generation requires vLLM and reserves one GPU per worker. For Qwen, use the `fp8` parameter or `--captioning-use-fp8-weights`; for Nemotron Nano 12B v2, select the precision-specific variant instead. Nemotron 3 Nano Omni uses one video per prompt, a 131,072-token model context, and vLLM compatibility patches that NeMo Curator applies automatically to the downloaded checkpoint. ## Quickstart #### Pipeline stages ```python from nemo_curator.pipeline import Pipeline from nemo_curator.stages.video.caption.caption_generation import CaptionGenerationStage from nemo_curator.stages.video.caption.caption_preparation import CaptionPreparationStage from nemo_curator.stages.video.preview.preview import PreviewStage model_variant = "qwen2.5" pipe = Pipeline(name="captions_preview") pipe.add_stage( CaptionPreparationStage( model_variant=model_variant, prompt_variant="default", sampling_fps=2.0, window_size=256, remainder_threshold=128, preprocess_dtype="float16", generate_previews=True, ) ) pipe.add_stage(PreviewStage(target_fps=1.0, target_height=240)) pipe.add_stage( CaptionGenerationStage( model_dir="/models", model_variant=model_variant, caption_batch_size=8, max_output_tokens=512, ) ) pipe.run() ``` #### Example script ```bash python tutorials/video/getting-started/video_split_clip_example.py \ ... \ --generate-captions \ --captioning-algorithm qwen2.5 \ --captioning-window-size 256 \ --captioning-remainder-threshold 128 \ --captioning-sampling-fps 2.0 \ --captioning-preprocess-dtype float16 \ --captioning-batch-size 8 \ --captioning-max-output-tokens 512 \ --generate-previews \ --preview-target-fps 1.0 \ --preview-target-height 240 ``` `--captioning-algorithm` accepts `qwen2.5`, `qwen3`, `nemotron`, `nemotron-bf16`, `nemotron-fp8`, `nemotron-nvfp4`, and `nemotron-3-nano-omni`. ## Prepare caption inputs `CaptionPreparationStage` splits each clip into windows, samples frames, and writes the model-ready value to `window.llm_inputs[model_variant]`. If `generate_previews=True`, it also keeps MP4 bytes for `PreviewStage`. ```python from nemo_curator.stages.video.caption.caption_preparation import CaptionPreparationStage prep = CaptionPreparationStage( model_variant="qwen3", prompt_variant="default", sampling_fps=2.0, window_size=256, remainder_threshold=128, preprocess_dtype="float16", generate_previews=True, ) ``` | Parameter | Type | Default | Description | | --------------------- | ---------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model_variant` | str | `"qwen2.5"` | One of the seven variants in the model table. Must match the generation stage. | | `prompt_variant` | `"default"`, `"av"`, `"av-surveillance"` | `"default"` | Built-in caption prompt used when `prompt_text` is not set. | | `prompt_text` | str \| None | `None` | Custom prompt that overrides `prompt_variant`. | | `sampling_fps` | float | 2.0 | Frames per second sampled from the source clip. | | `window_size` | int | 256 | Number of sampled frames in each caption window. | | `remainder_threshold` | int | 128 | Minimum remaining frames required to create a final shorter window. | | `preprocess_dtype` | str | `"float32"` | Raw-frame dtype. The Python stage defaults to `float32`; the example CLI accepts `float32`, `float16`, `bfloat16`, or `uint8` and overrides the default to `float16`. | | `generate_previews` | bool | `True` | Preserve per-window MP4 bytes for preview generation. | | `verbose` | bool | `False` | Emit additional logs. | ## Generate captions `CaptionGenerationStage` consumes `window.llm_inputs[model_variant]` and writes its result to `window.caption[model_variant]`. After generation it removes that prepared input and the window MP4 bytes. ```python from nemo_curator.stages.video.caption.caption_generation import CaptionGenerationStage gen = CaptionGenerationStage( model_dir="/models", model_variant="qwen3", caption_batch_size=8, fp8=False, max_output_tokens=512, disable_mmcache=False, generate_stage2_caption=False, ) ``` | Parameter | Type | Default | Description | | ------------------------- | ----------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model_dir` | str | `"models/qwen"` | Base directory under which checkpoint-specific directories are created. | | `model_variant` | str | `"qwen2.5"` | One of the seven variants in the model table. | | `caption_batch_size` | int | 16 | Generation batch size. The example CLI defaults to 8. | | `fp8` | bool | `False` | Quantize Qwen weights to FP8. Select `nemotron-fp8` for the Nemotron FP8 checkpoint. | | `max_output_tokens` | int | 512 | Maximum tokens generated for each caption. | | `disable_mmcache` | bool | `False` | Disable the vLLM multimodal cache for Qwen. The example CLI disables it unless `--captioning-use-vllm-mmcache` is set. | | `vllm_kwargs` | dict | `{}` | Additional keyword arguments forwarded only to the Qwen vLLM constructor; Nemotron variants do not use this field. | | `generate_stage2_caption` | bool | `False` | Run the same VLM a second time to refine its initial caption. | | `stage2_prompt_text` | str \| None | `None` | Custom prefix forwarded to Nemotron variants for same-VLM refinement. `CaptionGenerationStage` does not forward this field to Qwen variants, which always use `Please refine this caption:`. | | `verbose` | bool | `False` | Emit additional generation logs. | ## Refine or enhance captions NeMo Curator provides two distinct second-pass workflows: * **Same-VLM refinement** reruns the selected captioning model with its first caption. Set `generate_stage2_caption=True`, or pass `--captioning-stage2-caption`. The refined text replaces the first-pass value in `window.caption[model_variant]`. * **Text-only enhancement** runs a separate Qwen language model after caption generation. Add `CaptionEnhancementStage`, or pass `--enhance-captions`. The result is stored in `window.enhanced_caption["qwen_lm"]` and the original caption remains available. ```python from nemo_curator.stages.video.caption.caption_enhancement import CaptionEnhancementStage enhance = CaptionEnhancementStage( model_dir="/models", model_variant="qwen3", captioning_model_variant="qwen3", prompt_variant="default", model_batch_size=128, fp8=False, max_output_tokens=512, ) ``` The enhancement model can be `qwen2.5` (`Qwen/Qwen2.5-14B-Instruct`) or `qwen3` (`Qwen/Qwen3-14B`). `captioning_model_variant` must identify the caption key produced by the earlier generation stage; it does not have to match the enhancement model. In the example script, `--enhance-captions-algorithm` selects `qwen2.5` or `qwen3`. The separate `--enhanced-caption-models` output selector currently accepts only `qwen_lm`. ```bash python tutorials/video/getting-started/video_split_clip_example.py \ ... \ --generate-captions \ --captioning-algorithm nemotron-fp8 \ --enhance-captions \ --enhance-captions-algorithm qwen3 \ --enhanced-caption-models qwen_lm ``` | Parameter | Type | Default | Description | | -------------------------- | -------------------------------- | --------------- | -------------------------------------------------------------------------- | | `model_dir` | str | `"models/qwen"` | Base directory for Qwen language-model weights. | | `model_variant` | `"qwen2.5"`, `"qwen3"` | `"qwen2.5"` | Text-only model used for enhancement. | | `captioning_model_variant` | str | `"qwen2.5"` | Key in `window.caption` to read. Set it to the earlier captioning variant. | | `prompt_variant` | `"default"`, `"av-surveillance"` | `"default"` | Built-in enhancement prompt used when `prompt_text` is not set. | | `prompt_text` | str \| None | `None` | Custom enhancement system prompt. | | `model_batch_size` | int | 128 | Enhancement generation batch size. | | `fp8` | bool | `False` | Quantize the Qwen language-model weights to FP8. | | `vllm_kwargs` | dict | `{}` | Additional keyword arguments forwarded to vLLM. | | `max_output_tokens` | int | 512 | Maximum tokens generated for each enhanced caption. | | `verbose` | bool | `False` | Emit additional logs. | ## Migrate from the former `qwen` variant The unversioned `qwen` identifier is no longer accepted. Replace it with `qwen2.5` to retain the previous Qwen2.5-VL behavior, or choose `qwen3` explicitly. Update both preparation and generation so their keys continue to match. ```python # Before CaptionPreparationStage(model_variant="qwen") CaptionGenerationStage(model_variant="qwen") # After: equivalent Qwen2.5 behavior CaptionPreparationStage(model_variant="qwen2.5") CaptionGenerationStage(model_variant="qwen2.5") ``` For the example script, change `--captioning-algorithm qwen` to `--captioning-algorithm qwen2.5`. If you enhance captions programmatically, also set `captioning_model_variant="qwen2.5"`. Qwen video preprocessing is now always handled by the Hugging Face/vLLM processor; remove the former `model_does_preprocess` argument rather than replacing it. ## Preview Generation Generate lightweight `.webp` previews for each caption window to support review and QA workflows. A dedicated `PreviewStage` reads per-window `mp4` bytes and encodes WebP using `ffmpeg`. ### Preview Parameters * `target_fps` (default `1.0`): Target frames per second for preview generation. * `target_height` (default `240`): Output height. Width auto-scales to preserve aspect ratio. * `compression_level` (range `0–6`, default `6`): WebP compression level. `0` is lossless; higher values reduce size with lower quality. * `quality` (range `0–100`, default `50`): WebP quality. Higher values increase quality and size. * `num_cpus_per_worker` (default `4.0`): Number of CPU threads mapped to `ffmpeg -threads`. * `verbose` (default `False`): Emit more logs. Behavior notes: * If the input frame rate is lower than `target_fps` or the input height is lower than `target_height`, the stage logs a warning and preview quality can degrade. * If `ffmpeg` fails, the stage logs the error and skips assigning preview bytes for that window. ### Example: Configure PreviewStage ```python from nemo_curator.stages.video.preview.preview import PreviewStage preview = PreviewStage( target_fps=1.0, target_height=240, compression_level=6, quality=50, num_cpus_per_worker=4.0, verbose=False, ) ``` ### Outputs The stage writes `.webp` files under the `previews/` directory that `ClipWriterStage` manages. Use the helper to resolve the path: ```python from nemo_curator.stages.video.io.clip_writer import ClipWriterStage previews_dir = ClipWriterStage.get_output_path_previews("/outputs") ``` Refer to Save & Export for directory structure and file locations: [Save & Export](/curate-video/save-export). ### Requirements and Troubleshooting * `ffmpeg` with WebP (`libwebp`) support must be available in the environment. * If you observe warnings about low frame rate or height, consider lowering `target_fps` or `target_height` to better match inputs. * On encoding errors, check logs for the `ffmpeg` command and output to diagnose missing encoders.