nemo_rl.data.datasets.response_datasets.intent#

IntentDataset: HumanOmniV2 IntentTrain / IntentBench loader for GRPO.

Loads the PhilipC/IntentTrain (training) or PhilipC/IntentBench (validation) datasets that ship as a JSON manifest plus a videos.zip archive on HuggingFace, filters samples to the configured problem_type allow-list, and emits OpenAI-style messages whose user content carries both a video reference and the audio track extracted from that same video. Audio and video flow as two independent {type:audio} / {type:video} content items so the Qwen2.5-Omni chat template renders both <|VIDEO|> and <|AUDIO|> placeholders into the prompt – vLLM’s multimodal prompt replacement on the rollout side requires those placeholders to exist before it accepts matching mm_items. The use_audio_in_video=True time-alignment hint is NOT threaded through here because the installed transformers + vLLM stack rejected that path during Round 1 testing (see BitLesson BL-20260428-omni-use-audio-in-video).

Module Contents#

Classes#

IntentDataset

HumanOmniV2 IntentTrain / IntentBench loader for VLM GRPO.

IntentTrainDataset

Convenience wrapper that pins split="train" for IntentTrain.

IntentBenchDataset

Convenience wrapper that pins split="validation" for IntentBench.

Functions#

_format_options

Render a record’s multiple-choice options into the prompt text.

_extract_videos_zip_once

Idempotently extract videos.zip inside snapshot_dir.

_resolve_video_path

Resolve a manifest’s relative video path to an absolute file on disk.

_read_manifest

Data#

logger

_TYPE_TEMPLATE

_SPLIT_CONFIG

_EXTRACTION_SENTINEL

API#

nemo_rl.data.datasets.response_datasets.intent.logger#

‘getLogger(…)’

nemo_rl.data.datasets.response_datasets.intent._TYPE_TEMPLATE#

None

nemo_rl.data.datasets.response_datasets.intent._format_options(options: Any) → str#

Render a record’s multiple-choice options into the prompt text.

IntentTrain/IntentBench manifests store options as a list of strings like ["A.first choice", "B.second choice", ...] (occasionally as a string repr of that list). These MUST be appended to the prompt: without them the model only sees the question stem and has to emit a bare option letter blind (capping accuracy near chance). Mirrors HumanOmniV2’s prompt construction. Returns an empty string when no options are present.

nemo_rl.data.datasets.response_datasets.intent._SPLIT_CONFIG#

None

nemo_rl.data.datasets.response_datasets.intent._EXTRACTION_SENTINEL#

‘.intent_videos_extracted’

nemo_rl.data.datasets.response_datasets.intent._extract_videos_zip_once(snapshot_dir: str) → str#

Idempotently extract videos.zip inside snapshot_dir.

Returns the directory the archive was extracted into. A sentinel file is written after a successful extraction so subsequent constructions skip re-extraction.

nemo_rl.data.datasets.response_datasets.intent._resolve_video_path(snapshot_dir: str, relpath: str) → str | None#

Resolve a manifest’s relative video path to an absolute file on disk.

The IntentTrain/IntentBench archives extract their contents either directly under the snapshot directory or under a videos/ subdirectory. Try both and return the first path that exists, or None if neither does.

nemo_rl.data.datasets.response_datasets.intent._read_manifest(
snapshot_dir: str,
manifest_filename: str,
) → list[dict[str, Any]]#
class nemo_rl.data.datasets.response_datasets.intent.IntentDataset(
split: str = 'train',
allowed_problem_types: list[str] | None = None,
max_samples: int | None = None,
**kwargs: Any,
)#

Bases: nemo_rl.data.datasets.raw_dataset.RawDataset

HumanOmniV2 IntentTrain / IntentBench loader for VLM GRPO.

Each sample emits both a video file path and a 16 kHz mono audio array decoded from that same file as two independent content items ({type:video} and {type:audio}) plus a text prompt. The Qwen2.5-Omni processor and vLLM rollout both treat the two streams as independent multimodal sources; the explicit time-alignment via use_audio_in_video=True is intentionally not used in v1 because the installed transformers + vLLM stack rejected that path. Samples whose problem_type is not in allowed_problem_types are dropped before iteration.

Parameters:

  • split – "train" (PhilipC/IntentTrain) or "validation" (PhilipC/IntentBench).

  • allowed_problem_types – List of problem_type values to retain. Defaults to ["multiple choice"] per DEC-2.

  • max_samples – Optional cap on the number of samples after filtering. Useful for smoke runs.

Initialization

_download_and_extract() → str#

Download the HF dataset snapshot and extract videos.zip once.

_load_records() → list[dict[str, Any]]#
_filter_records(
records: list[dict[str, Any]],
) → list[dict[str, Any]]#
format_data(data: dict[str, Any]) → dict[str, Any]#

Format a manifest record into NeMo-RL OpenAI-style messages.

Each yielded sample carries the video file path AND the audio track decoded from that same file at 16 kHz mono. Both arrive as independent {type: video} / {type: audio} content items so the Qwen2.5-Omni chat template renders both <|VIDEO|> and <|AUDIO|> placeholders in the prompt; vLLM’s multimodal prompt replacement on the rollout side requires those placeholders to exist in the prompt before it will accept matching mm_items.

We deliberately do NOT pass use_audio_in_video=True to the processor in v1: that flag would entangle the audio and video placeholder accounting in ways the current installed transformers

  • vLLM stack does not handle (see Round 1 BitLesson). The model still receives both modalities; the only thing missing is the explicit time alignment hint.

class nemo_rl.data.datasets.response_datasets.intent.IntentTrainDataset(**kwargs: Any)#

Bases: nemo_rl.data.datasets.response_datasets.intent.IntentDataset

Convenience wrapper that pins split="train" for IntentTrain.

Initialization

class nemo_rl.data.datasets.response_datasets.intent.IntentBenchDataset(**kwargs: Any)#

Bases: nemo_rl.data.datasets.response_datasets.intent.IntentDataset

Convenience wrapper that pins split="validation" for IntentBench.

Initialization