nemo_rl.environments.nemo_gym_multimodal

Module Contents

Functions

_encode_single_image_source	Resolve, encode, and close one local image source.
normalize_media_in_examples	Replace local media paths in NeMo Gym examples with data URLs.
_looks_like_image_src	True when src plausibly points at an image the loader can open.
_extract_input_images_from_message	Pull PIL images out of a non-assistant Responses-API item.
_is_trainable_output_item	Report whether an output item becomes a trainable assistant turn.
_index_per_turn_images	Bin server-returned images by the trainable turn that saw them.
_without_initial_media_sources	Copy Responses messages and remove one ordered copy of initial images and videos.
_require_video_config_value
_resolve_local_video_path	Resolve a local video source and reject unsupported remote schemes.
_extract_static_video_messages	Convert one-video Responses input into HF multimodal chat messages.
_inject_vllm_mm_processor_kwargs
_remove_vllm_mm_processor_kwargs
_replace_cached_video_frames_with_native_video	Replace cached image parts with one lossless native-video manifest.
_strip_local_media_metadata
_compute_dynamic_prompt_length
_video_to_image_content
_make_overlength_filtered_video_example
nemo_gym_example_to_video_datum_spec	Preprocess static Gym video with vLLM-equivalent frame sampling.

Data

_NEMO_GYM_IMAGE_ENCODE_MAX_WORKERS
_IMAGE_SRC_PREFIXES
_VideoConfigValue
_LOCAL_VIDEO_METADATA_KEYS

API

nemo_rl.environments.nemo_gym_multimodal._NEMO_GYM_IMAGE_ENCODE_MAX_WORKERS

8

nemo_rl.environments.nemo_gym_multimodal._encode_single_image_source(source: str) → str

Resolve, encode, and close one local image source.

nemo_rl.environments.nemo_gym_multimodal.normalize_media_in_examples(

nemo_gym_examples: list[dict],

) → list[dict]

Replace local media paths in NeMo Gym examples with data URLs.

nemo_rl.environments.nemo_gym_multimodal._IMAGE_SRC_PREFIXES

(‘data:image/’, ‘http://’, ‘https://’, ‘file://’)

nemo_rl.environments.nemo_gym_multimodal._looks_like_image_src(src: str) → bool

True when src plausibly points at an image the loader can open.

Guards against tool responses (e.g. {"x": 0.65, "y": 0.83} from a click tool) that are strings but not image URLs. Without this, the indexer forwards the JSON payload to resolve_to_image → PIL.open, which treats it as a filesystem path and raises FileNotFoundError.

nemo_rl.environments.nemo_gym_multimodal._extract_input_images_from_message(

item: dict,

) → list[PIL.Image.Image]

Pull PIL images out of a non-assistant Responses-API item.

Handles both content-list items (user / tool messages carrying input_image/image/image_url parts) and function_call_output items whose output field is an image data URL. Tool outputs that are non-image strings (e.g. structured JSON returned by tools like click(x, y)) contribute zero images to the bucket.

nemo_rl.environments.nemo_gym_multimodal._is_trainable_output_item(item: dict) → bool

Report whether an output item becomes a trainable assistant turn.

The postprocess loop skips items whose generation_token_ids is missing or empty, so per-turn image binning has to use the same predicate or the two walks disagree and every later turn gets the wrong images.

nemo_rl.environments.nemo_gym_multimodal._index_per_turn_images(

output: list[dict],

input_messages: list[dict] | None = None,

) → list[list[PIL.Image.Image]]

Bin server-returned images by the trainable turn that saw them.

Walks the Responses-API items in order and flushes pending into a per-turn bucket each time it hits an item carrying truthy generation_token_ids — matching the exact gate that _postprocess_nemo_gym_to_nemo_rl_result uses to decide which items become trainable turns. Every other item (user turns, tool messages, function_call_output, non-trainable reasoning) contributes its images to pending for the next trainable turn. This ensures the returned list has one entry per trainable turn, aligned with the postprocess loop’s turn_idx even when the trainable item’s role is not assistant (e.g. a reasoning-only response, or a function_call).

input_messages is the initial responses_create_params.input list — images there (e.g. a single-shot user prompt for tool-based envs like circle-click) are consumed by the first trainable turn’s tokenized prompt and must land in the first bucket. Agents like gym_v_agent that keep input empty and inject observations as function_call_output items are unaffected — the seed is a no-op when input_messages is empty.

nemo_rl.environments.nemo_gym_multimodal._without_initial_media_sources(

messages: Any,

initial_sources: list[Any],

) → tuple[Any, bool]

Copy Responses messages and remove one ordered copy of initial images and videos.

nemo_rl.environments.nemo_gym_multimodal._VideoConfigValue

‘TypeVar(…)’

nemo_rl.environments.nemo_gym_multimodal._LOCAL_VIDEO_METADATA_KEYS

‘frozenset(…)’

nemo_rl.environments.nemo_gym_multimodal._require_video_config_value(

value: nemo_rl.environments.nemo_gym_multimodal._VideoConfigValue | None,

field_name: str,

) → nemo_rl.environments.nemo_gym_multimodal._VideoConfigValue

nemo_rl.environments.nemo_gym_multimodal._resolve_local_video_path(source: str) → str

Resolve a local video source and reject unsupported remote schemes.

nemo_rl.environments.nemo_gym_multimodal._extract_static_video_messages(

nemo_gym_example: dict[str, Any],

) → tuple[list[dict[str, Any]], str | None] | None

Convert one-video Responses input into HF multimodal chat messages.

A video may be represented either by one native video content part or by a sequence of cached input_image parts carrying _is_video_frame. The latter is the on-disk frame-cache format used by the video Gym recipes.

nemo_rl.environments.nemo_gym_multimodal._inject_vllm_mm_processor_kwargs(

nemo_gym_example: dict[str, Any],

mm_processor_kwargs: dict[str, Any],

) → None

nemo_rl.environments.nemo_gym_multimodal._remove_vllm_mm_processor_kwargs(

nemo_gym_example: dict[str, Any],

names: set[str],

) → None

nemo_rl.environments.nemo_gym_multimodal._replace_cached_video_frames_with_native_video(

nemo_gym_example: dict[str, Any],

) → None

Replace cached image parts with one lossless native-video manifest.

nemo_rl.environments.nemo_gym_multimodal._strip_local_media_metadata(

nemo_gym_example: dict[str, Any],

) → None

nemo_rl.environments.nemo_gym_multimodal._compute_dynamic_prompt_length(

processor: Any,

messages: list[dict[str, Any]],

template_kwargs: dict[str, Any],

) → int | None

nemo_rl.environments.nemo_gym_multimodal._video_to_image_content(

video_path: str,

*,

num_frames: int,

temporal_patch_size: int,

sampling_style: nemo_rl.models.generation.vllm.video_utils.VideoSamplingStyle,

) → tuple[list[dict[str, Any]], dict[str, Any]]

nemo_rl.environments.nemo_gym_multimodal._make_overlength_filtered_video_example(

nemo_gym_example: dict[str, Any],

) → dict[str, Any]

nemo_rl.environments.nemo_gym_multimodal.nemo_gym_example_to_video_datum_spec(

nemo_gym_example: dict[str, Any],

*,

processor: Any,

max_seq_length: int | None,

idx: int,

task_name: str,

data_config: nemo_rl.data.interfaces.TaskDataSpec | None = None,

) → dict[str, Any] | None

Preprocess static Gym video with vLLM-equivalent frame sampling.

The raw video remains in the outbound Gym request. Cached frames are sent as one native-video manifest so vLLM consumes the same lossless RGB frames as policy preprocessing. Those tensors are reattached to vLLM-authored prompt token IDs after the rollout.