bridge.models.nemotron_vl.nemotron_vl_utils#

Module Contents#

Functions#

encode_pil_to_jpeg_data_url

Encode a PIL image to a base64-encoded data URL.

sample_video_frames_to_data_urls

Sample frames from a video and return base64-encoded data URLs along with metadata.

maybe_path_or_url_to_data_urls

Convert a path or URL to data URLs, handling videos, images, and remote files.

pil_image_from_base64

Decode a base64-encoded image to a PIL image.

adjust_image_tokens

Adjust each image placeholder without merging samples in the batch.

Data#

logger

_ALLOW_PRIVATE_URL_FETCH_ENV

_is_safe_public_http_url

_safe_url_open

API#

bridge.models.nemotron_vl.nemotron_vl_utils.logger#

‘getLogger(…)’

bridge.models.nemotron_vl.nemotron_vl_utils._ALLOW_PRIVATE_URL_FETCH_ENV#

None

bridge.models.nemotron_vl.nemotron_vl_utils._is_safe_public_http_url#

None

bridge.models.nemotron_vl.nemotron_vl_utils._safe_url_open#

None

bridge.models.nemotron_vl.nemotron_vl_utils.encode_pil_to_jpeg_data_url(pil_image)#

Encode a PIL image to a base64-encoded data URL.

bridge.models.nemotron_vl.nemotron_vl_utils.sample_video_frames_to_data_urls(
video_path_local,
fps=1,
nframe=0,
nframe_max=-1,
)#

Sample frames from a video and return base64-encoded data URLs along with metadata.

Parameters:

  • video_path_local – Path to the video file

  • fps – Target frames per second for sampling (if > 0, uses fps-based sampling)

  • nframe – Number of frames to sample (used if fps <= 0)

  • nframe_max – Maximum number of frames to sample

Returns:

(frame_data_urls, metadata)

  • frame_data_urls: List of base64-encoded frame images

  • metadata: VideoMetadata dataclass containing info about the sampled frames:

    • total_num_frames: Number of sampled frames

    • fps: Effective frame rate of the sampled frames

    • duration: Duration covered by the sampled frames (in seconds)

    • video_backend: Backend used for video processing (‘decord’)

Return type:

tuple

bridge.models.nemotron_vl.nemotron_vl_utils.maybe_path_or_url_to_data_urls(
path_or_url,
fps=1,
nframe=0,
nframe_max=-1,
)#

Convert a path or URL to data URLs, handling videos, images, and remote files.

Parameters:

  • path_or_url – Path or URL to the media file

  • fps – Target frames per second for video sampling (if > 0, uses fps-based sampling)

  • nframe – Number of frames to sample from video (used if fps <= 0)

  • nframe_max – Maximum number of frames to sample

Returns:

(data_urls, metadata)

  • data_urls: List of base64-encoded data URLs

  • metadata: VideoMetadata dataclass with video metadata or None for images

Return type:

tuple

bridge.models.nemotron_vl.nemotron_vl_utils.pil_image_from_base64(b64_str: str) → PIL.Image.Image#

Decode a base64-encoded image to a PIL image.

bridge.models.nemotron_vl.nemotron_vl_utils.adjust_image_tokens(
input_ids: torch.Tensor | dict[str, torch.Tensor],
num_tiles: int | list[int] | torch.Tensor,
img_start_token_id: int,
img_end_token_id: int,
*,
padding_values: collections.abc.Mapping[str, int | float] | None = None,
) → torch.Tensor | dict[str, torch.Tensor]#

Adjust each image placeholder without merging samples in the batch.

num_tiles contains one desired media-token count per image, flattened in row-major sample order. A text-only row consumes no entry, while a row with multiple images consumes one entry for each <img>...</img> region. Every tensor in an input dictionary is adjusted at the same positions and rows are right-padded back to a common length. If the dictionary contains attention_mask, trailing masked positions are discarded before the adjustment so stale processor padding cannot pin the adjusted batch width.

.. rubric:: Example

input_ids decoded may look like this System: … User:… Image 1: … # adjust number of tokens to be num_tiles[0] Image 2: … # adjust number of tokens to be num_tiles[1] … etc

Parameters:

  • input_ids – A 2D token tensor, or a dictionary containing input_ids and other sequence-aligned 2D tensors with the same shape.

  • num_tiles – Desired media-token counts for all images in row-major order.

  • img_start_token_id – Token ID for <img>.

  • img_end_token_id – Token ID for </img>.

  • padding_values – Per-tensor values used when adjusted rows have different lengths. Unspecified tensors default to zero.

Returns:

The adjusted tensor or dictionary, preserving the input batch size.

Raises:

ValueError – If tensors are not aligned, image boundaries are malformed, or the number of tile counts does not match the number of images.