nemo_gym.core.media

Media carried in Responses messages, such as an input_image part holding a base64 data URI.

split_text_and_media decodes a message into its text and its media files, and MediaFile converts between bytes and data URIs. nemo_gym.sandbox.stage_media writes the decoded files into a sandbox for a harness CLI to read.

Images are the supported media today. Supporting another kind means adding its media types to _MEDIA_TYPE_EXTENSIONS and _MEDIA_TYPE_SIGNATURES and its Responses part type to _PART_URL_FIELDS, or a decoder for a part that does not hold a data URI.

Module Contents

Classes

Name	Description
`MediaFile`	Decoded media content and its media type, such as `image/png`.
`UnsupportedContentError`	A message holds content that cannot be passed on as text or as a media file.

Functions

Name	Description
`_media_type_of`	-
`_part_field`	-
`split_text_and_media`	Return a message’s text, joined across text parts, and its decoded media in order.

Data

_MEDIA_TYPE_ALIASES

_MEDIA_TYPE_EXTENSIONS

_MEDIA_TYPE_SIGNATURES

_PART_URL_FIELDS

_TEXT_PART_TYPES

API

class nemo_gym.core.media.MediaFile(
    media_type: str,
    data: bytes
)

Dataclass

Decoded media content and its media type, such as image/png.

The content always starts with the bytes that identify its media type.

data

bytes = field(repr=False)

extension

str

The file extension, without the dot, that files of this media type are written with.

media_type

str

nemo_gym.core.media.MediaFile.__post_init__() -> None

nemo_gym.core.media.MediaFile.from_data_uri(
    uri: str
) -> nemo_gym.core.media.MediaFile

classmethod

Decode a data:<media type>[;<parameter>]...;base64,<payload> URI.

The URI must name a supported media type, and the payload must hold data of a supported media type. When the two differ, as for a PNG labeled image/jpeg, the payload’s own type wins, so the file gets the extension its content needs.

nemo_gym.core.media.MediaFile.to_data_uri() -> str

Encode as a data:<media type>;base64,<payload> URI.

class nemo_gym.core.media.UnsupportedContentError()

Bases: ValueError

A message holds content that cannot be passed on as text or as a media file.

The caller should reject the request rather than drop the content.

nemo_gym.core.media._media_type_of(
    data: bytes
) -> str | None

nemo_gym.core.media._part_field(
    part: object,
    name: str
) -> object

nemo_gym.core.media.split_text_and_media(
    content: str | collections.abc.Iterable[object]
) -> tuple[str, list[nemo_gym.core.media.MediaFile]]

Return a message’s text, joined across text parts, and its decoded media in order.

Each part may be a mapping or a Responses content part model. Raises UnsupportedContentError for a part that is neither text nor supported media.

nemo_gym.core.media._MEDIA_TYPE_ALIASES = {'image/jpg': 'image/jpeg'}

nemo_gym.core.media._MEDIA_TYPE_EXTENSIONS = {'image/png': 'png', 'image/jpeg': 'jpg', 'image/gif': 'gif', 'image/webp': 'web...

nemo_gym.core.media._MEDIA_TYPE_SIGNATURES = {'image/png': re.compile(b'\\x89PNG\\r\\n\\x1a\\n.{4}IHDR', re.DOTALL), 'image/j...

nemo_gym.core.media._PART_URL_FIELDS = {'input_image': 'image_url'}

nemo_gym.core.media._TEXT_PART_TYPES = {'input_text', 'text'}