nemo_gym.core.tool_output_images

Images that a tool returns next to its JSON reply.

A Resources Server tool route can list images under tool_output_images in its JSON reply. Each entry is a ToolOutputImage: a base64 data: URL of a PNG, JPEG, GIF or WebP image, checked as MediaFile checks prompt images, and an optional label, which the model gets as text before the image. simple_agent sends the rest of the reply as the tool output and each image in a user message after the turn’s tool outputs. The MCP endpoint returns the images as image content after the tool result’s text. An agent that does not read the key, such as simple_agent_with_compaction, passes the whole reply to the model as text, base64 data included.

Module Contents

Classes

Name	Description
`ToolOutputImage`	One image in a tool reply.

Functions

Name	Description
`split_tool_output_images`	Return a tool reply without its `tool_output_images`, and those images.

Data

TOOL_OUTPUT_IMAGES_KEY

_TOOL_OUTPUT_IMAGES

API

class nemo_gym.core.tool_output_images.ToolOutputImage()

Bases: BaseModel

One image in a tool reply.

data

str

The image’s base64 data, without the data: prefix.

image_url

str

label

Optional[str] = None

media_type

str

The image’s media type, such as image/png.

model_config

= ConfigDict(extra='forbid')

nemo_gym.core.tool_output_images.ToolOutputImage._decode_image(
    image_url: str
) -> str

classmethod

nemo_gym.core.tool_output_images.split_tool_output_images(
    payload: object
) -> tuple[object, list[nemo_gym.core.tool_output_images.ToolOutputImage]]

Return a tool reply without its tool_output_images, and those images.

A reply that is not a JSON object, or has no tool_output_images key, comes back unchanged with no images. A null image list means no images. A malformed image list raises pydantic.ValidationError: it is a fault of the tool, not of the model.

nemo_gym.core.tool_output_images.TOOL_OUTPUT_IMAGES_KEY = 'tool_output_images'

nemo_gym.core.tool_output_images._TOOL_OUTPUT_IMAGES = TypeAdapter(list[ToolOutputImage])