Skip to navigation

nemo_gym.sandbox.compose_config

View as Markdown

Turn a published Harbor Compose overlay into a document the sandbox Compose adapter can start.

Harbor publishes environment/docker-compose.yaml as an overlay on its own base file: the main service is the task image, sidecars are prebuilt images. Starting it without Docker needs each image’s recorded OCI configuration (entrypoint, command, user, exposed ports, healthcheck), which image_configs supplies, keyed by image reference.

Adapted from Gym’s Terminal Bench 4 server (resources_servers/terminal_bench_4/compose_config.py).

Module Contents

Functions

NameDescription
_compose_literals-
_no_shared_storageThe refusal’s reason: the block service_name runs on has no shared storage, and where that is set.
_peer_urlWhether value is exactly one URL whose host is a Compose service, the shape resolve_environment rewrites.
adapt_non_root_sidecarsKeep a sidecar on its image’s default non-root user without host-file injection.
check_sidecar_extra_hostsRefuse extra_hosts on a sidecar the group runs without host injection, naming the service.
deployment_platformThe platform a deployment runs, which a service’s platform must match: the sandbox provider options’
document_volume_sourcesThe entries of volume_sources a planned Compose document resolves through it, and nothing else.
non_root_sidecar_usersThe sidecars whose recorded image USER is not root, with that user.
opensandbox_shm_labelsOpenSandbox takes a service’s shared-memory size as a label. Mutates and returns document.
plan_compose_volumesDecide how each service’s volumes run in a sandbox group. Mutates and returns per-service uploads.
resolve_composeApply Harbor’s prebuilt-image base and normalize the published task overlay.
sandbox_platformThe (os, architecture) the sandboxes run: the OpenSandbox platform provider option

Data

DEFAULT_SANDBOX_PLATFORM

_GPU_BLOCK

_TASK_BLOCK

log

API

nemo_gym.sandbox.compose_config._compose_literals(
value: typing.Any
) -> typing.Any
nemo_gym.sandbox.compose_config._no_shared_storage(
block: str,
service_name: str
) -> str

The refusal’s reason: the block service_name runs on has no shared storage, and where that is set.

nemo_gym.sandbox.compose_config._peer_url(
value: typing.Any,
services: dict
) -> bool

Whether value is exactly one URL whose host is a Compose service, the shape resolve_environment rewrites.

A compound value (Kafka’s comma-separated KAFKA_ADVERTISED_LISTENERS) parses to a hostname too, but its port text is not a number and the adapter would fail reading it; it is left as written.

nemo_gym.sandbox.compose_config.adapt_non_root_sidecars(
document: dict,
image_configs: dict
) -> dict

Keep a sidecar on its image’s default non-root user without host-file injection.

A non-root process cannot edit /etc/hosts, so peer names are resolved into the environment variables that carry http://<service> URLs instead, and the explicit user copied from image metadata is dropped so the provider does not try to su to the user it already runs as. Such a sidecar cannot take extra_hosts (check_sidecar_extra_hosts raises first). Mutates and returns document.

nemo_gym.sandbox.compose_config.check_sidecar_extra_hosts(
document: dict,
image_configs: dict
) -> None

Refuse extra_hosts on a sidecar the group runs without host injection, naming the service.

A non-root sidecar gets no hosts file entries (see adapt_non_root_sidecars), so host entries it declares would be silently lost; the adapter rejects that document at seed. Prepare runs this check once the image configurations are recorded, so the task fails there, with the same reason.

nemo_gym.sandbox.compose_config.deployment_platform(
provider_options: collections.abc.Mapping[str, typing.Any] | None
) -> str

The platform a deployment runs, which a service’s platform must match: the sandbox provider options’ platform (OpenSandbox’s {os, arch}) when set, else linux/amd64, the one prepare records images for. Prepare and the server read it from the same sandbox_provider_options so the two checks agree: the os/arch form of sandbox_platform, the reader the image checks use.

nemo_gym.sandbox.compose_config.document_volume_sources(
document: collections.abc.Mapping[str, typing.Any],
volume_sources: collections.abc.Mapping[str, str]
) -> dict[str, str]

The entries of volume_sources a planned Compose document resolves through it, and nothing else.

The adapter treats every volume name it finds in its mapping as externally managed (one shared path, no per-group directory, no copy-up), so a deployment-wide mapping handed over whole would make an ordinary volumes: {pgdata: null} of one task share the directory another task maps as external. Only the names this document declares external: true and mounts, and the host paths of its remaining (foreign) bind mounts, are kept; a key the document never mounts, or mounts as an ordinary named volume, is dropped, and that volume stays private to the group, as Harbor’s per-trial Compose project keeps it.

nemo_gym.sandbox.compose_config.non_root_sidecar_users(
document: dict,
image_configs: dict
) -> dict[str, str]

The sidecars whose recorded image USER is not root, with that user.

These are the services adapt_non_root_sidecars keeps on their own user without host injection; prepare uses the same predicate so what it accepts is what the group runs. image_configs is read by reference and by pinned image, so the published document and the resolved one both work.

nemo_gym.sandbox.compose_config.opensandbox_shm_labels(
document: dict,
label: str = 'nemo.nvidia.com/shm'
) -> dict

OpenSandbox takes a service’s shared-memory size as a label. Mutates and returns document.

nemo_gym.sandbox.compose_config.plan_compose_volumes(
document: dict,
base_dir: pathlib.Path,
root: pathlib.Path | None = None,
volume_sources: dict[str, str] | None = None,
gpu_volume_sources: dict[str, str] | None = None,
shared_storage: bool | None = None,
gpu_shared_storage: bool | None = None,
warn_read_only: bool = False
) -> dict[str, list[dict[str, str]]]

Decide how each service’s volumes run in a sandbox group. Mutates and returns per-service uploads.

A bind mount whose source is a file or folder of the task (under root, default base_dir; relative paths resolve against base_dir, as Compose resolves them against the Compose file) becomes an upload into the service once it starts: the entry leaves volumes and joins x-sandbox.uploads as {"source": <host path>, "target": <container path>}. With a task root given, such a source must be one the task digest covers (is_covered: under environment/, tests/, solution/ or steps/, or one of the top-level task files, and not a name the digest skips), or it raises naming the path: an upload the digest does not see could change without changing the task’s identity. Named volumes declared at top level stay, in long syntax, for the provider’s shared storage; a volume declared external: true, and a bind source outside the task, resolve through volume_sources (the sandbox block’s mapping of volume name or host path to a shared-storage path) and raise ComposeVolumeError naming the volume when it is missing. A sidecar that reserves GPUs through deploy.resources resolves through gpu_volume_sources instead when that is given (the gpu_sandbox_provider block’s mapping, the block the server runs it on; main follows the task, as check_compose_fields routes the capability check). A bind of Harbor’s /logs tree onto main leaves volumes and joins x-sandbox.logs as {"source": <logs path>, "target": <container path>}: the adapter creates the source in the agent’s sandbox before the service’s command, hooks and healthcheck, and links a target elsewhere to it, as Docker creates both of Harbor’s bind targets at container create. A sidecar that binds a path of the tree shares it with main the way Harbor’s one host folder is shared: the path becomes a named volume on the provider’s shared storage (declared as nemo-gym-<path>, with nocopy; a second path that spells the same name, /logs/a/b beside /logs/a-b, gets a digest of its spelling appended), mounted at the sidecar’s target and at the path itself in main, so what the sidecar writes there is what the verifier collects. A logs path with a .. segment (/logs/../etc) raises naming the path, on main or a sidecar: it is used as written. shared_storage says whether the deployment’s provider has shared storage (gpu_shared_storage for the services gpu_volume_sources routes): False refuses every named volume, mapped source and shared logs bind at prepare, naming the setting, instead of at every seed; None leaves that to the adapter. A source the task does not contain, or a symlink that leaves it, raises naming the path; so does a bound folder that contains a link, and a source reached through a link with an absolute target (see absolute_link). A service that already carries x-sandbox raises naming the field: that key is this planner’s output, not task input. A null top-level declaration (volumes: {pgdata: null}) is normalised to {}, the form the adapter reads.

nemo_gym.sandbox.compose_config.resolve_compose(
document: dict,
main_image: str,
image_configs: dict,
platform: tuple[str, str] = DEFAULT_SANDBOX_PLATFORM
) -> dict

Apply Harbor’s prebuilt-image base and normalize the published task overlay.

platform is the (os, architecture) the sandboxes run (sandbox_platform); a service whose recorded image is for another platform is refused, naming both.

nemo_gym.sandbox.compose_config.sandbox_platform(
provider_options: collections.abc.Mapping[str, typing.Any] | None
) -> tuple[str, str]

The (os, architecture) the sandboxes run: the OpenSandbox platform provider option ({os, arch}, see nemo_gym.sandbox.providers.opensandbox) when a deployment sets one, otherwise linux/amd64, the default of every provider.

nemo_gym.sandbox.compose_config.DEFAULT_SANDBOX_PLATFORM = ('linux', 'amd64')
nemo_gym.sandbox.compose_config._GPU_BLOCK = '`gpu_sandbox_provider`'
nemo_gym.sandbox.compose_config._TASK_BLOCK = '`sandbox:`'
nemo_gym.sandbox.compose_config.log = logging.getLogger(__name__)