Skip to navigation
API ReferenceFull Library ReferenceNemo GymNemo GymArtifactsExporters

nemo_gym.artifacts.exporters.isolated

View as Markdown

Module Contents

Classes

NameDescription
ExportPreparationErrorThe run cannot be handed to an isolated exporter’s worker; the message says why.
ExporterEnvironmentErrorThe virtual environment an isolated exporter needs could not be built.
IsolatedExporterAn exporter whose backend work runs in its own virtual environment.

Functions

NameDescription
_worker_environmentThis process’s environment minus what would point the worker’s Python back at Gym’s packages.
ensure_exporter_environmentBuild the exporter’s virtual environment if it is not ready, and return its Python.
exporter_venv_path<venv_root>/exporters/<name>/<digest of requirements>/.venv.

Data

WORKER_HARNESS_FPATH

API

class nemo_gym.artifacts.exporters.isolated.ExportPreparationError()

Bases: RuntimeError

The run cannot be handed to an isolated exporter’s worker; the message says why.

class nemo_gym.artifacts.exporters.isolated.ExporterEnvironmentError()

Bases: RuntimeError

The virtual environment an isolated exporter needs could not be built.

class nemo_gym.artifacts.exporters.isolated.IsolatedExporter(
    config: nemo_gym.config.config_types.RunExportConfig,
    venv_root: pathlib.Path
)

Bases: Exporter

An exporter whose backend work runs in its own virtual environment.

The SDK and its dependencies never enter Gym’s environment, and an SDK that updates itself cannot disturb it. The main process validates the config and prepares what the backend needs (build_payload); the worker, a standalone file run with the environment’s Python, does the backend call.

The payload goes to the worker as JSON on stdin and may carry credentials, which therefore stay out of the process list. The worker file defines run(payload) -> dict and must not import nemo_gym.

requirements
tuple[str, ...] = ()
venv_root
= Path(venv_root).resolve()
worker_fpath
Path
nemo_gym.artifacts.exporters.isolated.IsolatedExporter._run_worker(
    python: pathlib.Path,
    payload: dict[str, typing.Any],
    workdir: pathlib.Path
) -> nemo_gym.artifacts.exporters.base.ExportResult
nemo_gym.artifacts.exporters.isolated.IsolatedExporter.build_payload(
    artifacts: nemo_gym.artifacts.run_artifacts.RunArtifacts,
    workdir: pathlib.Path
) -> dict[str, typing.Any]
abstract

The JSON-serializable input for the worker. Files it stages go under workdir, removed afterwards.

Raise ExportPreparationError when the run cannot be exported.

nemo_gym.artifacts.exporters.isolated.IsolatedExporter.export(
    artifacts: nemo_gym.artifacts.run_artifacts.RunArtifacts
) -> nemo_gym.artifacts.exporters.base.ExportResult
classmethod
nemo_gym.artifacts.exporters.isolated.IsolatedExporter.required_packages() -> tuple[str, ...]
nemo_gym.artifacts.exporters.isolated.IsolatedExporter.worker_timeout_s(
    payload: dict[str, typing.Any]
) -> typing.Optional[float]

Seconds the worker may run before it is killed and the export fails; None for no limit.

A backend SDK can hang (a stalled upload, a retry loop), and an export that runs in a job script would hold the job’s allocation until then.

nemo_gym.artifacts.exporters.isolated._worker_environment(
    python: pathlib.Path
) -> dict[str, str]

This process’s environment minus what would point the worker’s Python back at Gym’s packages.

The exporter environment’s bin leads PATH, so a bare python or an SDK-installed command resolves there.

nemo_gym.artifacts.exporters.isolated.ensure_exporter_environment(
    venv_root: pathlib.Path,
    name: str,
    requirements: tuple[str, ...]
) -> pathlib.Path

Build the exporter’s virtual environment if it is not ready, and return its Python.

Concurrent exports share one build: setup_environment serializes installers with a file lock and skips a ready environment.

nemo_gym.artifacts.exporters.isolated.exporter_venv_path(
    venv_root: pathlib.Path,
    name: str,
    requirements: tuple[str, ...]
) -> pathlib.Path

<venv_root>/exporters/<name>/<digest of requirements>/.venv.

Keying on the requirements gives a changed requirement list a fresh environment, while a given list keeps its environment, which an SDK is free to update itself within.

nemo_gym.artifacts.exporters.isolated.WORKER_HARNESS_FPATH = Path(__file__).with_name('_isolated_worker.py')