> This page is for version Latest (default). > For other versions, use one of these documentation indexes: > - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt > - 0.8.0: https://docs.nvidia.com/nemo-helix/v0.8.0/llms.txt > - 0.7.1: https://docs.nvidia.com/nemo-helix/v0.7.1/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo-helix/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo-helix/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo-helix/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo-helix/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo-helix/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo-helix/v0.3.0/llms.txt > - 0.2.0: https://docs.nvidia.com/nemo-helix/v0.2.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server. # Evaluate a NeMo Gym Environment > Run an existing NeMo Gym environment through NeMo Helix Evals's Gym runner — Gym collects rollouts against its own resources-server and agent, and the SDK adapts them into trials scored on Gym's reward. [NeMo Gym](https://github.com/NVIDIA-NeMo/Gym) is an environment framework for agentic rollouts: a **resources-server** provides the environment, an **agent** acts in it, and each rollout carries a **reward**. If you already have a Gym environment, the **Gym runner** runs it and scores its reward through agent-eval — the same [`AgentEvaluator`](/documentation/evaluate-models/agent-eval) and the same result and bundle as the [quickstart](/documentation/evaluate-models/agent-eval/quickstart). Only the runner changes. ## Choose how to run the evaluation There are two user-facing interfaces: | Interface | Use it for | How it runs | | ---------------- | --------------------------------------------------------------- | --------------------------------------------------------------- | | **Local SDK** | Developing the task set and runner on your machine | `AgentEvaluator().run(...)` invokes local `gym` subprocesses | | **Platform job** | Durable, repeatable execution with platform storage and results | Submit `agent-evaluate`; the deployment controls where Gym runs | On the platform, sandbox placement is an operator decision rather than another submission mode. A sandbox-enabled deployment runs Gym in a separate `nhx-gym-host`. A deployment without OpenSandbox can still run trusted, built-in Gym components inside the `nhx-gym-tasks` job container; this compatibility path is called **colocated execution**. Submitters do not select between them, and custom environment FileSets always require the sandboxed path. Gym owns execution **and** scoring through either interface. Both produce the same trial shape, and `GymRewardMetric` surfaces Gym's per-attempt reward. > **Warning** > > For a local SDK run, this runner is **not** zero-dependency — it shells out to the `gym` CLI: > > * **CPython 3.13.14** — `nemo-gym` declares `Requires-Python: >=3.13.14` > * **NeMo Gym**, installed into its own `uv` environment — see [Install NeMo Gym](#install-nemo-gym) > * **The target environment's own dependencies** — each resources-server ships its own > `requirements.txt` (the `mcqa` example needs `tiktoken`) > * **Model credentials** for the collector, in an `env.yaml` — see > [Credentials for local SDK runs](#credentials-for-local-sdk-runs) ## Install NeMo Gym ```bash uv venv ~/gym-env --python 3.13.15 uv pip install --python ~/gym-env/bin/python "nemo-gym==0.5.0" tiktoken export PATH="$HOME/gym-env/bin:$PATH" ``` NeMo Gym 0.5.0 requires Python 3.13.14 or newer. Pin the Gym version so the local runner matches the NeMo Helix job image. Install Gym into **its own environment** and put that environment's `bin` on `PATH`. Gym imports Ray at module load, and `nemo-helix` excludes Ray by constraint, so the two generally cannot share a virtualenv. > **Note** > > `uv` downloads CPython 3.13.14 on demand, so no separate install is needed. A `uv` older than that > build carries no 3.13.14 in its embedded list and fails with `No interpreter found for Python > 3.13.14`; upgrade `uv` rather than dropping to an earlier Python. > > Gym starts each component server with `uv run` at the same floor, so a missing interpreter surfaces > later as `Process finished unexpectedly!`, with the real error only in the environment log. The runner resolves `gym` from `PATH` only — there is deliberately no setting pointing at a checkout or another venv, because this config becomes a serialized job spec when Gym runs as a platform job, and a local path means nothing on the other side of that boundary. In a colocated job image, the image owns `PATH` and this resolves normally. A sandboxed job runs the CLI in a separate `nhx-gym-host` image instead. ## Credentials for local SDK runs Gym's collector calls your model endpoint directly. It reads the credentials from an `env.yaml` in the directory you run from — **this SDK never reads or handles that file**: ```yaml policy_base_url: https:///v1 policy_api_key: policy_model_name: ``` Keep it out of version control. Gym searches the working directory first, then its install root. Platform jobs do not read this local file. Configure their model route with `hydra_params`, and map secret environment variables through `GymRunnerTarget.env_secrets`. See [Configure Sandboxed Gym](/documentation/evaluate-models/agent-eval/gym-sandbox-configuration). ## The dataset A Gym dataset is a **jsonl file**, one row per case. Environments ship their example data inside the `nemo-gym` wheel, so the bundled `mcqa` benchmark needs no checkout: ``` /resources_servers/mcqa/data/example.jsonl ``` `discover_gym_tasks` turns that file into tasks — one per **distinct** row: ```python from nhx_evals_sdk.agent_eval.runtimes.gym import GymRewardMetric, discover_gym_tasks tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()]) ``` Task identity is the row's **content hash**, which has two consequences worth knowing before you build a dataset: * Duplicate rows collapse into a single task, and the runner warns. Duplicates usually mean a data problem. * Repeating a row is **not** how you ask for repeated attempts. Use `num_repeats` — attempts are a run-level concern, not a dataset one. ## Run it ```python import asyncio from nhx_evals_sdk.agent_eval.evaluator import AgentEvaluator from nhx_evals_sdk.agent_eval.runtimes.gym import ( GymAgentTaskRunner, GymRewardMetric, GymRuntimeConfig, discover_gym_tasks, ) tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()]) runner = GymAgentTaskRunner( config=GymRuntimeConfig( agent="simple_agent", agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml", resources_server="mcqa", num_repeats=2, ) ) result = asyncio.run(AgentEvaluator().run(tasks=tasks, target=runner)) print(result.summary) ``` Run it from the directory holding `env.yaml`. The mapping is: * one Gym dataset → one run * each distinct row → one task * each attempt → one trial So `num_repeats=2` over a 5-row dataset yields 5 tasks and 10 trials. ### Local runner configuration | Field | Required | Notes | | ----------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `agent` | yes | agent name to collect rollouts with, e.g. `simple_agent` | | `agent_config` | yes | agent config passed to `gym env start`, resolved relative to the Gym install, e.g. `responses_api_agents/simple_agent/configs/simple_agent.yaml` | | `resources_server` | yes | resources-server (environment) name, e.g. `mcqa` | | `model_type` | no | `inference_provider` for OpenAI-compatible **chat** endpoints; `openai_model` uses the OpenAI **Responses API** and fails against chat-only endpoints | | `bind_resources_server` | no | auto-bind the agent's `resources_server.name` via a Hydra override, for a composable agent whose config leaves it unset (`simple_agent`). Set `False` for a self-contained agent that already binds its own | | `num_repeats` | no | attempts per row; each attempt becomes one trial | | `concurrency` | no | concurrent rollouts during collection — tune to your model endpoint's limits | | `hydra_params` | no | parameters merged into Gym's Hydra config, e.g. `{"model": {"temperature": 0.7}}` | | `env_vars` | no | environment variables set on the `gym` invocation | | `env_secrets` | no | `{ENV_NAME: secret-ref}` for values that must not be written down — local runs look for `MY_WORKSPACE_*` first, then the bare variable name; submitted jobs use platform secrets | | `reward_key` | no | key read from each rollout record (default `reward`) | | `startup_timeout_s` | no | max wait for `gym env start` readiness | | `collection_timeout_s` | no | max wait for collection; `None` is unbounded | | `shutdown_grace_s` | no | grace period for the Gym subprocess group to exit on `SIGTERM`, letting Ray shut down cleanly, before escalating to `SIGKILL` | Anything `GymRuntimeConfig` does not expose can go through `hydra_params`, which is flattened to Hydra's override grammar and applied to `gym env start`. For the full set of knobs, see the [NeMo Gym documentation](https://github.com/NVIDIA-NeMo/Gym). Platform job specs use `GymRunnerTarget`, the serializable counterpart of the local `GymRuntimeConfig`. Submitting a runner builds one, and the deployment-specific settings come from a `GymPlacement` passed alongside it rather than from the runner: * `environment` points to a FileSet containing Gym component configuration, code, and dependencies that are not built into the runtime image. * `agent_ref_name` identifies the agent instance registered inside the sandbox when that instance name differs from the `agent` component name. On the target these arrive as one field: `source` holds the agent as `{"component": ..., "config": ..., "instance": ...}`, so a spec names exactly one agent, and a hand-written spec gives `source` directly: ```json {"kind": "gym", "source": {"component": "simple_agent", "config": "responses_api_agents/simple_agent/configs/simple_agent.yaml"}, "resources_server": "mcqa"} ``` They live on the placement because the local `gym` CLI has no equivalent of either — it stages no packages and resolves its agent from Hydra config — so a runner carrying them would hold fields that do nothing wherever it actually runs. See [Run a Custom Gym Environment](/documentation/evaluate-models/agent-eval/gym-custom-environment). ## Attempts and concurrency `num_repeats` sets how many attempts each row gets, and every attempt becomes one trial — five rows at `num_repeats=2` produce five tasks and ten trials. Throughput has **two** separate knobs, and they govern different phases: | Knob | Phase | Bounds | | ------------------------------------------ | ---------- | --------------------------------------------------------------------- | | `GymRuntimeConfig.concurrency` (default 4) | collection | concurrent rollouts `gym eval run` drives against your model endpoint | | `AgentEvalRunConfig.parallelism` | scoring | concurrent metric evaluation after Gym returns the trials | Raising `parallelism` does not speed up collection, and it is deliberately not mapped onto Gym's `--concurrency`: scoring is SDK-side and cheap, while rollouts run against an endpoint with its own rate limits. Tune `concurrency` to that endpoint. Three timeouts bound the subprocess: `startup_timeout_s` (default 240s) for `gym env start` readiness, `collection_timeout_s` (default `None`, unbounded) for the run itself, and `shutdown_grace_s` (default 30s) before the Gym process group is escalated from `SIGTERM` to `SIGKILL`. A collection timeout scales with dataset size x `num_repeats` / `concurrency`, so set one only once you know roughly how long a run takes. ## Read the results Gym's reward arrives as `gym_reward.reward`, and Gym's own aggregates are imported alongside the SDK's under a `runner.gym.*` prefix — the prefix is what tells you which side computed a number: ```python for score in result.summary.scores.scores: print(score.name) # gym_reward.reward # gym_reward.reward.pass@1 # runner.gym.pass@1/accuracy # runner.gym.input_tokens ``` Aggregates arrive in more than one shape, and `.mean` is empty on some of them. A `scalar` aggregate is a single number the runner already computed, held in `.value`; a `range` aggregate is a distribution, with `.mean` plus `.min`, `.max`, `.median` and `sample_std_dev`. Gym produces both — metrics that arrive with a complete mean/min/max/median/std family become `range`, and every remaining key becomes `scalar` — so the shape does not follow the name prefix. Read **`headline_value`** rather than `.mean`, and the distinction stops mattering: it returns the one number each aggregate reports, whatever its shape. ```python for aggregate in result.summary.scores.scores: print(f"{aggregate.name}: {aggregate.headline_value}") # gym_reward.reward: 0.8 # runner.gym.input_tokens: 216.4 # runner.gym.pass@1/accuracy: 80.0 ``` Compute on `gym_reward.reward`. The SDK derives it from the same rollouts on its own **0–1** scale, and drops Gym's duplicate of it. Treat `runner.gym.*` as Gym's own reporting, kept in Gym's units so it can be checked against Gym's output: accuracy there is **0–100**, so `runner.gym.pass@1/accuracy` of `80.0` is the same result as a `gym_reward.reward` mean of `0.8`. Trials, scores, and the run bundle are otherwise read exactly as in [Reading Results](/documentation/evaluate-models/agent-eval/reading-results). ## Output directories Each run writes to a fresh temporary directory by default. To choose one, set `work_dir` on the run config: ```python from pathlib import Path from nhx_evals_sdk.agent_eval.tasks import AgentEvalRunConfig result = asyncio.run( AgentEvaluator().run( tasks=tasks, target=runner, config=AgentEvalRunConfig(work_dir=Path("gym-run-1")) ) ) ``` Give every run its own. The runner **refuses** to reuse a directory that already holds Gym rollout output, raising `FileExistsError`: Gym appends to its failures sidecar, so reusing one would mix two runs together, and the runner raises rather than clearing a previous run's results. Gym's own artifacts land in a `gym_run/` subdirectory — `rollouts.jsonl`, `rollouts_failures.jsonl`, `rollouts_aggregate_metrics.json`, and the materialized `gym_input.jsonl` handed to collection. ## How local SDK and colocated platform runs execute Gym The runner uses Gym's **two-step** flow, which reads a dataset file directly — no split-driven data preparation and no HuggingFace downloads: 1. `gym env start …` brings up the resources-server, agent, and model servers. 2. `gym eval run --no-serve --input …` collects rollouts against them. The dataset handed to step 2 is **not** your source file. The runner materializes a normalized one into the run's work directory, one row per requested task, with `_ng_task_index` stamped explicitly. Gym honors a caller-supplied `_ng_task_index` and echoes it back on every rollout record, so rollouts join back to tasks through a map the runner owns rather than a guess about Gym's row ordering. That is also what lets you run a **subset** of tasks and roll out only that subset. ### Logs Gym's subprocess output is streamed to files in the run's work directory — `gym_env.log` for startup, and `gym_eval.stdout.log` / `gym_eval.stderr.log` for collection — and mirrored to the `nhx_evals_sdk.agent_eval.runtimes.gym` logger at `DEBUG`. Startup and collection failures name the relevant file and inline its last lines. To watch Gym's output in your own terminal: ```python import logging logging.getLogger("nhx_evals_sdk.agent_eval.runtimes.gym").setLevel(logging.DEBUG) ``` ## Submit as a platform job A Gym runner can be submitted as a durable platform job **from the live runner object**, rather than described a second time as a job spec — the configuration you validated locally is the configuration that runs. `submit` takes a stored taskset, so the Gym rows have to be stored first. A Gym taskset is not an ordinary one: the job rebuilds the Gym dataset from each task, so every task must carry the row that `discover_gym_tasks` split across `inputs['gym_row']` and `metadata['gym_row_extras']`. Build the tasks with `discover_gym_tasks` and store both halves: ```python from nemo_evals.api.fields import MetadataItem, MetricInline from nemo_evals.api.schemas import ( EvaluatorTaskDefinition, TaskInput, TaskInputs, TaskRef, TasksetInput, TasksetRef, ) from nemo_evals.shared.metric_bundles.bundles import bundle_metric from nemo_evals.shared.metric_bundles.inline import InlineMetricBundlePackager from nhx_evals_sdk.agent_eval.runtimes.gym import ( GymAgentTaskRunner, GymRewardMetric, GymRuntimeConfig, discover_gym_tasks, ) from nemo_helix_plugin.client.client import NemoClient from nemo_evals.sdk import Evaluator client = NemoClient(base_url="http://localhost:8080", workspace="default") evaluator = Evaluator.from_client(client) runner = GymAgentTaskRunner( config=GymRuntimeConfig( agent="simple_agent", agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml", resources_server="mcqa", ) ) tasks = discover_gym_tasks("path/to/example.jsonl") # GymRewardMetric is a built-in metric type, so it bundles inline — no cloudpickle opt-in. reward = MetricInline.model_validate( bundle_metric(GymRewardMetric(), InlineMetricBundlePackager()).model_dump(mode="json") ) names = [] for index, task in enumerate(tasks): name = f"mcqa-{index}" names.append(name) evaluator.tasks.create( name, task=TaskInput( spec=EvaluatorTaskDefinition( kind="evaluator", intent=task.intent, inputs=TaskInputs(**task.inputs), metrics=[reward], ), metadata=[MetadataItem(key=key, value=value) for key, value in task.metadata.items()], ), ) evaluator.tasksets.create("mcqa-suite", taskset=TasksetInput(tasks=[TaskRef(n) for n in names])) job = evaluator.submit(tasks=TasksetRef("mcqa-suite"), target=runner) job.wait_until_done() ``` Two things that bite here: * **Do not name the task after `task.id`.** It is a 64-character hex digest, already past the 63-character cap on entity names, and it may begin with a digit where a name must start with a letter. Derive a name, as above; the digest stays the task's own `id`. * **`gym_row` rides on `inputs` and `gym_row_extras` on the task's `metadata`** — the field beside `spec`, not inside it. A task missing either is rejected job-side with `task '' is missing inputs['gym_row'] and/or metadata['gym_row_extras']`. A value with no JSON form — a callable in `hydra_params`, say — is refused with `UnsubmittableRunnerError` at submit time, rather than failing inside the transport with an error that names neither the runner nor the field. The returned handle is an `AgentEvaluatorJobResource`. Unlike a dataset-driven job it has no `get_result()` or `download_artifacts()`. Wait for completion, then read the queryable record through `evaluator.agent_eval_results` or download the named `agent-eval-results` job result. See [Reading Results](/documentation/evaluate-models/agent-eval/reading-results#results-from-platform-jobs). The async client takes the same call — `await async_evaluator.submit(tasks=..., target=runner)` on an `AsyncEvaluator`, returning an `AsyncAgentEvaluatorJobResource` with the same status and polling methods. See [SDK Resources](/documentation/evaluate-models/sdk-resources). Submitting from the live runner is not the only route. The same evaluation can be described as an `agent-evaluate` spec with a `GymRunnerTarget` and submitted with the CLI, which is how a job that stages a custom environment is usually run: ```bash nemo evals agent-evaluate --spec-file gym_agent_eval.json ``` [Run a Custom Gym Environment](/documentation/evaluate-models/agent-eval/gym-custom-environment) has a worked spec. On a sandbox-enabled deployment, the Evaluator step runs in `nhx-tasks` and calls a separate `nhx-gym-host`. On a deployment using the compatibility path, Evaluator and Gym run together in `nhx-gym-tasks`. A target with an `environment` FileSet is rejected on the compatibility path and adds `stage-environment` before `agent-evaluate` on a sandbox-enabled deployment. ## Run a registered agent An agent registered on the platform with `nemo agents create` can be the Gym target's `source`, in place of a Gym `component`: ```json { "tasks": "default/my-gym-taskset", "target": { "kind": "gym", "source": {"agent": "calculator-agent"}, "resources_server": "mcqa", "num_repeats": 2 } } ``` At submit time the service resolves the agent exactly as `nemo agents deploy` would (models bound to the workspace's Inference Gateway, the optional `source.environment` merged on top) and keeps the translated config on the target as `resolved_config`. Gym's `policy_*` settings default to the agent's default model, so the policy model server proxies to the same model the agent would call when deployed; set them in `hydra_params` to point it elsewhere. ```json { "tasks": "default/my-gym-taskset", "target": { "kind": "gym", "source": { "agent": "calculator-agent", "environment": { "mcp": {"calculator": {"url": "http://mock-calculator:8000/mcp"}}, "env": {"CALC_MODE": "strict"}, "secrets": {"CALC_TOKEN": "calc-token"} } }, "resources_server": "mcqa", "hydra_params": { "policy_base_url": "http://host.docker.internal:8080/apis/inference-gateway/v2/workspaces/default/openai/-/v1" } } } ``` `environment.mcp.calculator.url` replaces the URL of the `calculator` server the agent declares (a server it does not declare is rejected), `env` is plain process environment for the component, and `secrets` binds an environment variable to a Secrets entry by name. `hydra_params.policy_base_url` is the laptop case: the policy model server runs inside the sandbox, so it reaches the gateway through `host.docker.internal`. A value you set in `hydra_params` always wins over the default derived from the agent. The job's `stage-environment` step then assembles a `wheels-v1` environment package: the platform's Gym agent component, an agent-instance config carrying the resolved config, the agent's Ethos files (skills and prompts it refers to by relative path, from the snapshot submission took), and a wheelhouse with the Fabric harness the agent needs, downloaded for the Gym host's interpreter. The sandboxed Gym host installs that wheelhouse offline and runs the component like any package-supplied agent: the registered agent's identity, harness, instructions, tools, MCP servers and skills are what run; only its default model is rebound to Gym's policy model server. Secrets bound to the agent's environment travel as `env_secrets` and reach the component's process, where a stdio MCP server's `${NAME}` templates are filled. What this requires: * Sandboxed execution. Colocated Gym ignores the staged package, so a registered agent is refused there. * If you also supply a `target.environment` FileSet, it must be `wheels-v1`; the agent's package is added to it, and a `native-v1` package has no offline way to install the harness. * The sandbox must be allowed to reach the Inference Gateway URL the agent's model is bound to (`sandbox_policy_base_urls`); on a laptop that means the gateway through `host.docker.internal`, not `localhost`. * The wheelhouse is downloaded from a package index by the job container on every run, tens of megabytes per harness. Image-side packaging is planned to replace this. * The wheelhouse is resolved under the Gym host image's own package pins, so the harness installs into the host without moving anything the host already has. Nothing to configure: the pins ship with the evaluator and follow the image's lock. * The agent's `runtime.timeout_seconds` is its per-rollout task budget, enforced by Fabric from the moment the harness starts the task; an agent that declares none gets 600 seconds. Adapter start-up and wheelhouse installation inside the host do not count against it. * A registered agent that binds several models keeps only its default through Gym's policy server; named secondary models still call the gateway directly. Relay/ATIF trajectory capture from inside the Gym host is not wired yet; the rollout record carries the agent's messages as Responses items. ## Caveats * **Per-environment dependencies are heterogeneous.** `mcqa` needs only `tiktoken`; other Gym environments pull in `torch`, COMET, a GPU, or Docker. Providing a Gym runtime with those installed is the caller's responsibility. * **`--no-serve --input` bypasses Gym's data-prep** — prompt templating and dataset materialization. Rows that are already complete, like the bundled `example.jsonl`, are faithful; an environment whose rows need templating would need that step run first. * **Custom environments require platform support.** Package the environment as a FileSet and use sandboxed execution; `environment` and `agent_ref_name` (the target's `source.instance`) are `GymPlacement` fields supplied at submission, so a live `GymAgentTaskRunner` cannot carry them. `env_secrets` is *not* in that group — it lives on `GymRuntimeConfig`. Local runs look for the workspace-prefixed environment variable first, then the bare name. Pass `secret_resolver=` to `GymAgentTaskRunner` for a custom `EnvSecretSource` that names the environment variable holding each secret. Submitted jobs carry only the references and resolve them with the platform's source. * **No retries and no re-run cache.** A failed rollout is not retried, and nothing is reused between runs: every run collects from scratch, and refuses to run into a directory that already holds Gym rollout output — rollouts, the failures sidecar, or model-call captures. (An existing empty directory is fine.) Re-running a large suite pays the full collection cost again. (The [Harbor runner](/documentation/evaluate-models/agent-eval/harbor-runner) does have both — do not carry that expectation across.) ## Next steps #### [Agent Evaluation (concepts)](/documentation/evaluate-models/agent-eval) #### [Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners) #### [Run a Custom Gym Environment](/documentation/evaluate-models/agent-eval/gym-custom-environment) #### [Configure Sandboxed Gym](/documentation/evaluate-models/agent-eval/gym-sandbox-configuration)