> This page is for version 0.11.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt
> - 0.11: https://docs.nvidia.com/nemo-helix/v0.11/llms.txt
> - 0.10: https://docs.nvidia.com/nemo-helix/v0.10/llms.txt
> - 0.9: https://docs.nvidia.com/nemo-helix/v0.9/llms.txt
> - 0.8: https://docs.nvidia.com/nemo-helix/v0.8/llms.txt
> - 0.7: https://docs.nvidia.com/nemo-helix/v0.7/llms.txt
> - 0.6: https://docs.nvidia.com/nemo-helix/v0.6/llms.txt
> - 0.5: https://docs.nvidia.com/nemo-helix/v0.5/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.

# Evaluate a NeMo Gym Environment

> Run an existing NeMo Gym environment through NeMo Helix Evals's Gym runner — Gym collects rollouts against its own resources-server and agent, and the SDK adapts them into trials scored on Gym's reward.

[NeMo Gym](https://github.com/NVIDIA-NeMo/Gym) is an environment framework for agentic rollouts: a
**resources-server** provides the environment, an **agent** acts in it, and each rollout carries a
**reward**. If you already have a Gym environment, the **Gym runner** runs it and scores its reward
through agent-eval — the same [`AgentEvaluator`](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval) and the
same result and bundle as the [quickstart](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval/quickstart).
Only the runner changes.

## Choose how to run the evaluation

There are two user-facing interfaces:

| Interface        | Use it for                                                      | How it runs                                                     |
| ---------------- | --------------------------------------------------------------- | --------------------------------------------------------------- |
| **Local SDK**    | Developing the task set and runner on your machine              | `AgentEvaluator().run(...)` invokes local `gym` subprocesses    |
| **Platform job** | Durable, repeatable execution with platform storage and results | Submit `agent-evaluate`; the deployment controls where Gym runs |

On the platform, sandbox placement is an operator decision rather than another submission mode. A
sandbox-enabled deployment runs Gym in a separate `nhx-gym-host`. A deployment without OpenSandbox
can still run trusted, built-in Gym components inside the `nhx-gym-tasks` job container; this
compatibility path is called **colocated execution**. Submitters do not select between them, and
custom environment FileSets always require the sandboxed path.

Gym owns execution **and** scoring through either interface. Both produce the same trial shape, and
`GymRewardMetric` surfaces Gym's per-attempt reward.

> **Warning**
>
> For a local SDK run, this runner is **not** zero-dependency — it shells out to the `gym` CLI:
>
> * **CPython 3.13.14** — `nemo-gym` declares `Requires-Python: >=3.13.14`
> * **NeMo Gym**, installed into its own `uv` environment — see [Install NeMo Gym](#install-nemo-gym)
> * **The target environment's own dependencies** — each resources-server ships its own
>   `requirements.txt` (the `mcqa` example needs `tiktoken`)
> * **Model credentials** for the collector, in an `env.yaml` — see
>   [Credentials for local SDK runs](#credentials-for-local-sdk-runs)

## Install NeMo Gym

```bash
uv venv ~/gym-env --python 3.13.15
uv pip install --python ~/gym-env/bin/python "nemo-gym==0.5.0" tiktoken
export PATH="$HOME/gym-env/bin:$PATH"
```

NeMo Gym 0.5.0 requires Python 3.13.14 or newer. Pin the Gym version so the local runner matches the
NeMo Helix job image. Install Gym into **its own environment** and put that environment's `bin` on `PATH`. Gym imports Ray
at module load, and `nemo-helix` excludes Ray by constraint, so the two generally cannot share a
virtualenv.

> **Note**
>
> `uv` downloads CPython 3.13.14 on demand, so no separate install is needed. A `uv` older than that
> build carries no 3.13.14 in its embedded list and fails with `No interpreter found for Python
> 3.13.14`; upgrade `uv` rather than dropping to an earlier Python.
>
> Gym starts each component server with `uv run` at the same floor, so a missing interpreter surfaces
> later as `Process <name> finished unexpectedly!`, with the real error only in the environment log.

The runner resolves `gym` from `PATH` only — there is deliberately no setting pointing at a checkout
or another venv, because this config becomes a serialized job spec when Gym runs as a platform job,
and a local path means nothing on the other side of that boundary. In a colocated job image, the
image owns `PATH` and this resolves normally. A sandboxed job runs the CLI in a separate
`nhx-gym-host` image instead.

## Credentials for local SDK runs

Gym's collector calls your model endpoint directly. It reads the credentials from an `env.yaml` in
the directory you run from — **this SDK never reads or handles that file**:

```yaml
policy_base_url: https://<your-openai-compatible-endpoint>/v1
policy_api_key: <key>
policy_model_name: <model, e.g. meta/llama-3.1-8b-instruct>
```

Keep it out of version control. Gym searches the working directory first, then its install root.
Platform jobs do not read this local file. Configure their model route with `hydra_params`, and map
secret environment variables through `GymRunnerTarget.env_secrets`. See
[Configure Sandboxed Gym](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval/gym-sandbox-configuration).

## The dataset

A Gym dataset is a **jsonl file**, one row per case. Environments ship their example data inside the
`nemo-gym` wheel, so the bundled `mcqa` benchmark needs no checkout:

```
<site-packages>/resources_servers/mcqa/data/example.jsonl
```

`discover_gym_tasks` turns that file into tasks — one per **distinct** row:

```python
from nhx_evals_sdk.agent_eval.runtimes.gym import GymRewardMetric, discover_gym_tasks

tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])
```

Unless rows are named by a key (`id_field`, see [Keep tasks across edits](#keep-tasks-across-edits)),
task identity is the row's content hash. Each task's id is `gym-` followed by 59 hex digits of
the hash, which is also a valid platform entity name. Content-hash identity has two consequences
worth knowing before you build a dataset:

* Duplicate rows collapse into a single task, and the runner warns. Duplicates usually mean a data
  problem.
* Repeating a row is **not** how you ask for repeated attempts. Use `num_repeats` — attempts are a
  run-level concern, not a dataset one.

## Run it

```python
import asyncio

from nhx_evals_sdk.agent_eval.evaluator import AgentEvaluator
from nhx_evals_sdk.agent_eval.runtimes.gym import (
    GymAgentTaskRunner,
    GymRewardMetric,
    GymRuntimeConfig,
    discover_gym_tasks,
)

tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])

runner = GymAgentTaskRunner(
    config=GymRuntimeConfig(
        agent="simple_agent",
        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
        resources_server="mcqa",
        num_repeats=2,
    )
)

result = asyncio.run(AgentEvaluator().run(tasks=tasks, target=runner))
print(result.summary)
```

Run it from the directory holding `env.yaml`.

The mapping is:

* one Gym dataset → one run
* each distinct row → one task
* each attempt → one trial

So `num_repeats=2` over a 5-row dataset yields 5 tasks and 10 trials.

### Local runner configuration

| Field                   | Required | Notes                                                                                                                                                                                                       |
| ----------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent`                 | yes      | agent name to collect rollouts with, e.g. `simple_agent`                                                                                                                                                    |
| `agent_config`          | yes      | agent config passed to `gym env start`, resolved relative to the Gym install, e.g. `responses_api_agents/simple_agent/configs/simple_agent.yaml`                                                            |
| `resources_server`      | yes      | resources-server (environment) name, e.g. `mcqa`                                                                                                                                                            |
| `model_type`            | no       | `inference_provider` for OpenAI-compatible **chat** endpoints; `openai_model` uses the OpenAI **Responses API** and fails against chat-only endpoints                                                       |
| `bind_resources_server` | no       | auto-bind the agent's `resources_server.name` via a Hydra override, for a composable agent whose config leaves it unset (`simple_agent`). Set `False` for a self-contained agent that already binds its own |
| `num_repeats`           | no       | attempts per row; each attempt becomes one trial                                                                                                                                                            |
| `concurrency`           | no       | concurrent rollouts during collection — tune to your model endpoint's limits                                                                                                                                |
| `hydra_params`          | no       | parameters merged into Gym's Hydra config, e.g. `{"model": {"temperature": 0.7}}`                                                                                                                           |
| `env_vars`              | no       | environment variables set on the `gym` invocation                                                                                                                                                           |
| `env_secrets`           | no       | `{ENV_NAME: secret-ref}` for values that must not be written down — local runs look for `MY_WORKSPACE_*` first, then the bare variable name; submitted jobs use platform secrets                            |
| `reward_key`            | no       | key read from each rollout record (default `reward`)                                                                                                                                                        |
| `startup_timeout_s`     | no       | max wait for `gym env start` readiness                                                                                                                                                                      |
| `collection_timeout_s`  | no       | max wait for collection; `None` is unbounded                                                                                                                                                                |
| `shutdown_grace_s`      | no       | grace period for the Gym subprocess group to exit on `SIGTERM`, letting Ray shut down cleanly, before escalating to `SIGKILL`                                                                               |

Anything `GymRuntimeConfig` does not expose can go through `hydra_params`, which is flattened to
Hydra's override grammar and applied to `gym env start`. For the full set of knobs, see the
[NeMo Gym documentation](https://github.com/NVIDIA-NeMo/Gym).

Platform job specs use `GymRunnerTarget`, the serializable counterpart of the local
`GymRuntimeConfig`. Submitting a runner builds one, and the deployment-specific settings come from a
`GymPlacement` passed alongside it rather than from the runner:

* `environment` points to a FileSet containing Gym component configuration, code, and dependencies
  that are not built into the runtime image.
* `agent_ref_name` identifies the agent instance registered inside the sandbox when that instance
  name differs from the `agent` component name.

On the target these arrive as one field: `source` holds the agent as `{"component": ..., "config": ...,
"instance": ...}`, so a spec names exactly one agent, and a hand-written spec gives `source` directly:

```json
{"kind": "gym", "source": {"component": "simple_agent", "config": "responses_api_agents/simple_agent/configs/simple_agent.yaml"}, "resources_server": "mcqa"}
```

They live on the placement because the local `gym` CLI has no equivalent of either — it stages
no packages and resolves its agent from Hydra config — so a runner carrying them would hold fields
that do nothing wherever it actually runs.
See [Run a Custom Gym Environment](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval/gym-custom-environment).

## Attempts and concurrency

`num_repeats` sets how many attempts each row gets, and every attempt becomes one trial — five rows
at `num_repeats=2` produce five tasks and ten trials.

Throughput has **two** separate knobs, and they govern different phases:

| Knob                                       | Phase      | Bounds                                                                |
| ------------------------------------------ | ---------- | --------------------------------------------------------------------- |
| `GymRuntimeConfig.concurrency` (default 4) | collection | concurrent rollouts `gym eval run` drives against your model endpoint |
| `AgentEvalRunConfig.parallelism`           | scoring    | concurrent metric evaluation after Gym returns the trials             |

Raising `parallelism` does not speed up collection, and it is deliberately not mapped onto Gym's
`--concurrency`: scoring is SDK-side and cheap, while rollouts run against an endpoint with its own
rate limits. Tune `concurrency` to that endpoint.

Three timeouts bound the subprocess: `startup_timeout_s` (default 240s) for `gym env start`
readiness, `collection_timeout_s` (default `None`, unbounded) for the run itself, and
`shutdown_grace_s` (default 30s) before the Gym process group is escalated from `SIGTERM` to
`SIGKILL`. A collection timeout scales with dataset size x `num_repeats` / `concurrency`, so set one
only once you know roughly how long a run takes.

## Read the results

Gym's reward arrives as `gym_reward.reward`, and Gym's own aggregates are imported alongside the
SDK's under a `runner.gym.*` prefix — the prefix is what tells you which side computed a number:

```python
for score in result.summary.scores.scores:
    print(score.name)
# gym_reward.reward
# gym_reward.reward.pass@1
# runner.gym.pass@1/accuracy
# runner.gym.input_tokens
```

Aggregates arrive in more than one shape, and `.mean` is empty on some of them. A `scalar` aggregate
is a single number the runner already computed, held in `.value`; a `range` aggregate is a
distribution, with `.mean` plus `.min`, `.max`, `.median` and `sample_std_dev`. Gym produces both —
metrics that arrive with a complete mean/min/max/median/std family become `range`, and every
remaining key becomes `scalar` — so the shape does not follow the name prefix.

Read **`headline_value`** rather than `.mean`, and the distinction stops mattering: it returns the
one number each aggregate reports, whatever its shape.

```python
for aggregate in result.summary.scores.scores:
    print(f"{aggregate.name}: {aggregate.headline_value}")
# gym_reward.reward: 0.8
# runner.gym.input_tokens: 216.4
# runner.gym.pass@1/accuracy: 80.0
```

Compute on `gym_reward.reward`. The SDK derives it from the same rollouts on its own **0–1** scale,
and drops Gym's duplicate of it. Treat `runner.gym.*` as Gym's own reporting, kept in Gym's units so
it can be checked against Gym's output: accuracy there is **0–100**, so
`runner.gym.pass@1/accuracy` of `80.0` is the same result as a `gym_reward.reward` mean of `0.8`.
Trials, scores, and the run bundle are otherwise read exactly as in
[Reading Results](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval/reading-results).

### Several scores per row

A resources server can score each rollout on several objectives. Gym's `BaseMultiRewardVerifyResponse`
returns them in `reward_components`, beside the scalar `reward`. Each component a trial reports
becomes a score of its own, `gym_reward.<component>`, with its own mean and pass\@k. A task built
with a plain `GymRewardMetric()` reports them too: the runner adds the components it finds in the
rollouts. A trial that doesn't report a component has no score for it, rather than a 0.

```python
# a server returning reward_components={"exact": ..., "relaxed": ...}
# gym_reward.reward
# gym_reward.exact
# gym_reward.relaxed
```

Since the SDK scores those components itself, it drops Gym's own aggregates of them, as it does
for `reward`. Other `runner.gym.*` metrics, including any a server computes in `compute_metrics`,
are imported as before.

To rescore saved trials without a runner, name the components on the metric:
`GymRewardMetric(reward_components=("exact", "relaxed"))`.

## Output directories

Each run writes to a fresh temporary directory by default. To choose one, set `work_dir` on the run
config:

```python
from pathlib import Path

from nhx_evals_sdk.agent_eval.tasks import AgentEvalRunConfig

result = asyncio.run(
    AgentEvaluator().run(
        tasks=tasks, target=runner, config=AgentEvalRunConfig(work_dir=Path("gym-run-1"))
    )
)
```

Give every run its own. The runner **refuses** to reuse a directory that already holds Gym rollout
output, raising `FileExistsError`: Gym appends to its failures sidecar, so reusing one would mix two
runs together, and the runner raises rather than clearing a previous run's results.

Gym's own artifacts land in a `gym_run/` subdirectory — `rollouts.jsonl`,
`rollouts_failures.jsonl`, `rollouts_aggregate_metrics.json`, and the materialized
`gym_input.jsonl` handed to collection.

To keep files an environment produces, such as generated media or checkpoints, have its servers write
them to the directory named by the `NHX_GYM_OUTPUT_DIR` environment variable. The runner creates it as
`gym_run/outputs/`, refusing to start if it already holds files, and sets the variable to its absolute
path for every Gym process, overriding any value in `env_vars`. A
colocated platform job saves it with the rest of `gym_run/` in the `agent-eval-results` result. Only
regular files are uploaded; symbolic links are skipped. Sandboxed Gym runs don't return this directory.

## How local SDK and colocated platform runs execute Gym

The runner uses Gym's **two-step** flow, which reads a dataset file directly — no split-driven data
preparation and no HuggingFace downloads:

1. `gym env start …` brings up the resources-server, agent, and model servers.
2. `gym eval run --no-serve --input <dataset> …` collects rollouts against them.

The dataset handed to step 2 is **not** your source file. The runner materializes a normalized one
into the run's work directory, one row per requested task, with `_ng_task_index` stamped explicitly.
Gym honors a caller-supplied `_ng_task_index` and echoes it back on every rollout record, so rollouts
join back to tasks through a map the runner owns rather than a guess about Gym's row ordering. That
is also what lets you run a **subset** of tasks and roll out only that subset.

### Logs

Gym's subprocess output is streamed to files in the run's work directory — `gym_env.log` for startup,
and `gym_eval.stdout.log` / `gym_eval.stderr.log` for collection — and mirrored to the
`nhx_evals_sdk.agent_eval.runtimes.gym` logger at `DEBUG`. Startup and collection failures name
the relevant file and inline its last lines. To watch Gym's output in your own terminal:

```python
import logging

logging.getLogger("nhx_evals_sdk.agent_eval.runtimes.gym").setLevel(logging.DEBUG)
```

## Submit as a platform job

A Gym runner can be submitted as a durable platform job **from the live runner object**, rather than
described a second time as a job spec — the configuration you validated locally is the configuration
that runs.

`submit` takes a stored taskset, so the Gym rows have to be stored first. A Gym taskset is not an
ordinary one: the job rebuilds the Gym dataset from each task, so every task must carry the row that
`discover_gym_tasks` split across `inputs['gym_row']` and `metadata['gym_row_extras']`. Build the
tasks with `discover_gym_tasks` and store both halves:

```python
from nemo_evals.api.fields import MetadataItem, MetricInline
from nemo_evals.api.schemas import (
    EvaluatorTaskDefinition,
    TaskInput,
    TaskInputs,
    TaskRef,
    TasksetInput,
    TasksetRef,
)
from nemo_evals.shared.metric_bundles.bundles import bundle_metric
from nemo_evals.shared.metric_bundles.inline import InlineMetricBundlePackager
from nhx_evals_sdk.agent_eval.runtimes.gym import (
    GymAgentTaskRunner,
    GymRewardMetric,
    GymRuntimeConfig,
    discover_gym_tasks,
)
from nemo_helix_plugin.client.client import NemoClient
from nemo_evals.sdk import Evaluator

client = NemoClient(base_url="http://localhost:8080", workspace="default")
evaluator = Evaluator.from_client(client)

runner = GymAgentTaskRunner(
    config=GymRuntimeConfig(
        agent="simple_agent",
        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
        resources_server="mcqa",
    )
)

tasks = discover_gym_tasks("path/to/example.jsonl")

# GymRewardMetric is a built-in metric type, so it bundles inline — no cloudpickle opt-in.
reward = MetricInline.model_validate(
    bundle_metric(GymRewardMetric(), InlineMetricBundlePackager()).model_dump(mode="json")
)

for task in tasks:
    # Naming the task after task.id, a valid task name, makes replace() idempotent: an
    # unchanged row publishes no new revision, however the file is reordered.
    evaluator.tasks.replace(
        task.id,
        task=TaskInput(
            spec=EvaluatorTaskDefinition(
                kind="evaluator",
                intent=task.intent,
                inputs=TaskInputs(**task.inputs),
                metrics=[reward],
            ),
            metadata=[MetadataItem(key=key, value=value) for key, value in task.metadata.items()],
        ),
    )

evaluator.tasksets.replace("mcqa-suite", taskset=TasksetInput(tasks=[TaskRef(task.id) for task in tasks]))

job = evaluator.submit(tasks=TasksetRef("mcqa-suite"), target=runner)
job.wait_until_done()
```

If your dataset is already in a FileSet, skip this step and see
[Run from a FileSet](#run-from-a-fileset).

Two things that bite here:

* **Name each task after `task.id`.** A name derived from the row's position, such as `mcqa-{index}`,
  renames every later row when one is inserted, and each rename publishes a new task.
* **`gym_row` rides on `inputs` and `gym_row_extras` on the task's `metadata`** — the field beside
  `spec`, not inside it. A task missing either is rejected job-side with `task '<id>' is missing
  inputs['gym_row'] and/or metadata['gym_row_extras']`.

A value with no JSON form — a callable in `hydra_params`, say — is refused with
`UnsubmittableRunnerError` at submit time, rather than failing inside the transport with an error
that names neither the runner nor the field.

The returned handle is an `AgentEvaluatorJobResource`. Unlike a dataset-driven job it has no
`get_result()` or `download_artifacts()`. Wait for completion, then read the queryable record through
`evaluator.agent_eval_results` or download the named `agent-eval-results` job result. See
[Reading Results](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval/reading-results#results-from-platform-jobs).

The async client takes the same call — `await async_evaluator.submit(tasks=..., target=runner)` on
an `AsyncEvaluator`, returning an `AsyncAgentEvaluatorJobResource` with the same status and polling
methods. See [SDK Resources](/nemo-helix/v0.11/documentation/evaluate-models/sdk-resources).

Submitting from the live runner is not the only route. The same evaluation can be described as an
`agent-evaluate` spec with a `GymRunnerTarget` and submitted with the CLI, which is how a job that
stages a custom environment is usually run:

```bash
nemo evals agent-evaluate --spec-file gym_agent_eval.json
```

[Run a Custom Gym Environment](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval/gym-custom-environment) has
a worked spec.

On a sandbox-enabled deployment, the Evaluator step runs in `nhx-tasks` and calls a separate
`nhx-gym-host`. On a deployment using the compatibility path, Evaluator and Gym run together in
`nhx-gym-tasks`. A target with an `environment` FileSet is rejected on the compatibility path and
adds `stage-environment` before `agent-evaluate` on a sandbox-enabled deployment.

## Run from a FileSet

An `agent-evaluate` job can read its tasks straight from a Gym dataset file in a FileSet, with no
client step. The file is read and converted at submission, as the submitter, exactly as
`discover_gym_tasks` converts a local file. A bad row fails the submit with HTTP 422 and a message that
names the file and line:

```json
{
  "target": {"kind": "gym", "source": {"component": "simple_agent", "config": "responses_api_agents/simple_agent/configs/simple_agent.yaml"}, "resources_server": "mcqa"},
  "tasks": {"fileset": "default/golden#gym/golden.jsonl", "snapshot": "default/golden-suite"}
}
```

```bash
nemo evals agent-evaluate --spec-file gym_fileset_eval.json
```

* `fileset` names one file: `workspace/fileset#path/to/file.jsonl`. A whole FileSet or a glob is
  refused. The FileSet can be backed by external storage such as your own S3 bucket.
* `snapshot` is optional. Without it, the tasks run inline and nothing is stored but the job.

A file may be at most 64 MiB and 50,000 rows, because conversion runs inside the Evals API while the
submission waits.

### Snapshots

With `snapshot`, the submission also publishes the file's tasks and a revision of the named taskset,
then runs exactly those tasks. Publishing is the submission's last step, so a refused submission
leaves the taskset as it was. Submit whenever the file changes:

| The file           | Tasks                                                                       | Taskset                                            |
| ------------------ | --------------------------------------------------------------------------- | -------------------------------------------------- |
| is unchanged       | Nothing written                                                             | No new revision                                    |
| has rows reordered | Nothing written                                                             | One new revision, because the file's bytes changed |
| has an edited row  | The edited row is a new task, or with `id_field` a new revision of its task | One new revision                                   |
| has a row removed  | The task is kept, so older revisions still resolve                          | One new revision without it                        |

Each taskset revision records the file it came from in its metadata (`source_fileset`,
`source_sha256`) and carries the tag `src-<sha256>`, so `default/golden-suite#src-<sha256>` selects the
revision built from a given file. Members are pinned to the exact task revisions the submission
published, and each task in the job spec records the stored revision it was frozen from.

To snapshot, the submitter needs to read the source FileSet and to read and write tasks and tasksets
in the taskset's workspace. It also needs to create FileSets there, because the first task in a workspace
stores the Gym reward metric's bundle.

The first snapshot of a file writes one task per row, so for thousands of rows it can take minutes;
give that submission a long client timeout. Later submissions write only the rows that changed, and a
submission of an unchanged file writes no task.

### Keep tasks across edits

By default a task's id is a hash of its row's content, so an edited row is a new task: nothing in a
row says it is an earlier row changed. If your rows carry a stable key, name it with `id_field` and
choose an `id_prefix`, and a task is named by its key instead:

```json
"tasks": {"fileset": "default/golden#gym/golden.jsonl", "snapshot": "default/golden-suite", "id_field": "uid", "id_prefix": "gold"}
```

* Each task's id is `<id_prefix>-` and a hash of the row's `id_field` value, 63 characters in all. Only
  the key is hashed, so editing any other field keeps the task's id. With `snapshot`, the edit is a
  new revision of that task, and older taskset revisions still pin the old one.
* `id_prefix` is 1-8 characters: a lowercase letter, then lowercase letters or digits. It can't be
  `gym`, which ids without a key use. Use one prefix
  per dataset, so two files that both use keys like `1`, `2`, `3` keep separate tasks in a workspace.
* Every row needs a non-empty string or finite number in `id_field`. A missing key or a key two rows share
  fails the submit with HTTP 422 naming the line. The key is hashed in its JSON form, so `1` and
  `"1"` are different keys.
* `discover_gym_tasks(path, id_field="uid", id_prefix="gold")` gives the same ids locally.

## Run a registered agent

An agent registered on the platform with `nemo agents register` can be the Gym target's `source`, in
place of a Gym `component`:

```json
{
  "tasks": "default/my-gym-taskset",
  "target": {
    "kind": "gym",
    "source": {"agent": "calculator-agent"},
    "resources_server": "mcqa",
    "num_repeats": 2
  }
}
```

At submit time the service resolves the agent exactly as `nemo agents deploy` would (models bound to
the workspace's Inference Gateway, the optional `source.environment` merged on top) and keeps the
translated config on the target as `resolved_config`. Gym's `policy_*` settings default to the agent's
default model, so the policy model server proxies to the same model the agent would call when deployed;
set them in `hydra_params` to point it elsewhere.

```json
{
  "tasks": "default/my-gym-taskset",
  "target": {
    "kind": "gym",
    "source": {
      "agent": "calculator-agent",
      "environment": {
        "mcp": {"calculator": {"url": "http://mock-calculator:8000/mcp"}},
        "env": {"CALC_MODE": "strict"},
        "secrets": {"CALC_TOKEN": "calc-token"}
      }
    },
    "resources_server": "mcqa",
    "hydra_params": {
      "policy_base_url": "http://host.docker.internal:8080/apis/inference-gateway/v2/workspaces/default/openai/-/v1"
    }
  }
}
```

`environment.mcp.calculator.url` replaces the URL of the `calculator` server the agent declares (a server it
does not declare is rejected), `env` is plain process environment for the component, and `secrets` binds
an environment variable to a Secrets entry by name. `hydra_params.policy_base_url` is the laptop case:
the policy model server runs inside the sandbox, so it reaches the gateway through `host.docker.internal`.
A value you set in `hydra_params` always wins over the default derived from the agent.

The job's `stage-environment` step then assembles a `wheels-v1` environment package: the platform's
Gym agent component, an agent-instance config carrying the resolved config, the agent's Ethos files
(skills and prompts it refers to by relative path, from the snapshot submission took), and a wheelhouse with the Fabric harness the
agent needs, downloaded for the Gym host's interpreter. The sandboxed Gym host installs that
wheelhouse offline and runs the component like any package-supplied agent: the registered agent's
identity, harness, instructions, tools, MCP servers and skills are what run; only its default model is
rebound to Gym's policy model server. Secrets bound to the agent's environment travel as
`env_secrets` and reach the component's process, where a stdio MCP server's `${NAME}` templates are
filled. A submission is rejected when a template names a variable that neither `env_secrets` nor
`env_vars` sets.

What this requires:

* Sandboxed execution. Colocated Gym ignores the staged package, so a registered agent is refused
  there.
* If you also supply a `target.environment` FileSet, it must be `wheels-v1`; the agent's package is
  added to it, and a `native-v1` package has no offline way to install the harness.
* The sandbox must be allowed to reach the Inference Gateway URL the agent's model is bound to
  (`sandbox_policy_base_urls`); on a laptop that means the gateway through `host.docker.internal`,
  not `localhost`.
* The wheelhouse is downloaded from a package index by the job container on every run, tens of
  megabytes per harness. Image-side packaging is planned to replace this.
* The wheelhouse is resolved under the Gym host image's own package pins, so the harness installs into
  the host without moving anything the host already has. Nothing to configure: the pins ship with the
  evaluator and follow the image's lock.
* The agent's `runtime.timeout_seconds` is its per-rollout task budget, enforced by Fabric from the moment
  the harness starts the task; an agent that declares none gets 600 seconds. Adapter start-up and
  wheelhouse installation inside the host do not count against it.
* A registered agent that binds several models keeps only its default through Gym's policy server;
  named secondary models still call the gateway directly. Relay/ATIF trajectory capture from inside
  the Gym host is not wired yet; the rollout record carries the agent's messages as Responses items.

## Caveats

* **Per-environment dependencies are heterogeneous.** `mcqa` needs only `tiktoken`; other Gym
  environments pull in `torch`, COMET, a GPU, or Docker. Providing a Gym runtime with those installed
  is the caller's responsibility.
* **`--no-serve --input` bypasses Gym's data-prep** — prompt templating and dataset materialization.
  Rows that are already complete, like the bundled `example.jsonl`, are faithful; an environment
  whose rows need templating would need that step run first.
* **Custom environments require platform support.** Package the environment as a FileSet and use
  sandboxed execution; `environment` and `agent_ref_name` (the target's `source.instance`) are
  `GymPlacement` fields supplied at
  submission, so a live `GymAgentTaskRunner` cannot carry them. `env_secrets` is *not* in that
  group — it lives on `GymRuntimeConfig`. Local runs look for the workspace-prefixed environment
  variable first, then the bare name. Pass `secret_resolver=` to `GymAgentTaskRunner` for a custom
  `EnvSecretSource` that names the environment variable holding each secret. Submitted jobs carry
  only the references and resolve them with the platform's source.
* **A failed rollout fails only its own trial.** When an agent's `/run` fails, for example because the
  model endpoint refused the request or the resources server raised while verifying, Gym records the
  failure and collection continues. That trial is `FAILED`, with no reward and with the server's reply
  in its error; the other rows are scored, and the summary's `error_count` counts the failures. Local
  runs can restore Gym's default, which ends the whole collection at the first failure, with
  `hydra_params={"route_failures_to_sidecar": False}`. Sandboxed runs always keep going.
* **No retries and no re-run cache.** A failed rollout is not retried, and nothing is reused between
  runs: every run collects from scratch, and refuses to run into a directory that already holds Gym
  rollout output — rollouts, the failures sidecar, or model-call captures. (An existing empty
  directory is fine.) Re-running a large suite pays the full collection cost again. (The
  [Harbor runner](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval/harbor-runner) does have both — do not
  carry that expectation across.)

## Next steps

#### [Agent Evaluation (concepts)](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval)

#### [Targets and Runners](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval/targets-and-runners)

#### [Run a Custom Gym Environment](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval/gym-custom-environment)

#### [Configure Sandboxed Gym](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval/gym-sandbox-configuration)