> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Evaluate a NeMo Gym Environment

> Run an existing NeMo Gym environment through NeMo Evaluator's Gym runner — Gym collects rollouts against its own resources-server and agent, and the SDK adapts them into trials scored on Gym's reward.

[NeMo Gym](https://github.com/NVIDIA-NeMo/Gym) is an environment framework for agentic rollouts: a
**resources-server** provides the environment, an **agent** acts in it, and each rollout carries a
**reward**. If you already have a Gym environment, the **Gym runner** runs it and scores its reward
through agent-eval — the same [`AgentEvaluator`](/documentation/evaluate-models/agent-eval) and the
same result and bundle as the [quickstart](/documentation/evaluate-models/agent-eval/quickstart).
Only the runner changes.

Gym owns execution **and** scoring here. The runner shells out to the `gym` CLI and adapts the
rollout bundle into trials; `GymRewardMetric` surfaces Gym's per-attempt reward.

Like the Harbor runner, this one is **not** zero-dependency — it shells out to the `gym` CLI:

* **NeMo Gym**, installed into its own `uv` environment (below)
* **The target environment's own dependencies** — each resources-server ships its own
  `requirements.txt` (the `mcqa` example needs `tiktoken`)
* **Model credentials** for the collector, in an `env.yaml` (below)

```bash
uv venv ~/gym-env --python 3.12
uv pip install --python ~/gym-env/bin/python nemo-gym tiktoken
export PATH="$HOME/gym-env/bin:$PATH"
```

Install Gym into **its own environment** and put that environment's `bin` on `PATH`. Gym imports Ray
at module load, and `nemo-platform` excludes Ray by constraint, so the two generally cannot share a
virtualenv. The runner resolves `gym` from `PATH` only — there is deliberately no setting pointing at
a checkout or another venv, because this config becomes a serialized job spec when Gym runs as a
platform job, and a local path means nothing on the other side of that boundary. In a job image, the
image owns `PATH` and this resolves normally.

## Credentials

Gym's collector calls your model endpoint directly. It reads the credentials from an `env.yaml` in
the directory you run from — **this SDK never reads or handles that file**:

```yaml
policy_base_url: https://<your-openai-compatible-endpoint>/v1
policy_api_key: <key>
policy_model_name: <model, e.g. meta/llama-3.1-8b-instruct>
```

Keep it out of version control. Gym searches the working directory first, then its install root.

## The dataset

A Gym dataset is a **jsonl file**, one row per case. Environments ship their example data inside the
`nemo-gym` wheel, so the bundled `mcqa` benchmark needs no checkout:

```
<site-packages>/resources_servers/mcqa/data/example.jsonl
```

`discover_gym_tasks` turns that file into tasks — one per **distinct** row:

```python
from nemo_evaluator_sdk.agent_eval.runtimes.gym import GymRewardMetric, discover_gym_tasks

tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])
```

Task identity is the row's **content hash**, which has two consequences worth knowing before you
build a dataset:

* Duplicate rows collapse into a single task, and the runner warns. Duplicates usually mean a data
  problem.
* Repeating a row is **not** how you ask for repeated attempts. Use `num_repeats` — attempts are a
  run-level concern, not a dataset one.

## Run it

```python
import asyncio

from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.gym import (
    GymAgentTaskRunner,
    GymRewardMetric,
    GymRuntimeConfig,
    discover_gym_tasks,
)

tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])

runner = GymAgentTaskRunner(
    config=GymRuntimeConfig(
        agent="simple_agent",
        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
        resources_server="mcqa",
        num_repeats=2,
    )
)

result = asyncio.run(AgentEvaluator().run(tasks=tasks, target=runner))
print(result.summary)
```

Run it from the directory holding `env.yaml`.

The mapping is:

* one Gym dataset → one run
* each distinct row → one task
* each attempt → one trial

So `num_repeats=2` over a 5-row dataset yields 5 tasks and 10 trials.

### Configuration

| Field                   | Required | Notes                                                                                                                                                                                                       |
| ----------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `agent`                 | yes      | agent name to collect rollouts with, e.g. `simple_agent`                                                                                                                                                    |
| `agent_config`          | yes      | agent config passed to `gym env start`, resolved relative to the Gym install, e.g. `responses_api_agents/simple_agent/configs/simple_agent.yaml`                                                            |
| `resources_server`      | yes      | resources-server (environment) name, e.g. `mcqa`                                                                                                                                                            |
| `model_type`            | no       | `inference_provider` for OpenAI-compatible **chat** endpoints; `openai_model` uses the OpenAI **Responses API** and fails against chat-only endpoints                                                       |
| `bind_resources_server` | no       | auto-bind the agent's `resources_server.name` via a Hydra override, for a composable agent whose config leaves it unset (`simple_agent`). Set `False` for a self-contained agent that already binds its own |
| `num_repeats`           | no       | attempts per row; each attempt becomes one trial                                                                                                                                                            |
| `concurrency`           | no       | concurrent rollouts during collection — tune to your model endpoint's limits                                                                                                                                |
| `hydra_params`          | no       | parameters merged into Gym's Hydra config, e.g. `{"model": {"temperature": 0.7}}`                                                                                                                           |
| `env_vars`              | no       | environment variables set on the `gym` invocation                                                                                                                                                           |
| `reward_key`            | no       | key read from each rollout record (default `reward`)                                                                                                                                                        |
| `startup_timeout_s`     | no       | max wait for `gym env start` readiness                                                                                                                                                                      |
| `collection_timeout_s`  | no       | max wait for collection; `None` is unbounded                                                                                                                                                                |
| `shutdown_grace_s`      | no       | grace period for the Gym subprocess group to exit on `SIGTERM`, letting Ray shut down cleanly, before escalating to `SIGKILL`                                                                               |

Anything `GymRuntimeConfig` does not expose can go through `hydra_params`, which is flattened to
Hydra's override grammar and applied to `gym env start`. For the full set of knobs, see the
[NeMo Gym documentation](https://github.com/NVIDIA-NeMo/Gym).

## Read the results

Gym's reward arrives as `gym_reward.reward`, and Gym's own aggregates are imported alongside the
SDK's under a `runner.gym.*` prefix — the prefix is what tells you which side computed a number:

```python
for score in result.summary.scores.scores:
    print(score.name)
# gym_reward.reward
# gym_reward.reward.pass@1
# runner.gym.pass@1/accuracy
# runner.gym.input_tokens
```

Gym reports accuracy on a **0–100** scale where the SDK uses **0–1**, so `runner.gym.pass@1/accuracy`
of `50.0` corresponds to a `gym_reward.reward` mean of `0.5`. Trials, scores, and the run bundle are
otherwise read exactly as in
[Reading Results](/documentation/evaluate-models/agent-eval/reading-results).

## Output directories

Each run writes to a fresh temporary directory by default. To choose one, set `work_dir` on the run
config:

```python
from pathlib import Path

from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig

result = asyncio.run(
    AgentEvaluator().run(
        tasks=tasks, target=runner, config=AgentEvalRunConfig(work_dir=Path("gym-run-1"))
    )
)
```

Give every run its own. The runner **refuses** to reuse a directory that already holds Gym rollout
output, raising `FileExistsError`: Gym appends to its failures sidecar, so reusing one would mix two
runs together, and the runner raises rather than clearing a previous run's results.

Gym's own artifacts land in a `gym_run/` subdirectory — `rollouts.jsonl`,
`rollouts_failures.jsonl`, `rollouts_aggregate_metrics.json`, and the materialized
`gym_input.jsonl` handed to collection.

## How it runs Gym

The runner uses Gym's **two-step** flow, which reads a dataset file directly — no split-driven data
preparation and no HuggingFace downloads:

1. `gym env start …` brings up the resources-server, agent, and model servers.
2. `gym eval run --no-serve --input <dataset> …` collects rollouts against them.

The dataset handed to step 2 is **not** your source file. The runner materializes a normalized one
into the run's work directory, one row per requested task, with `_ng_task_index` stamped explicitly.
Gym honors a caller-supplied `_ng_task_index` and echoes it back on every rollout record, so rollouts
join back to tasks through a map the runner owns rather than a guess about Gym's row ordering. That
is also what lets you run a **subset** of tasks and roll out only that subset.

### Logs

Gym's subprocess output is streamed to files in the run's work directory — `gym_env.log` for startup,
and `gym_eval.stdout.log` / `gym_eval.stderr.log` for collection — and mirrored to the
`nemo_evaluator_sdk.agent_eval.runtimes.gym` logger at `DEBUG`. Startup and collection failures name
the relevant file and inline its last lines. To watch Gym's output in your own terminal:

```python
import logging

logging.getLogger("nemo_evaluator_sdk.agent_eval.runtimes.gym").setLevel(logging.DEBUG)
```

## Submit as a platform job

A Gym runner can be submitted as a durable platform job **from the live runner object**, rather than
described a second time as a job spec — the configuration you validated locally is the configuration
that runs.

`submit` takes a stored taskset, so the Gym rows have to be stored first. A Gym taskset is not an
ordinary one: the job rebuilds the Gym dataset from each task, so every task must carry the row that
`discover_gym_tasks` split across `inputs['gym_row']` and `metadata['gym_row_extras']`. Build the
tasks with `discover_gym_tasks` and store both halves:

```python
from nemo_evaluator.api.fields import MetadataItem, MetricInline
from nemo_evaluator.api.schemas import (
    EvaluatorTaskDefinition,
    TaskInput,
    TaskInputs,
    TaskRef,
    TasksetInput,
    TasksetRef,
)
from nemo_evaluator.shared.metric_bundles.bundles import bundle_metric
from nemo_evaluator.shared.metric_bundles.inline import InlineMetricBundlePackager
from nemo_evaluator_sdk.agent_eval.runtimes.gym import (
    GymAgentTaskRunner,
    GymRewardMetric,
    GymRuntimeConfig,
    discover_gym_tasks,
)
from nemo_platform import NeMoPlatform

client = NeMoPlatform(base_url="http://localhost:8080", workspace="default")

runner = GymAgentTaskRunner(
    config=GymRuntimeConfig(
        agent="simple_agent",
        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
        resources_server="mcqa",
    )
)

tasks = discover_gym_tasks("path/to/example.jsonl")

# GymRewardMetric is a built-in metric type, so it bundles inline — no cloudpickle opt-in.
reward = MetricInline.model_validate(
    bundle_metric(GymRewardMetric(), InlineMetricBundlePackager()).model_dump(mode="json")
)

names = []
for index, task in enumerate(tasks):
    name = f"mcqa-{index}"
    names.append(name)
    client.evaluator.tasks.create(
        name,
        task=TaskInput(
            spec=EvaluatorTaskDefinition(
                kind="evaluator",
                intent=task.intent,
                inputs=TaskInputs(**task.inputs),
                metrics=[reward],
            ),
            metadata=[MetadataItem(key=key, value=value) for key, value in task.metadata.items()],
        ),
    )

client.evaluator.tasksets.create("mcqa-suite", taskset=TasksetInput(tasks=[TaskRef(n) for n in names]))

job = client.evaluator.submit(tasks=TasksetRef("mcqa-suite"), target=runner)
job.wait_until_done()
```

Two things that bite here:

* **Do not name the task after `task.id`.** It is a 64-character hex digest, already past the
  63-character cap on entity names, and it may begin with a digit where a name must start with a
  letter. Derive a name, as above; the digest stays the task's own `id`.
* **`gym_row` rides on `inputs` and `gym_row_extras` on the task's `metadata`** — the field beside
  `spec`, not inside it. A task missing either is rejected job-side with `task '<id>' is missing
  inputs['gym_row'] and/or metadata['gym_row_extras']`.

A value with no JSON form — a callable in `hydra_params`, say — is refused with
`UnsubmittableRunnerError` at submit time, rather than failing inside the transport with an error
that names neither the runner nor the field.

The returned handle is an `AgentEvaluatorJobResource`. Unlike a dataset-driven job it has no
`get_result()` or `download_artifacts()`, because an agent evaluation publishes agent-eval results
and a summary rather than row scores. Read the scores through
`client.evaluator.agent_eval_results`.

Gym jobs run on their own `nmp-gym-tasks` container image rather than the shared CPU task image,
because Gym requires Ray. No configuration is needed — the target selects it.

## Caveats

* **Per-environment dependencies are heterogeneous.** `mcqa` needs only `tiktoken`; other Gym
  environments pull in `torch`, COMET, a GPU, or Docker. Providing a Gym runtime with those installed
  is the caller's responsibility.
* **`--no-serve --input` bypasses Gym's data-prep** — prompt templating and dataset materialization.
  Rows that are already complete, like the bundled `example.jsonl`, are faithful; an environment
  whose rows need templating would need that step run first.
* **Service-side execution** — Docker or Kubernetes, Ray provisioning — is out of scope for this SDK
  path. That is the evaluator plugin's concern.

## Next steps

#### [Agent Evaluation (concepts)](/documentation/evaluate-models/agent-eval)

#### [Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners)

#### [Evaluate a Harbor Task Suite](/documentation/evaluate-models/agent-eval/harbor-runner)