Evaluate a NeMo Gym Environment

NeMo Gym is an environment framework for agentic rollouts: a resources-server provides the environment, an agent acts in it, and each rollout carries a reward. If you already have a Gym environment, the Gym runner runs it and scores its reward through agent-eval — the same AgentEvaluator and the same result and bundle as the quickstart. Only the runner changes.

Gym owns execution and scoring here. The runner shells out to the gym CLI and adapts the rollout bundle into trials; GymRewardMetric surfaces Gym’s per-attempt reward.

$
uv venv ~/gym-env --python 3.12
$
uv pip install --python ~/gym-env/bin/python nemo-gym tiktoken
$
export PATH="$HOME/gym-env/bin:$PATH"

Credentials

Gym’s collector calls your model endpoint directly. It reads the credentials from an env.yaml in the directory you run from — this SDK never reads or handles that file:

1
policy_base_url: https://<your-openai-compatible-endpoint>/v1
2
policy_api_key: <key>
3
policy_model_name: <model, e.g. meta/llama-3.1-8b-instruct>

Keep it out of version control. Gym searches the working directory first, then its install root.

The dataset

A Gym dataset is a jsonl file, one row per case. Environments ship their example data inside the nemo-gym wheel, so the bundled mcqa benchmark needs no checkout:

<site-packages>/resources_servers/mcqa/data/example.jsonl

discover_gym_tasks turns that file into tasks — one per distinct row:

1
from nemo_evaluator_sdk.agent_eval.runtimes.gym import GymRewardMetric, discover_gym_tasks
2
3
tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])

Task identity is the row’s content hash, which has two consequences worth knowing before you build a dataset:

• Duplicate rows collapse into a single task, and the runner warns. Duplicates usually mean a data problem.
• Repeating a row is not how you ask for repeated attempts. Use num_repeats — attempts are a run-level concern, not a dataset one.

Run it

1
import asyncio
2
3
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
4
from nemo_evaluator_sdk.agent_eval.runtimes.gym import (
5
    GymAgentTaskRunner,
6
    GymRewardMetric,
7
    GymRuntimeConfig,
8
    discover_gym_tasks,
9
)
10
11
tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])
12
13
runner = GymAgentTaskRunner(
14
    config=GymRuntimeConfig(
15
        agent="simple_agent",
16
        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
17
        resources_server="mcqa",
18
        num_repeats=2,
19
    )
20
)
21
22
result = asyncio.run(AgentEvaluator().run(tasks=tasks, target=runner))
23
print(result.summary)

Run it from the directory holding env.yaml.

The mapping is:

• one Gym dataset → one run
• each distinct row → one task
• each attempt → one trial

So num_repeats=2 over a 5-row dataset yields 5 tasks and 10 trials.

Configuration

Anything GymRuntimeConfig does not expose can go through hydra_params, which is flattened to Hydra’s override grammar and applied to gym env start. For the full set of knobs, see the NeMo Gym documentation.

Read the results

Gym’s reward arrives as gym_reward.reward, and Gym’s own aggregates are imported alongside the SDK’s under a runner.gym.* prefix — the prefix is what tells you which side computed a number:

1
for score in result.summary.scores.scores:
2
    print(score.name)
3
# gym_reward.reward
4
# gym_reward.reward.pass@1
5
# runner.gym.pass@1/accuracy
6
# runner.gym.input_tokens

Gym reports accuracy on a 0–100 scale where the SDK uses 0–1, so runner.gym.pass@1/accuracy of 50.0 corresponds to a gym_reward.reward mean of 0.5. Trials, scores, and the run bundle are otherwise read exactly as in Reading Results.

Output directories

Each run writes to a fresh temporary directory by default. To choose one, set work_dir on the run config:

1
from pathlib import Path
2
3
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig
4
5
result = asyncio.run(
6
    AgentEvaluator().run(
7
        tasks=tasks, target=runner, config=AgentEvalRunConfig(work_dir=Path("gym-run-1"))
8
    )
9
)

Give every run its own. The runner refuses to reuse a directory that already holds Gym rollout output, raising FileExistsError: Gym appends to its failures sidecar, so reusing one would mix two runs together, and the runner raises rather than clearing a previous run’s results.

Gym’s own artifacts land in a gym_run/ subdirectory — rollouts.jsonl, rollouts_failures.jsonl, rollouts_aggregate_metrics.json, and the materialized gym_input.jsonl handed to collection.

How it runs Gym

The runner uses Gym’s two-step flow, which reads a dataset file directly — no split-driven data preparation and no HuggingFace downloads:

1. gym env start … brings up the resources-server, agent, and model servers.
2. gym eval run --no-serve --input <dataset> … collects rollouts against them.

The dataset handed to step 2 is not your source file. The runner materializes a normalized one into the run’s work directory, one row per requested task, with _ng_task_index stamped explicitly. Gym honors a caller-supplied _ng_task_index and echoes it back on every rollout record, so rollouts join back to tasks through a map the runner owns rather than a guess about Gym’s row ordering. That is also what lets you run a subset of tasks and roll out only that subset.

Logs

Gym’s subprocess output is streamed to files in the run’s work directory — gym_env.log for startup, and gym_eval.stdout.log / gym_eval.stderr.log for collection — and mirrored to the nemo_evaluator_sdk.agent_eval.runtimes.gym logger at DEBUG. Startup and collection failures name the relevant file and inline its last lines. To watch Gym’s output in your own terminal:

1
import logging
2
3
logging.getLogger("nemo_evaluator_sdk.agent_eval.runtimes.gym").setLevel(logging.DEBUG)

Submit as a platform job

A Gym runner can be submitted as a durable platform job from the live runner object, rather than described a second time as a job spec — the configuration you validated locally is the configuration that runs.

submit takes a stored taskset, so the Gym rows have to be stored first. A Gym taskset is not an ordinary one: the job rebuilds the Gym dataset from each task, so every task must carry the row that discover_gym_tasks split across inputs['gym_row'] and metadata['gym_row_extras']. Build the tasks with discover_gym_tasks and store both halves:

1
from nemo_evaluator.api.fields import MetadataItem, MetricInline
2
from nemo_evaluator.api.schemas import (
3
    EvaluatorTaskDefinition,
4
    TaskInput,
5
    TaskInputs,
6
    TaskRef,
7
    TasksetInput,
8
    TasksetRef,
9
)
10
from nemo_evaluator.shared.metric_bundles.bundles import bundle_metric
11
from nemo_evaluator.shared.metric_bundles.inline import InlineMetricBundlePackager
12
from nemo_evaluator_sdk.agent_eval.runtimes.gym import (
13
    GymAgentTaskRunner,
14
    GymRewardMetric,
15
    GymRuntimeConfig,
16
    discover_gym_tasks,
17
)
18
from nemo_platform import NeMoPlatform
19
20
client = NeMoPlatform(base_url="http://localhost:8080", workspace="default")
21
22
runner = GymAgentTaskRunner(
23
    config=GymRuntimeConfig(
24
        agent="simple_agent",
25
        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
26
        resources_server="mcqa",
27
    )
28
)
29
30
tasks = discover_gym_tasks("path/to/example.jsonl")
31
32
# GymRewardMetric is a built-in metric type, so it bundles inline — no cloudpickle opt-in.
33
reward = MetricInline.model_validate(
34
    bundle_metric(GymRewardMetric(), InlineMetricBundlePackager()).model_dump(mode="json")
35
)
36
37
names = []
38
for index, task in enumerate(tasks):
39
    name = f"mcqa-{index}"
40
    names.append(name)
41
    client.evaluator.tasks.create(
42
        name,
43
        task=TaskInput(
44
            spec=EvaluatorTaskDefinition(
45
                kind="evaluator",
46
                intent=task.intent,
47
                inputs=TaskInputs(**task.inputs),
48
                metrics=[reward],
49
            ),
50
            metadata=[MetadataItem(key=key, value=value) for key, value in task.metadata.items()],
51
        ),
52
    )
53
54
client.evaluator.tasksets.create("mcqa-suite", taskset=TasksetInput(tasks=[TaskRef(n) for n in names]))
55
56
job = client.evaluator.submit(tasks=TasksetRef("mcqa-suite"), target=runner)
57
job.wait_until_done()

Two things that bite here:

• Do not name the task after task.id. It is a 64-character hex digest, already past the 63-character cap on entity names, and it may begin with a digit where a name must start with a letter. Derive a name, as above; the digest stays the task’s own id.
• gym_row rides on inputs and gym_row_extras on the task’s metadata — the field beside spec, not inside it. A task missing either is rejected job-side with task '<id>' is missing inputs['gym_row'] and/or metadata['gym_row_extras'].

A value with no JSON form — a callable in hydra_params, say — is refused with UnsubmittableRunnerError at submit time, rather than failing inside the transport with an error that names neither the runner nor the field.

The returned handle is an AgentEvaluatorJobResource. Unlike a dataset-driven job it has no get_result() or download_artifacts(), because an agent evaluation publishes agent-eval results and a summary rather than row scores. Read the scores through client.evaluator.agent_eval_results.

Gym jobs run on their own nmp-gym-tasks container image rather than the shared CPU task image, because Gym requires Ray. No configuration is needed — the target selects it.

Caveats

• Per-environment dependencies are heterogeneous. mcqa needs only tiktoken; other Gym environments pull in torch, COMET, a GPU, or Docker. Providing a Gym runtime with those installed is the caller’s responsibility.
• --no-serve --input bypasses Gym’s data-prep — prompt templating and dataset materialization. Rows that are already complete, like the bundled example.jsonl, are faithful; an environment whose rows need templating would need that step run first.
• Service-side execution — Docker or Kubernetes, Ray provisioning — is out of scope for this SDK path. That is the evaluator plugin’s concern.

Next steps