Evaluate a NeMo Gym Environment

NeMo Gym is an environment framework for agentic rollouts: a resources-server provides the environment, an agent acts in it, and each rollout carries a reward. If you already have a Gym environment, the Gym runner runs it and scores its reward through agent-eval — the same AgentEvaluator and the same result and bundle as the quickstart. Only the runner changes.

Choose how to run the evaluation

There are two user-facing interfaces:

On the platform, sandbox placement is an operator decision rather than another submission mode. A sandbox-enabled deployment runs Gym in a separate nhx-gym-host. A deployment without OpenSandbox can still run trusted, built-in Gym components inside the nhx-gym-tasks job container; this compatibility path is called colocated execution. Submitters do not select between them, and custom environment FileSets always require the sandboxed path.

Gym owns execution and scoring through either interface. Both produce the same trial shape, and GymRewardMetric surfaces Gym’s per-attempt reward.

Install NeMo Gym

uv venv ~/gym-env --python 3.13.15
uv pip install --python ~/gym-env/bin/python "nemo-gym==0.5.0" tiktoken
export PATH="$HOME/gym-env/bin:$PATH"

NeMo Gym 0.5.0 requires Python 3.13.14 or newer. Pin the Gym version so the local runner matches the NeMo Helix job image. Install Gym into its own environment and put that environment’s bin on PATH. Gym imports Ray at module load, and nemo-helix excludes Ray by constraint, so the two generally cannot share a virtualenv.

The runner resolves gym from PATH only — there is deliberately no setting pointing at a checkout or another venv, because this config becomes a serialized job spec when Gym runs as a platform job, and a local path means nothing on the other side of that boundary. In a colocated job image, the image owns PATH and this resolves normally. A sandboxed job runs the CLI in a separate nhx-gym-host image instead.

Credentials for local SDK runs

Gym’s collector calls your model endpoint directly. It reads the credentials from an env.yaml in the directory you run from — this SDK never reads or handles that file:

policy_base_url: https://<your-openai-compatible-endpoint>/v1
policy_api_key: <key>
policy_model_name: <model, e.g. meta/llama-3.1-8b-instruct>

Keep it out of version control. Gym searches the working directory first, then its install root. Platform jobs do not read this local file. Configure their model route with hydra_params, and map secret environment variables through GymRunnerTarget.env_secrets. See Configure Sandboxed Gym.

The dataset

A Gym dataset is a jsonl file, one row per case. Environments ship their example data inside the nemo-gym wheel, so the bundled mcqa benchmark needs no checkout:

<site-packages>/resources_servers/mcqa/data/example.jsonl

discover_gym_tasks turns that file into tasks — one per distinct row:

from nemo_evaluator_sdk.agent_eval.runtimes.gym import GymRewardMetric, discover_gym_tasks
tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])

Task identity is the row’s content hash, which has two consequences worth knowing before you build a dataset:

• Duplicate rows collapse into a single task, and the runner warns. Duplicates usually mean a data problem.
• Repeating a row is not how you ask for repeated attempts. Use num_repeats — attempts are a run-level concern, not a dataset one.

Run it

import asyncio
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.gym import (
    GymAgentTaskRunner,
    GymRewardMetric,
    GymRuntimeConfig,
    discover_gym_tasks,
)
tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])
runner = GymAgentTaskRunner(
    config=GymRuntimeConfig(
        agent="simple_agent",
        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
        resources_server="mcqa",
        num_repeats=2,
    )
)
result = asyncio.run(AgentEvaluator().run(tasks=tasks, target=runner))
print(result.summary)

Run it from the directory holding env.yaml.

The mapping is:

• one Gym dataset → one run
• each distinct row → one task
• each attempt → one trial

So num_repeats=2 over a 5-row dataset yields 5 tasks and 10 trials.

Local runner configuration

Anything GymRuntimeConfig does not expose can go through hydra_params, which is flattened to Hydra’s override grammar and applied to gym env start. For the full set of knobs, see the NeMo Gym documentation.

Platform job specs use GymRunnerTarget, the serializable counterpart of the local GymRuntimeConfig. Submitting a runner builds one, and the deployment-specific settings come from a GymPlacement passed alongside it rather than from the runner:

• environment points to a FileSet containing Gym component configuration, code, and dependencies that are not built into the runtime image.
• agent_ref_name identifies the agent instance registered inside the sandbox when that instance name differs from the agent component name.

They live on the placement because the local gym CLI has no equivalent of either — it stages no packages and resolves its agent from Hydra config — so a runner carrying them would hold fields that do nothing wherever it actually runs. See Run a Custom Gym Environment.

Attempts and concurrency

num_repeats sets how many attempts each row gets, and every attempt becomes one trial — five rows at num_repeats=2 produce five tasks and ten trials.

Throughput has two separate knobs, and they govern different phases:

Raising parallelism does not speed up collection, and it is deliberately not mapped onto Gym’s --concurrency: scoring is SDK-side and cheap, while rollouts run against an endpoint with its own rate limits. Tune concurrency to that endpoint.

Three timeouts bound the subprocess: startup_timeout_s (default 240s) for gym env start readiness, collection_timeout_s (default None, unbounded) for the run itself, and shutdown_grace_s (default 30s) before the Gym process group is escalated from SIGTERM to SIGKILL. A collection timeout scales with dataset size x num_repeats / concurrency, so set one only once you know roughly how long a run takes.

Read the results

Gym’s reward arrives as gym_reward.reward, and Gym’s own aggregates are imported alongside the SDK’s under a runner.gym.* prefix — the prefix is what tells you which side computed a number:

for score in result.summary.scores.scores:
    print(score.name)
# gym_reward.reward
# gym_reward.reward.pass@1
# runner.gym.pass@1/accuracy
# runner.gym.input_tokens

Aggregates arrive in more than one shape, and .mean is empty on some of them. A scalar aggregate is a single number the runner already computed, held in .value; a range aggregate is a distribution, with .mean plus .min, .max, .median and sample_std_dev. Gym produces both — metrics that arrive with a complete mean/min/max/median/std family become range, and every remaining key becomes scalar — so the shape does not follow the name prefix.

Read headline_value rather than .mean, and the distinction stops mattering: it returns the one number each aggregate reports, whatever its shape.

for aggregate in result.summary.scores.scores:
    print(f"{aggregate.name}: {aggregate.headline_value}")
# gym_reward.reward: 0.8
# runner.gym.input_tokens: 216.4
# runner.gym.pass@1/accuracy: 80.0

Compute on gym_reward.reward. The SDK derives it from the same rollouts on its own 0–1 scale, and drops Gym’s duplicate of it. Treat runner.gym.* as Gym’s own reporting, kept in Gym’s units so it can be checked against Gym’s output: accuracy there is 0–100, so runner.gym.pass@1/accuracy of 80.0 is the same result as a gym_reward.reward mean of 0.8. Trials, scores, and the run bundle are otherwise read exactly as in Reading Results.

Output directories

Each run writes to a fresh temporary directory by default. To choose one, set work_dir on the run config:

from pathlib import Path
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig
result = asyncio.run(
    AgentEvaluator().run(
        tasks=tasks, target=runner, config=AgentEvalRunConfig(work_dir=Path("gym-run-1"))
    )
)

Give every run its own. The runner refuses to reuse a directory that already holds Gym rollout output, raising FileExistsError: Gym appends to its failures sidecar, so reusing one would mix two runs together, and the runner raises rather than clearing a previous run’s results.

Gym’s own artifacts land in a gym_run/ subdirectory — rollouts.jsonl, rollouts_failures.jsonl, rollouts_aggregate_metrics.json, and the materialized gym_input.jsonl handed to collection.

How local SDK and colocated platform runs execute Gym

The runner uses Gym’s two-step flow, which reads a dataset file directly — no split-driven data preparation and no HuggingFace downloads:

1. gym env start … brings up the resources-server, agent, and model servers.
2. gym eval run --no-serve --input <dataset> … collects rollouts against them.

The dataset handed to step 2 is not your source file. The runner materializes a normalized one into the run’s work directory, one row per requested task, with _ng_task_index stamped explicitly. Gym honors a caller-supplied _ng_task_index and echoes it back on every rollout record, so rollouts join back to tasks through a map the runner owns rather than a guess about Gym’s row ordering. That is also what lets you run a subset of tasks and roll out only that subset.

Logs

Gym’s subprocess output is streamed to files in the run’s work directory — gym_env.log for startup, and gym_eval.stdout.log / gym_eval.stderr.log for collection — and mirrored to the nemo_evaluator_sdk.agent_eval.runtimes.gym logger at DEBUG. Startup and collection failures name the relevant file and inline its last lines. To watch Gym’s output in your own terminal:

import logging
logging.getLogger("nemo_evaluator_sdk.agent_eval.runtimes.gym").setLevel(logging.DEBUG)

Submit as a platform job

A Gym runner can be submitted as a durable platform job from the live runner object, rather than described a second time as a job spec — the configuration you validated locally is the configuration that runs.

submit takes a stored taskset, so the Gym rows have to be stored first. A Gym taskset is not an ordinary one: the job rebuilds the Gym dataset from each task, so every task must carry the row that discover_gym_tasks split across inputs['gym_row'] and metadata['gym_row_extras']. Build the tasks with discover_gym_tasks and store both halves:

from nemo_evaluator.api.fields import MetadataItem, MetricInline
from nemo_evaluator.api.schemas import (
    EvaluatorTaskDefinition,
    TaskInput,
    TaskInputs,
    TaskRef,
    TasksetInput,
    TasksetRef,
)
from nemo_evaluator.shared.metric_bundles.bundles import bundle_metric
from nemo_evaluator.shared.metric_bundles.inline import InlineMetricBundlePackager
from nemo_evaluator_sdk.agent_eval.runtimes.gym import (
    GymAgentTaskRunner,
    GymRewardMetric,
    GymRuntimeConfig,
    discover_gym_tasks,
)
from nemo_helix_plugin.client.client import NemoClient
from nemo_evaluator.sdk import Evaluator
client = NemoClient(base_url="http://localhost:8080", workspace="default")
evaluator = Evaluator.from_client(client)
runner = GymAgentTaskRunner(
    config=GymRuntimeConfig(
        agent="simple_agent",
        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
        resources_server="mcqa",
    )
)
tasks = discover_gym_tasks("path/to/example.jsonl")
# GymRewardMetric is a built-in metric type, so it bundles inline — no cloudpickle opt-in.
reward = MetricInline.model_validate(
    bundle_metric(GymRewardMetric(), InlineMetricBundlePackager()).model_dump(mode="json")
)
names = []
for index, task in enumerate(tasks):
    name = f"mcqa-{index}"
    names.append(name)
    evaluator.tasks.create(
        name,
        task=TaskInput(
            spec=EvaluatorTaskDefinition(
                kind="evaluator",
                intent=task.intent,
                inputs=TaskInputs(**task.inputs),
                metrics=[reward],
            ),
            metadata=[MetadataItem(key=key, value=value) for key, value in task.metadata.items()],
        ),
    )
evaluator.tasksets.create("mcqa-suite", taskset=TasksetInput(tasks=[TaskRef(n) for n in names]))
job = evaluator.submit(tasks=TasksetRef("mcqa-suite"), target=runner)
job.wait_until_done()

Two things that bite here:

• Do not name the task after task.id. It is a 64-character hex digest, already past the 63-character cap on entity names, and it may begin with a digit where a name must start with a letter. Derive a name, as above; the digest stays the task’s own id.
• gym_row rides on inputs and gym_row_extras on the task’s metadata — the field beside spec, not inside it. A task missing either is rejected job-side with task '<id>' is missing inputs['gym_row'] and/or metadata['gym_row_extras'].

A value with no JSON form — a callable in hydra_params, say — is refused with UnsubmittableRunnerError at submit time, rather than failing inside the transport with an error that names neither the runner nor the field.

The returned handle is an AgentEvaluatorJobResource. Unlike a dataset-driven job it has no get_result() or download_artifacts(). Wait for completion, then read the queryable record through evaluator.agent_eval_results or download the named agent-eval-results job result. See Reading Results.

The async client takes the same call — await async_evaluator.submit(tasks=..., target=runner) on an AsyncEvaluator, returning an AsyncAgentEvaluatorJobResource with the same status and polling methods. See SDK Resources.

Submitting from the live runner is not the only route. The same evaluation can be described as an agent-evaluate spec with a GymRunnerTarget and submitted with the CLI, which is how a job that stages a custom environment is usually run:

nemo evaluator agent-evaluate --spec-file gym_agent_eval.json

Run a Custom Gym Environment has a worked spec.

On a sandbox-enabled deployment, the Evaluator step runs in nhx-tasks and calls a separate nhx-gym-host. On a deployment using the compatibility path, Evaluator and Gym run together in nhx-gym-tasks. A target with an environment FileSet is rejected on the compatibility path and adds stage-environment before agent-evaluate on a sandbox-enabled deployment.

Caveats

• Per-environment dependencies are heterogeneous. mcqa needs only tiktoken; other Gym environments pull in torch, COMET, a GPU, or Docker. Providing a Gym runtime with those installed is the caller’s responsibility.
• --no-serve --input bypasses Gym’s data-prep — prompt templating and dataset materialization. Rows that are already complete, like the bundled example.jsonl, are faithful; an environment whose rows need templating would need that step run first.
• Custom environments require platform support. Package the environment as a FileSet and use sandboxed execution; environment and agent_ref_name are GymPlacement fields supplied at submission, so a live GymAgentTaskRunner cannot carry them. env_secrets is not in that group — it lives on GymRuntimeConfig and resolves locally from your environment, so it works the same way in both places.
• No retries and no re-run cache. A failed rollout is not retried, and nothing is reused between runs: every run collects from scratch, and refuses to run into a directory that already holds Gym rollout output — rollouts, the failures sidecar, or model-call captures. (An existing empty directory is fine.) Re-running a large suite pays the full collection cost again. (The Harbor runner does have both — do not carry that expectation across.)

Next steps