Evaluate a NeMo Gym Environment

NeMo Gym is an environment framework for agentic rollouts: a resources-server provides the environment, an agent acts in it, and each rollout carries a reward. If you already have a Gym environment, the Gym runner runs it and scores its reward through agent-eval — the same AgentEvaluator and the same result and bundle as the quickstart. Only the runner changes.

Choose how to run the evaluation

There are two user-facing interfaces:

On the platform, sandbox placement is an operator decision rather than another submission mode. A sandbox-enabled deployment runs Gym in a separate nmp-gym-host. A deployment without OpenSandbox can still run trusted, built-in Gym components inside the nmp-gym-tasks job container; this compatibility path is called colocated execution. Submitters do not select between them, and custom environment FileSets always require the sandboxed path.

Gym owns execution and scoring through either interface. Both produce the same trial shape, and GymRewardMetric surfaces Gym’s per-attempt reward.

uv venv ~/gym-env --python 3.12
uv pip install --python ~/gym-env/bin/python nemo-gym tiktoken
export PATH="$HOME/gym-env/bin:$PATH"

Credentials for local SDK runs

Gym’s collector calls your model endpoint directly. It reads the credentials from an env.yaml in the directory you run from — this SDK never reads or handles that file:

policy_base_url: https://<your-openai-compatible-endpoint>/v1
policy_api_key: <key>
policy_model_name: <model, e.g. meta/llama-3.1-8b-instruct>

Keep it out of version control. Gym searches the working directory first, then its install root. Platform jobs do not read this local file. Configure their model route with hydra_params, and map secret environment variables through GymRunnerTarget.env_secrets. See Configure Sandboxed Gym.

The dataset

A Gym dataset is a jsonl file, one row per case. Environments ship their example data inside the nemo-gym wheel, so the bundled mcqa benchmark needs no checkout:

<site-packages>/resources_servers/mcqa/data/example.jsonl

discover_gym_tasks turns that file into tasks — one per distinct row:

from nemo_evaluator_sdk.agent_eval.runtimes.gym import GymRewardMetric, discover_gym_tasks
tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])

Task identity is the row’s content hash, which has two consequences worth knowing before you build a dataset:

• Duplicate rows collapse into a single task, and the runner warns. Duplicates usually mean a data problem.
• Repeating a row is not how you ask for repeated attempts. Use num_repeats — attempts are a run-level concern, not a dataset one.

Run it

import asyncio
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.gym import (
    GymAgentTaskRunner,
    GymRewardMetric,
    GymRuntimeConfig,
    discover_gym_tasks,
)
tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])
runner = GymAgentTaskRunner(
    config=GymRuntimeConfig(
        agent="simple_agent",
        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
        resources_server="mcqa",
        num_repeats=2,
    )
)
result = asyncio.run(AgentEvaluator().run(tasks=tasks, target=runner))
print(result.summary)

Run it from the directory holding env.yaml.

The mapping is:

• one Gym dataset → one run
• each distinct row → one task
• each attempt → one trial

So num_repeats=2 over a 5-row dataset yields 5 tasks and 10 trials.

Local runner configuration

Anything GymRuntimeConfig does not expose can go through hydra_params, which is flattened to Hydra’s override grammar and applied to gym env start. For the full set of knobs, see the NeMo Gym documentation.

Platform job specs use GymRunnerTarget, the serializable counterpart of the local GymRuntimeConfig. Submitting a runner builds one, and the deployment-specific settings come from a GymPlacement passed alongside it rather than from the runner:

• environment points to a FileSet containing Gym component configuration, code, and dependencies that are not built into the runtime image.
• agent_ref_name identifies the agent instance registered inside the sandbox when that instance name differs from the agent component name.

They live on the placement because the local gym CLI has no equivalent of either — it stages no packages and resolves its agent from Hydra config — so a runner carrying them would hold fields that do nothing wherever it actually runs. See Run a Custom Gym Environment.

Read the results

Gym’s reward arrives as gym_reward.reward, and Gym’s own aggregates are imported alongside the SDK’s under a runner.gym.* prefix — the prefix is what tells you which side computed a number:

for score in result.summary.scores.scores:
    print(score.name)
# gym_reward.reward
# gym_reward.reward.pass@1
# runner.gym.pass@1/accuracy
# runner.gym.input_tokens

Gym reports accuracy on a 0–100 scale where the SDK uses 0–1, so runner.gym.pass@1/accuracy of 50.0 corresponds to a gym_reward.reward mean of 0.5. Trials, scores, and the run bundle are otherwise read exactly as in Reading Results.

Output directories

Each run writes to a fresh temporary directory by default. To choose one, set work_dir on the run config:

from pathlib import Path
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig
result = asyncio.run(
    AgentEvaluator().run(
        tasks=tasks, target=runner, config=AgentEvalRunConfig(work_dir=Path("gym-run-1"))
    )
)

Give every run its own. The runner refuses to reuse a directory that already holds Gym rollout output, raising FileExistsError: Gym appends to its failures sidecar, so reusing one would mix two runs together, and the runner raises rather than clearing a previous run’s results.

Gym’s own artifacts land in a gym_run/ subdirectory — rollouts.jsonl, rollouts_failures.jsonl, rollouts_aggregate_metrics.json, and the materialized gym_input.jsonl handed to collection.

How local SDK and colocated platform runs execute Gym

The runner uses Gym’s two-step flow, which reads a dataset file directly — no split-driven data preparation and no HuggingFace downloads:

1. gym env start … brings up the resources-server, agent, and model servers.
2. gym eval run --no-serve --input <dataset> … collects rollouts against them.

The dataset handed to step 2 is not your source file. The runner materializes a normalized one into the run’s work directory, one row per requested task, with _ng_task_index stamped explicitly. Gym honors a caller-supplied _ng_task_index and echoes it back on every rollout record, so rollouts join back to tasks through a map the runner owns rather than a guess about Gym’s row ordering. That is also what lets you run a subset of tasks and roll out only that subset.

Logs

Gym’s subprocess output is streamed to files in the run’s work directory — gym_env.log for startup, and gym_eval.stdout.log / gym_eval.stderr.log for collection — and mirrored to the nemo_evaluator_sdk.agent_eval.runtimes.gym logger at DEBUG. Startup and collection failures name the relevant file and inline its last lines. To watch Gym’s output in your own terminal:

import logging
logging.getLogger("nemo_evaluator_sdk.agent_eval.runtimes.gym").setLevel(logging.DEBUG)

Submit as a platform job

A Gym runner can be submitted as a durable platform job from the live runner object, rather than described a second time as a job spec — the configuration you validated locally is the configuration that runs.

submit takes a stored taskset, so the Gym rows have to be stored first. A Gym taskset is not an ordinary one: the job rebuilds the Gym dataset from each task, so every task must carry the row that discover_gym_tasks split across inputs['gym_row'] and metadata['gym_row_extras']. Build the tasks with discover_gym_tasks and store both halves:

from nemo_evaluator.api.fields import MetadataItem, MetricInline
from nemo_evaluator.api.schemas import (
    EvaluatorTaskDefinition,
    TaskInput,
    TaskInputs,
    TaskRef,
    TasksetInput,
    TasksetRef,
)
from nemo_evaluator.shared.metric_bundles.bundles import bundle_metric
from nemo_evaluator.shared.metric_bundles.inline import InlineMetricBundlePackager
from nemo_evaluator_sdk.agent_eval.runtimes.gym import (
    GymAgentTaskRunner,
    GymRewardMetric,
    GymRuntimeConfig,
    discover_gym_tasks,
)
from nemo_platform import NeMoPlatform
client = NeMoPlatform(base_url="http://localhost:8080", workspace="default")
runner = GymAgentTaskRunner(
    config=GymRuntimeConfig(
        agent="simple_agent",
        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
        resources_server="mcqa",
    )
)
tasks = discover_gym_tasks("path/to/example.jsonl")
# GymRewardMetric is a built-in metric type, so it bundles inline — no cloudpickle opt-in.
reward = MetricInline.model_validate(
    bundle_metric(GymRewardMetric(), InlineMetricBundlePackager()).model_dump(mode="json")
)
names = []
for index, task in enumerate(tasks):
    name = f"mcqa-{index}"
    names.append(name)
    client.evaluator.tasks.create(
        name,
        task=TaskInput(
            spec=EvaluatorTaskDefinition(
                kind="evaluator",
                intent=task.intent,
                inputs=TaskInputs(**task.inputs),
                metrics=[reward],
            ),
            metadata=[MetadataItem(key=key, value=value) for key, value in task.metadata.items()],
        ),
    )
client.evaluator.tasksets.create("mcqa-suite", taskset=TasksetInput(tasks=[TaskRef(n) for n in names]))
job = client.evaluator.submit(tasks=TasksetRef("mcqa-suite"), target=runner)
job.wait_until_done()

Two things that bite here:

• Do not name the task after task.id. It is a 64-character hex digest, already past the 63-character cap on entity names, and it may begin with a digit where a name must start with a letter. Derive a name, as above; the digest stays the task’s own id.
• gym_row rides on inputs and gym_row_extras on the task’s metadata — the field beside spec, not inside it. A task missing either is rejected job-side with task '<id>' is missing inputs['gym_row'] and/or metadata['gym_row_extras'].

A value with no JSON form — a callable in hydra_params, say — is refused with UnsubmittableRunnerError at submit time, rather than failing inside the transport with an error that names neither the runner nor the field.

The returned handle is an AgentEvaluatorJobResource. Unlike a dataset-driven job it has no get_result() or download_artifacts(). Wait for completion, then read the queryable record through client.evaluator.agent_eval_results or download the named agent-eval-results job result. See Reading Results.

On a sandbox-enabled deployment, the Evaluator step runs in nmp-cpu-tasks and calls a separate nmp-gym-host. On a deployment using the compatibility path, Evaluator and Gym run together in nmp-gym-tasks. A target with an environment FileSet is rejected on the compatibility path and adds stage-environment before agent-evaluate on a sandbox-enabled deployment.

Caveats

• Per-environment dependencies are heterogeneous. mcqa needs only tiktoken; other Gym environments pull in torch, COMET, a GPU, or Docker. Providing a Gym runtime with those installed is the caller’s responsibility.
• --no-serve --input bypasses Gym’s data-prep — prompt templating and dataset materialization. Rows that are already complete, like the bundled example.jsonl, are faithful; an environment whose rows need templating would need that step run first.
• Custom environments require platform support. Package the environment as a FileSet and use sandboxed execution; a live GymAgentTaskRunner cannot carry environment, agent_ref_name, or env_secrets.

Next steps