Evaluate a NeMo Gym Environment

NeMo Gym is an environment framework for agentic rollouts: a resources-server provides the environment, an agent acts in it, and each rollout carries a reward. If you already have a Gym environment, the Gym runner runs it and scores its reward through agent-eval — the same AgentEvaluator and the same result and bundle as the quickstart. Only the runner changes.

Choose how to run the evaluation

There are two user-facing interfaces:

On the platform, sandbox placement is an operator decision rather than another submission mode. A sandbox-enabled deployment runs Gym in a separate nhx-gym-host. A deployment without OpenSandbox can still run trusted, built-in Gym components inside the nhx-gym-tasks job container; this compatibility path is called colocated execution. Submitters do not select between them, and custom environment FileSets always require the sandboxed path.

Gym owns execution and scoring through either interface. Both produce the same trial shape, and GymRewardMetric surfaces Gym’s per-attempt reward.

Install NeMo Gym

uv venv ~/gym-env --python 3.13.15
uv pip install --python ~/gym-env/bin/python "nemo-gym==0.5.0" tiktoken
export PATH="$HOME/gym-env/bin:$PATH"

NeMo Gym 0.5.0 requires Python 3.13.14 or newer. Pin the Gym version so the local runner matches the NeMo Helix job image. Install Gym into its own environment and put that environment’s bin on PATH. Gym imports Ray at module load, and nemo-helix excludes Ray by constraint, so the two generally cannot share a virtualenv.

The runner resolves gym from PATH only — there is deliberately no setting pointing at a checkout or another venv, because this config becomes a serialized job spec when Gym runs as a platform job, and a local path means nothing on the other side of that boundary. In a colocated job image, the image owns PATH and this resolves normally. A sandboxed job runs the CLI in a separate nhx-gym-host image instead.

Credentials for local SDK runs

Gym’s collector calls your model endpoint directly. It reads the credentials from an env.yaml in the directory you run from — this SDK never reads or handles that file:

policy_base_url: https://<your-openai-compatible-endpoint>/v1
policy_api_key: <key>
policy_model_name: <model, e.g. meta/llama-3.1-8b-instruct>

Keep it out of version control. Gym searches the working directory first, then its install root. Platform jobs do not read this local file. Configure their model route with hydra_params, and map secret environment variables through GymRunnerTarget.env_secrets. See Configure Sandboxed Gym.

The dataset

A Gym dataset is a jsonl file, one row per case. Environments ship their example data inside the nemo-gym wheel, so the bundled mcqa benchmark needs no checkout:

<site-packages>/resources_servers/mcqa/data/example.jsonl

discover_gym_tasks turns that file into tasks — one per distinct row:

from nhx_evals_sdk.agent_eval.runtimes.gym import GymRewardMetric, discover_gym_tasks
tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])

Task identity is the row’s content hash, which has two consequences worth knowing before you build a dataset:

• Duplicate rows collapse into a single task, and the runner warns. Duplicates usually mean a data problem.
• Repeating a row is not how you ask for repeated attempts. Use num_repeats — attempts are a run-level concern, not a dataset one.

Run it

import asyncio
from nhx_evals_sdk.agent_eval.evaluator import AgentEvaluator
from nhx_evals_sdk.agent_eval.runtimes.gym import (
    GymAgentTaskRunner,
    GymRewardMetric,
    GymRuntimeConfig,
    discover_gym_tasks,
)
tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()])
runner = GymAgentTaskRunner(
    config=GymRuntimeConfig(
        agent="simple_agent",
        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
        resources_server="mcqa",
        num_repeats=2,
    )
)
result = asyncio.run(AgentEvaluator().run(tasks=tasks, target=runner))
print(result.summary)

Run it from the directory holding env.yaml.

The mapping is:

• one Gym dataset → one run
• each distinct row → one task
• each attempt → one trial

So num_repeats=2 over a 5-row dataset yields 5 tasks and 10 trials.

Local runner configuration

Anything GymRuntimeConfig does not expose can go through hydra_params, which is flattened to Hydra’s override grammar and applied to gym env start. For the full set of knobs, see the NeMo Gym documentation.

Platform job specs use GymRunnerTarget, the serializable counterpart of the local GymRuntimeConfig. Submitting a runner builds one, and the deployment-specific settings come from a GymPlacement passed alongside it rather than from the runner:

• environment points to a FileSet containing Gym component configuration, code, and dependencies that are not built into the runtime image.
• agent_ref_name identifies the agent instance registered inside the sandbox when that instance name differs from the agent component name.

On the target these arrive as one field: source holds the agent as {"component": ..., "config": ..., "instance": ...}, so a spec names exactly one agent, and a hand-written spec gives source directly:

{"kind": "gym", "source": {"component": "simple_agent", "config": "responses_api_agents/simple_agent/configs/simple_agent.yaml"}, "resources_server": "mcqa"}

They live on the placement because the local gym CLI has no equivalent of either — it stages no packages and resolves its agent from Hydra config — so a runner carrying them would hold fields that do nothing wherever it actually runs. See Run a Custom Gym Environment.

Attempts and concurrency

num_repeats sets how many attempts each row gets, and every attempt becomes one trial — five rows at num_repeats=2 produce five tasks and ten trials.

Throughput has two separate knobs, and they govern different phases:

Raising parallelism does not speed up collection, and it is deliberately not mapped onto Gym’s --concurrency: scoring is SDK-side and cheap, while rollouts run against an endpoint with its own rate limits. Tune concurrency to that endpoint.

Three timeouts bound the subprocess: startup_timeout_s (default 240s) for gym env start readiness, collection_timeout_s (default None, unbounded) for the run itself, and shutdown_grace_s (default 30s) before the Gym process group is escalated from SIGTERM to SIGKILL. A collection timeout scales with dataset size x num_repeats / concurrency, so set one only once you know roughly how long a run takes.

Read the results

Gym’s reward arrives as gym_reward.reward, and Gym’s own aggregates are imported alongside the SDK’s under a runner.gym.* prefix — the prefix is what tells you which side computed a number:

for score in result.summary.scores.scores:
    print(score.name)
# gym_reward.reward
# gym_reward.reward.pass@1
# runner.gym.pass@1/accuracy
# runner.gym.input_tokens

Aggregates arrive in more than one shape, and .mean is empty on some of them. A scalar aggregate is a single number the runner already computed, held in .value; a range aggregate is a distribution, with .mean plus .min, .max, .median and sample_std_dev. Gym produces both — metrics that arrive with a complete mean/min/max/median/std family become range, and every remaining key becomes scalar — so the shape does not follow the name prefix.

Read headline_value rather than .mean, and the distinction stops mattering: it returns the one number each aggregate reports, whatever its shape.

for aggregate in result.summary.scores.scores:
    print(f"{aggregate.name}: {aggregate.headline_value}")
# gym_reward.reward: 0.8
# runner.gym.input_tokens: 216.4
# runner.gym.pass@1/accuracy: 80.0

Compute on gym_reward.reward. The SDK derives it from the same rollouts on its own 0–1 scale, and drops Gym’s duplicate of it. Treat runner.gym.* as Gym’s own reporting, kept in Gym’s units so it can be checked against Gym’s output: accuracy there is 0–100, so runner.gym.pass@1/accuracy of 80.0 is the same result as a gym_reward.reward mean of 0.8. Trials, scores, and the run bundle are otherwise read exactly as in Reading Results.

Output directories

Each run writes to a fresh temporary directory by default. To choose one, set work_dir on the run config:

from pathlib import Path
from nhx_evals_sdk.agent_eval.tasks import AgentEvalRunConfig
result = asyncio.run(
    AgentEvaluator().run(
        tasks=tasks, target=runner, config=AgentEvalRunConfig(work_dir=Path("gym-run-1"))
    )
)

Give every run its own. The runner refuses to reuse a directory that already holds Gym rollout output, raising FileExistsError: Gym appends to its failures sidecar, so reusing one would mix two runs together, and the runner raises rather than clearing a previous run’s results.

Gym’s own artifacts land in a gym_run/ subdirectory — rollouts.jsonl, rollouts_failures.jsonl, rollouts_aggregate_metrics.json, and the materialized gym_input.jsonl handed to collection.

How local SDK and colocated platform runs execute Gym

The runner uses Gym’s two-step flow, which reads a dataset file directly — no split-driven data preparation and no HuggingFace downloads:

1. gym env start … brings up the resources-server, agent, and model servers.
2. gym eval run --no-serve --input <dataset> … collects rollouts against them.

The dataset handed to step 2 is not your source file. The runner materializes a normalized one into the run’s work directory, one row per requested task, with _ng_task_index stamped explicitly. Gym honors a caller-supplied _ng_task_index and echoes it back on every rollout record, so rollouts join back to tasks through a map the runner owns rather than a guess about Gym’s row ordering. That is also what lets you run a subset of tasks and roll out only that subset.

Logs

Gym’s subprocess output is streamed to files in the run’s work directory — gym_env.log for startup, and gym_eval.stdout.log / gym_eval.stderr.log for collection — and mirrored to the nhx_evals_sdk.agent_eval.runtimes.gym logger at DEBUG. Startup and collection failures name the relevant file and inline its last lines. To watch Gym’s output in your own terminal:

import logging
logging.getLogger("nhx_evals_sdk.agent_eval.runtimes.gym").setLevel(logging.DEBUG)

Submit as a platform job

A Gym runner can be submitted as a durable platform job from the live runner object, rather than described a second time as a job spec — the configuration you validated locally is the configuration that runs.

submit takes a stored taskset, so the Gym rows have to be stored first. A Gym taskset is not an ordinary one: the job rebuilds the Gym dataset from each task, so every task must carry the row that discover_gym_tasks split across inputs['gym_row'] and metadata['gym_row_extras']. Build the tasks with discover_gym_tasks and store both halves:

from nemo_evals.api.fields import MetadataItem, MetricInline
from nemo_evals.api.schemas import (
    EvaluatorTaskDefinition,
    TaskInput,
    TaskInputs,
    TaskRef,
    TasksetInput,
    TasksetRef,
)
from nemo_evals.shared.metric_bundles.bundles import bundle_metric
from nemo_evals.shared.metric_bundles.inline import InlineMetricBundlePackager
from nhx_evals_sdk.agent_eval.runtimes.gym import (
    GymAgentTaskRunner,
    GymRewardMetric,
    GymRuntimeConfig,
    discover_gym_tasks,
)
from nemo_helix_plugin.client.client import NemoClient
from nemo_evals.sdk import Evaluator
client = NemoClient(base_url="http://localhost:8080", workspace="default")
evaluator = Evaluator.from_client(client)
runner = GymAgentTaskRunner(
    config=GymRuntimeConfig(
        agent="simple_agent",
        agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml",
        resources_server="mcqa",
    )
)
tasks = discover_gym_tasks("path/to/example.jsonl")
# GymRewardMetric is a built-in metric type, so it bundles inline — no cloudpickle opt-in.
reward = MetricInline.model_validate(
    bundle_metric(GymRewardMetric(), InlineMetricBundlePackager()).model_dump(mode="json")
)
names = []
for index, task in enumerate(tasks):
    name = f"mcqa-{index}"
    names.append(name)
    evaluator.tasks.create(
        name,
        task=TaskInput(
            spec=EvaluatorTaskDefinition(
                kind="evaluator",
                intent=task.intent,
                inputs=TaskInputs(**task.inputs),
                metrics=[reward],
            ),
            metadata=[MetadataItem(key=key, value=value) for key, value in task.metadata.items()],
        ),
    )
evaluator.tasksets.create("mcqa-suite", taskset=TasksetInput(tasks=[TaskRef(n) for n in names]))
job = evaluator.submit(tasks=TasksetRef("mcqa-suite"), target=runner)
job.wait_until_done()

Two things that bite here:

• Do not name the task after task.id. It is a 64-character hex digest, already past the 63-character cap on entity names, and it may begin with a digit where a name must start with a letter. Derive a name, as above; the digest stays the task’s own id.
• gym_row rides on inputs and gym_row_extras on the task’s metadata — the field beside spec, not inside it. A task missing either is rejected job-side with task '<id>' is missing inputs['gym_row'] and/or metadata['gym_row_extras'].

A value with no JSON form — a callable in hydra_params, say — is refused with UnsubmittableRunnerError at submit time, rather than failing inside the transport with an error that names neither the runner nor the field.

The returned handle is an AgentEvaluatorJobResource. Unlike a dataset-driven job it has no get_result() or download_artifacts(). Wait for completion, then read the queryable record through evaluator.agent_eval_results or download the named agent-eval-results job result. See Reading Results.

The async client takes the same call — await async_evaluator.submit(tasks=..., target=runner) on an AsyncEvaluator, returning an AsyncAgentEvaluatorJobResource with the same status and polling methods. See SDK Resources.

Submitting from the live runner is not the only route. The same evaluation can be described as an agent-evaluate spec with a GymRunnerTarget and submitted with the CLI, which is how a job that stages a custom environment is usually run:

nemo evals agent-evaluate --spec-file gym_agent_eval.json

Run a Custom Gym Environment has a worked spec.

On a sandbox-enabled deployment, the Evaluator step runs in nhx-tasks and calls a separate nhx-gym-host. On a deployment using the compatibility path, Evaluator and Gym run together in nhx-gym-tasks. A target with an environment FileSet is rejected on the compatibility path and adds stage-environment before agent-evaluate on a sandbox-enabled deployment.

Run a registered agent

An agent registered on the platform with nemo agents create can be the Gym target’s source, in place of a Gym component:

{
  "tasks": "default/my-gym-taskset",
  "target": {
    "kind": "gym",
    "source": {"agent": "calculator-agent"},
    "resources_server": "mcqa",
    "num_repeats": 2
  }
}

At submit time the service resolves the agent exactly as nemo agents deploy would (models bound to the workspace’s Inference Gateway, the optional source.environment merged on top) and keeps the translated config on the target as resolved_config. Gym’s policy_* settings default to the agent’s default model, so the policy model server proxies to the same model the agent would call when deployed; set them in hydra_params to point it elsewhere.

{
  "tasks": "default/my-gym-taskset",
  "target": {
    "kind": "gym",
    "source": {
      "agent": "calculator-agent",
      "environment": {
        "mcp": {"calculator": {"url": "http://mock-calculator:8000/mcp"}},
        "env": {"CALC_MODE": "strict"},
        "secrets": {"CALC_TOKEN": "calc-token"}
      }
    },
    "resources_server": "mcqa",
    "hydra_params": {
      "policy_base_url": "http://host.docker.internal:8080/apis/inference-gateway/v2/workspaces/default/openai/-/v1"
    }
  }
}

environment.mcp.calculator.url replaces the URL of the calculator server the agent declares (a server it does not declare is rejected), env is plain process environment for the component, and secrets binds an environment variable to a Secrets entry by name. hydra_params.policy_base_url is the laptop case: the policy model server runs inside the sandbox, so it reaches the gateway through host.docker.internal. A value you set in hydra_params always wins over the default derived from the agent.

The job’s stage-environment step then assembles a wheels-v1 environment package: the platform’s Gym agent component, an agent-instance config carrying the resolved config, the agent’s Ethos files (skills and prompts it refers to by relative path, from the snapshot submission took), and a wheelhouse with the Fabric harness the agent needs, downloaded for the Gym host’s interpreter. The sandboxed Gym host installs that wheelhouse offline and runs the component like any package-supplied agent: the registered agent’s identity, harness, instructions, tools, MCP servers and skills are what run; only its default model is rebound to Gym’s policy model server. Secrets bound to the agent’s environment travel as env_secrets and reach the component’s process, where a stdio MCP server’s ${NAME} templates are filled.

What this requires:

• Sandboxed execution. Colocated Gym ignores the staged package, so a registered agent is refused there.
• If you also supply a target.environment FileSet, it must be wheels-v1; the agent’s package is added to it, and a native-v1 package has no offline way to install the harness.
• The sandbox must be allowed to reach the Inference Gateway URL the agent’s model is bound to (sandbox_policy_base_urls); on a laptop that means the gateway through host.docker.internal, not localhost.
• The wheelhouse is downloaded from a package index by the job container on every run, tens of megabytes per harness. Image-side packaging is planned to replace this.
• The wheelhouse is resolved under the Gym host image’s own package pins, so the harness installs into the host without moving anything the host already has. Nothing to configure: the pins ship with the evaluator and follow the image’s lock.
• The agent’s runtime.timeout_seconds is its per-rollout task budget, enforced by Fabric from the moment the harness starts the task; an agent that declares none gets 600 seconds. Adapter start-up and wheelhouse installation inside the host do not count against it.
• A registered agent that binds several models keeps only its default through Gym’s policy server; named secondary models still call the gateway directly. Relay/ATIF trajectory capture from inside the Gym host is not wired yet; the rollout record carries the agent’s messages as Responses items.

Caveats

• Per-environment dependencies are heterogeneous. mcqa needs only tiktoken; other Gym environments pull in torch, COMET, a GPU, or Docker. Providing a Gym runtime with those installed is the caller’s responsibility.
• --no-serve --input bypasses Gym’s data-prep — prompt templating and dataset materialization. Rows that are already complete, like the bundled example.jsonl, are faithful; an environment whose rows need templating would need that step run first.
• Custom environments require platform support. Package the environment as a FileSet and use sandboxed execution; environment and agent_ref_name (the target’s source.instance) are GymPlacement fields supplied at submission, so a live GymAgentTaskRunner cannot carry them. env_secrets is not in that group — it lives on GymRuntimeConfig. Local runs look for the workspace-prefixed environment variable first, then the bare name. Pass secret_resolver= to GymAgentTaskRunner for a custom EnvSecretSource that names the environment variable holding each secret. Submitted jobs carry only the references and resolve them with the platform’s source.
• No retries and no re-run cache. A failed rollout is not retried, and nothing is reused between runs: every run collects from scratch, and refuses to run into a directory that already holds Gym rollout output — rollouts, the failures sidecar, or model-call captures. (An existing empty directory is fine.) Re-running a large suite pays the full collection cost again. (The Harbor runner does have both — do not carry that expectation across.)

Next steps