GRPO Environment Packages

Package a GRPO reward environment as a FileSet. Job JSON, LoRA, metrics, and cluster flags are in GRPO and Reward Environments.

A GRPO job takes two FileSets:

Do not put .jsonl in the environment package. If a Gym config lists datasets[].jsonl_fpath, omit that block and the data/ directory; prompts belong in the dataset FileSet.

Prerequisites

Before packaging an environment, ensure you have:

1. A NeMo Platform configured with platform.runtime: kubernetes — GRPO provisions a Ray cluster and has no local Docker fallback
2. Sandboxed Gym enabled on the cluster by your platform operator (sandbox_cluster_capable and a job-storage PVC claim). Refer to Cluster prerequisites; installing OpenSandbox is covered in OpenSandbox
3. The nemo CLI on your PATH, or a source checkout where uv run --package nmp-rl resolves
4. The training image tag your cluster runs — vendoring a wheel closure requires the nemo-gym, ray, and openai versions it reports
5. A machine with internet access for the packaging step. Training clusters consume uploaded FileSets only. For details on which environment type require internet access too, see details below.

Pick a format

All three run on the same Gym runtime: the training image starts a package’s servers exactly as it starts Gym’s own. The format decides only where dependencies come from and where config_paths may live. It does not change what is supported.

Use pi-to-gym-conversion to produce adapter-wheels-v1. Set adapter.agent to verifiers_agent.

How Gym YAML names servers

Gym loads every file in config_paths and starts instances:

1
<instance_name>:           # unique at runtime; dataset agent_ref.name and YAML refs use this
2
  <server_type>:           # resources_servers | responses_api_agents | responses_api_models
3
    <implementation>:      # directory: {server_type}/{implementation}/
4
      entrypoint: app.py

Refs are {type, name} where name is the instance, not the implementation folder. Dataset rows follow the same rule: agent_ref.name is an instance.

Instance and implementation are often spelled the same in Gym’s own configs (math_with_judge / math_with_judge), which is why the distinction is easy to miss. example_single_tool_call.yaml is the counter-example: instance example_single_tool_call_simple_agent, implementation simple_agent.

The job looks up {server_type}/{implementation}/ in the environment FileSet first, then in the Gym tree on the training image. A server directory counts only if it includes requirements.txt or pyproject.toml — never both. The process runs python app.py from that directory.

You can reference simple_agent, vllm_model, and verifiers_agent without uploading their app.py — shipping only responses_api_agents/simple_agent/configs/*.yaml is correct and intended. Any other implementation must appear as {type}/{impl}/ in the FileSet (native-v1 and wheels-v1). Put YAML alone under configs/ only when every implementation is one of those image servers (adapter-wheels-v1).

Include a policy_model definition in config_paths. Without it, Gym reports ServerRefNotFoundError (Available responses_api_models: (none)).

1
policy_model:
2
  responses_api_models:
3
    vllm_model:
4
      entrypoint: app.py
5
      base_url: ${policy_base_url}
6
      api_key: ${policy_api_key}
7
      model: ${policy_model_name}
8
      return_token_id_information: true
9
      uses_reasoning_parser: true

${policy_base_url}, ${policy_api_key}, and ${policy_model_name} resolve to the job’s vLLM endpoint. List this file first in config_paths.

The native-v1 path only has to start with one of the three server-type prefixes; the folder underneath it is free. responses_api_models/policy_model/configs/policy_model.yaml — matching the instance — passes the same check. Either is fine, because Gym runs responses_api_models/vllm_model/ from the YAML body, not from the config file’s directory. Pick one convention and keep it: naming the folder after the implementation makes the on-disk tree mirror what actually runs.

Dependencies

Each Gym server installs into its own virtualenv when the job starts. There are two installs, and they behave differently — this is the most common source of confusion about offline runs.

1. Gym builds the virtualenv from that server’s requirements.txt or pyproject.toml, plus nemo-gym at the training-image version and Gym’s pinned ray[default] and openai. The package’s wheels/ directory is offered to this step as a candidate pool (UV_FIND_LINKS) — a package index is still enabled, so anything the wheelhouse misses is fetched from it.
2. The job then installs the package’s vendored closure into each agent and resources-server virtualenv with --no-index --find-links=wheels/. Fully offline, but only for distributions already in wheels/, and only after step 1 has succeeded.

For a wheels-v1 job to start with no egress, wheels/ must contain:

• everything the server’s requirements.txt names, and their transitive dependencies;
• nemo-gym at exactly the version the training image reports;
• ray[default] and openai at the versions the image pins — the virtualenv is created with uv venv --seed, so nothing is inherited from the image.

Completeness of wheels/ is what makes a sandboxed run work; the format name alone does not.

requirements.txt in a custom server directory

The file’s presence is what makes Gym treat the directory as a server. Its contents then build the virtualenv:

That rewrite is why a Gym server directory copies cleanly out of the source tree. It is also why vendoring the image’s exact nemo-gym version matters: a mismatch means your wheel is ignored and the version is resolved from PyPI.

adapter-wheels-v1

Convert on a machine with internet. Upload the result; training nodes do not fetch from the hub.

From a platform install, run pi-to-gym-conversion. From this repository:

$
uv run --package nmp-rl pi-to-gym-conversion \
>
  --hub-id primeintellect/ascii-tree \
>
  --hub-version 0.1.5 \
>
  --out-dir ./ascii-tree-pkg \
>
  --dataset-dir ./ascii-tree-data \
>
  --validation-fraction 0.1

Pin --hub-version so the vendored release is reproducible.

ascii-tree-pkg/
  nemo-environment.yaml
  configs/policy_model.yaml
  configs/verifiers_agent.yaml
  wheels/*.whl
ascii-tree-data/
  training.jsonl
  validation.jsonl                 # when --validation-fraction > 0

1
# nemo-environment.yaml
2
format: adapter-wheels-v1
3
config_paths:
4
  - configs/policy_model.yaml
5
  - configs/verifiers_agent.yaml
6
adapter:
7
  agent: verifiers_agent
8
  agent_type: responses_api_agents
9
metadata:
10
  name: ascii-tree
11
  hub_id: primeintellect/ascii-tree
12
  vf_env_id: ascii-tree
13
  adapter_agent: verifiers_agent

Set adapter.agent to verifiers_agent. Dataset rows use agent_ref.name: verifiers_agent and vf_env_id equal to metadata.vf_env_id.

configs/verifiers_agent.yaml sets max_tokens (default 8192). That value caps generation for this agent.

$
uv run --package nmp-rl pi-to-gym-conversion --validate-only ./ascii-tree-pkg

This checks layout (manifest, config_paths, wheels/). It does not check wheel platform tags, whether the closure is complete, whether config_paths define the servers your rows route to, or dataset rows. Those surface at job start.

native-v1

Keep Gym directory names. Do not flatten the tree.

weather-env/
├── nemo-environment.yaml
├── resources_servers/
│   └── weather_verifier/
│       ├── app.py
│       ├── requirements.txt
│       └── configs/
│           └── weather_verifier.yaml
├── responses_api_agents/
│   └── simple_agent/
│       └── configs/
│           └── weather_agent.yaml
└── responses_api_models/
    └── vllm_model/
        └── configs/
            └── policy_model.yaml

responses_api_agents/simple_agent/configs/ configures the image simple_agent. resources_servers/weather_verifier/ is your server, so it includes app.py and requirements.txt.

1
format: native-v1
2
config_paths:
3
  - responses_api_models/vllm_model/configs/policy_model.yaml
4
  - resources_servers/weather_verifier/configs/weather_verifier.yaml
5
  - responses_api_agents/simple_agent/configs/weather_agent.yaml
6
metadata:
7
  name: weather-grpo

1
# resources_servers/weather_verifier/configs/weather_verifier.yaml
2
weather_verifier:
3
  resources_servers:
4
    weather_verifier:
5
      entrypoint: app.py
6
      domain: agent
7
      description: Weather tool + verify

1
# responses_api_agents/simple_agent/configs/weather_agent.yaml
2
weather_simple_agent:
3
  responses_api_agents:
4
    simple_agent:
5
      entrypoint: app.py
6
      resources_server:
7
        type: resources_servers
8
        name: weather_verifier
9
      model_server:
10
        type: responses_api_models
11
        name: policy_model

resources_servers/weather_verifier/app.py is a SimpleResourcesServer: one route per tool, plus the verify() that returns the reward.

1
from fastapi import FastAPI
2
from pydantic import BaseModel
3
4
from nemo_gym.base_resources_server import (
5
    BaseResourcesServerConfig,
6
    BaseVerifyRequest,
7
    BaseVerifyResponse,
8
    SimpleResourcesServer,
9
)
10
11
12
class WeatherResourcesServerConfig(BaseResourcesServerConfig):
13
    pass
14
15
16
class GetWeatherRequest(BaseModel):
17
    city: str
18
19
20
class GetWeatherResponse(BaseModel):
21
    city: str
22
    weather_description: str
23
24
25
class WeatherResourcesServer(SimpleResourcesServer):
26
    config: WeatherResourcesServerConfig
27
28
    def setup_webserver(self) -> FastAPI:
29
        app = super().setup_webserver()
30
        app.post("/get_weather")(self.get_weather)
31
        return app
32
33
    async def get_weather(self, body: GetWeatherRequest) -> GetWeatherResponse:
34
        return GetWeatherResponse(city=body.city, weather_description=f"The weather in {body.city} is cold.")
35
36
    async def verify(self, body: BaseVerifyRequest) -> BaseVerifyResponse:
37
        # Score the rollout in `body` and return the reward. 1.0 is a placeholder.
38
        return BaseVerifyResponse(**body.model_dump(), reward=1.0)
39
40
41
if __name__ == "__main__":
42
    WeatherResourcesServer.run_webserver()

resources_servers/weather_verifier/requirements.txt lists anything else app.py imports. nemo-gym is prepended for you, so a server with no other dependencies needs only a comment line — but not an empty file.

Set dataset agent_ref.name to weather_simple_agent, the agent instance.

wheels-v1

Use the same server directories as native-v1, and add wheels/ so job start does not need a package index. config_paths may list files under configs/ or under the server trees. Custom implementations need {type}/{impl}/ with app.py and requirements.txt.

weather-env/
├── nemo-environment.yaml
├── configs/
│   ├── policy_model.yaml
│   ├── weather_verifier.yaml
│   └── weather_agent.yaml
├── resources_servers/
│   └── weather_verifier/
│       ├── app.py
│       └── requirements.txt
└── wheels/
    ├── nemo_gym-<image-version>-*.whl
    └── …                                    # match the training nodes' arch; CPython 3.13; one version per project

Wheels must match the architecture of the nodes that run GRPO training, not the machine that builds the package. The training images are published for both linux/amd64 and linux/arm64, so there is no single correct answer — check what your cluster runs and target that:

$
kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.architecture}'   # amd64 or arm64
$
ARCH=x86_64        # amd64 nodes
$
# ARCH=aarch64     # arm64 nodes

The interpreter is Python 3.13 on every architecture. A closure resolved for the wrong architecture passes --validate-only, which checks layout rather than wheel tags, and then fails on the cluster with has no wheels with a matching platform tag. Three versions must match what the Gym host process reports: nemo-gym, which Gym pins each virtualenv to, and ray[default] / openai, which it appends to every install. Gym runs from its own actor virtualenv under /opt/ray_venvs, so read them from there:

$
IMAGE=<training-image>
$
read GYM_VERSION RAY_VERSION OPENAI_VERSION < <(docker run --rm "$IMAGE" sh -c '
>
  PY=$(ls -d /opt/ray_venvs/*NemoGym*/bin/python 2>/dev/null | head -1)
>
  "${PY:-python}" -c "import importlib.metadata as m; print(m.version(\"nemo-gym\"), m.version(\"ray\"), m.version(\"openai\"))"')
$
$
mkdir -p weather-env/wheels
$
pip download --dest weather-env/wheels \
>
  --only-binary=:all: \
>
  --python-version 3.13 \
>
  --platform "manylinux_2_39_$ARCH" \
>
  --platform "manylinux_2_28_$ARCH" \
>
  --platform "manylinux_2_17_$ARCH" \
>
  --platform "manylinux2014_$ARCH" \
>
  "nemo-gym==$GYM_VERSION" "ray[default]==$RAY_VERSION" "openai==$OPENAI_VERSION" \
>
  -r resources_servers/weather_verifier/requirements.txt
$
uv run --package nmp-rl pi-to-gym-conversion --validate-only ./weather-env

--platform is repeated because pip matches these tags literally rather than expanding a compatibility range.

--validate-only does not check wheel tags. Empty wheels/ and rebuild it so you do not ship two versions of the same project.

Dataset rows

1
{
2
  "responses_create_params": {
3
    "input": [{"role": "user", "content": "what's it like in sf?"}]
4
  },
5
  "agent_ref": {
6
    "type": "responses_api_agents",
7
    "name": "weather_simple_agent"
8
  }
9
}

Hub conversions also set vf_env_id, task_idx, and agent_ref.name: verifiers_agent. Extra fields are passed to the resources server.

When the agent is simple_agent and the environment exposes function-calling tools, each row must also declare them under responses_create_params.tools — simple_agent reads the tool list from the row, not from the resources server:

1
{
2
  "responses_create_params": {
3
    "input": [{"role": "user", "content": "what's it like in sf?"}],
4
    "tools": [{"type": "function", "name": "get_weather", "description": "",
5
               "parameters": {"type": "object", "properties": {"city": {"type": "string"}},
6
                              "required": ["city"], "additionalProperties": false},
7
               "strict": true}]
8
  },
9
  "agent_ref": {"type": "responses_api_agents", "name": "weather_simple_agent"}
10
}

Upload and submit

Keep relative paths. A trailing slash on nemo files upload ./pkg/ uploads the directory contents; without it, everything nests one level deeper under the directory’s basename and the manifest is no longer at the FileSet root. Run nemo files list to confirm before submitting.

--purpose environment is enforced at submit, not cosmetic: an environment FileSet whose purpose is anything but environment or generic is rejected.

$
nemo files filesets create weather-env --workspace default --purpose environment --exist-ok
$
nemo files upload ./weather-env/ weather-env --workspace default
$
nemo files list weather-env --workspace default

Job JSON: GRPO and Reward Environments. Sandboxed Gym is configured on the platform (NMP_RL_SANDBOXED_GYM_DEFAULT), not in the job payload. Cluster setup: OpenSandbox.

Next Steps

• Submit the job and tune its hyperparameters in GRPO and Reward Environments.
• Shape prompt rows for your environment’s agent in Format a Training Dataset.
• Monitor reward and the other training curves in Check Customization Job Metrics.