GRPO and Reward Environments

GRPO (Group Relative Policy Optimization) trains a model against a reward environment instead of labeled completions. For each prompt the platform samples a group of responses, the environment scores them, and the policy is updated toward the responses that scored better than the others in their group.

That last part is why GRPO behaves differently from every other customization method here: the learning signal is the spread of rewards inside a group. If every response to a prompt earns the same score, that prompt contributes nothing.

GRPO runs on the rl backend, the same one that runs DPO, and requires a NeMo Platform configured with platform.runtime: kubernetes — it provisions a Ray cluster and has no local Docker fallback.

What you need

Unlike SFT or DPO, GRPO takes two FileSets:

Plus a registered model entity, exactly as for the other backends.

Choose an environment format

The environment package declares its format in a nemo-environment.yaml manifest at its root. Three formats are supported.

All three formats run on the same Gym runtime and are supported equally. The format decides two things only: where a package’s dependencies come from, and where its config_paths may live.

The manifest

Every package declares itself in nemo-environment.yaml at its root. The shape is the same in all three formats:

1
format: wheels-v1              # or native-v1, adapter-wheels-v1
2
config_paths:                  # at least one, relative to the package root
3
  - configs/policy_model.yaml
4
  - configs/my_env.yaml
5
metadata:
6
  name: my-env                 # required
7
  description: Custom reward environment

adapter-wheels-v1 adds one additional block, because the agent harness comes from the training image rather than the package:

1
adapter:
2
  agent: verifiers_agent       # the only harness built into the image
3
  agent_type: responses_api_agents

Rules that apply to every format:

• nemo-environment.yaml sits at the package root.
• Every config_paths entry is relative, contains no .., is not a symlink, and exists in the package.
• No .jsonl anywhere in the package.
• wheels/ is non-empty and contains only .whl files, for the two wheels formats.
• Unknown manifest keys are rejected.

How dependencies are installed

The format you choose does not change what runs — all three start on the same Gym runtime. It changes where dependencies come from, and that is what usually decides whether a job starts at all.

There are two installs at job start:

1. Gym builds one venv per server from that server’s pyproject.toml or requirements.txt, plus nemo-gym at the image version and Gym’s pinned ray[default] and openai. A package’s wheels/ directory is offered here as a candidate pool — a package index is still enabled, so anything the wheelhouse misses is fetched from it.
2. The job then installs the package’s vendored closure into each agent and resources-server venv with --no-index. Fully offline, but only for what wheels/ already carries, and only after step 1 succeeded.

A wheels-v1 job runs with no egress only when wheels/ carries the server’s full requirement closure, nemo-gym at exactly the training-image version, and Gym’s pinned ray[default] and openai — the venv is seeded empty, so nothing carries over from the image. Completeness of wheels/ is what makes an offline run work, not the format name. Full detail: GRPO Environment Packages.

Egress is an operator setting, not a job field: NMP_RL_SANDBOX_ALLOW_INTERNET, plus NMP_RL_SANDBOX_PUBLIC_DNS_ALLOW for hosts outside the built-in *.com / *.org allowance. Ask your platform operator which is configured before choosing a format.

Vendoring a wheel closure

Wheels must match the architecture of the nodes that run GRPO training, not the machine that builds the package. The training images are published for both linux/amd64 and linux/arm64, so there is no single correct answer — check what your cluster runs and target that:

$
kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.architecture}'   # amd64 or arm64
$
ARCH=x86_64        # amd64 nodes
$
# ARCH=aarch64     # arm64 nodes

The interpreter is Python 3.13 on every architecture. A closure resolved for the wrong architecture passes --validate-only, which checks layout rather than wheel tags, and then fails on the cluster with has no wheels with a matching platform tag.

$
pip download <requirements> --dest my-env/wheels \
>
  --only-binary=:all: \
>
  --python-version 3.13 \
>
  --platform "manylinux_2_39_$ARCH" \
>
  --platform "manylinux_2_28_$ARCH" \
>
  --platform "manylinux_2_17_$ARCH" \
>
  --platform "manylinux2014_$ARCH"

--platform is repeated because pip matches these tags literally rather than expanding a compatibility range. Name each tag you need, including older glibc floors.

Clear wheels/ before rebuilding into it. Copies overwrite by filename, so a wheel from an earlier run survives whenever the new closure resolved that project to a different version, and the package then vendors both.

Build the environment package

Pick the subsection matching the format you chose above. Each produces a directory you validate and then upload.

native-v1 — a Gym server tree, dependencies resolved at spin-up

For native-v1, the package is a slice of the Gym source tree with its directory structure preserved, plus a manifest you add at the root:

resources_servers/example_single_tool_call/
  app.py
  configs/example_single_tool_call.yaml
  requirements.txt
nemo-environment.yaml

1
format: native-v1
2
config_paths:
3
  - resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml
4
metadata:
5
  name: example-single-tool-call

native-v1 ships no wheels, so its server’s dependencies resolve from a package index at spin-up and the job needs egress. If that is not available on your cluster, package the same server tree as wheels-v1 instead and vendor the closure — see How dependencies are installed.

wheels-v1 — any environment, dependencies vendored

Use the same directory layout as native-v1 — the server’s own tree, copied from a Gym checkout — and add a wheels/ directory holding the full dependency closure so nothing is fetched at job start. config_paths may live anywhere in the package. This is the format to choose when the cluster has no egress. Layout, the closure contents, and a worked example: GRPO Environment Packages.

adapter-wheels-v1 — a verifiers environment, via the converter

A verifiers environment — a Prime Intellect hub package, for example — is the one case with a scripted path. pi-to-gym-conversion downloads the package, vendors its full wheel closure, writes the configs and manifest, and builds the prompt JSONL.

Run it on a machine with internet access. Training clusters have no hub egress and consume uploaded FileSets only. From a platform install the converter is on PATH; from this repository:

$
uv run --package nmp-rl pi-to-gym-conversion \
>
  --hub-id primeintellect/ascii-tree \
>
  --hub-version 0.1.5 \
>
  --out-dir ./ascii-tree-pkg \
>
  --dataset-dir ./ascii-tree-data \
>
  --validation-fraction 0.1

The converter writes:

ascii-tree-pkg/
  nemo-environment.yaml
  configs/policy_model.yaml       # points Gym at the job's own vLLM engine
  configs/verifiers_agent.yaml    # vf_env_id, vf_env_args, max_tokens, temperature
  wheels/*.whl
ascii-tree-data/
  training.jsonl
  validation.jsonl                # only with --validation-fraction > 0

Add --upload (with NMP_BASE_URL set) to create both FileSets and upload them in the same command.

Every package needs a policy_model server

Gym server configs reference the model they roll out against by name, conventionally policy_model, and Gym resolves every reference when it merges configs — long before any rollout. If nothing in your config_paths defines that server, spin-up fails with ServerRefNotFoundError: ... Available responses_api_models: (none).

The converter writes this file for you. Hand-built packages must add it, in every format:

1
policy_model:
2
  responses_api_models:
3
    vllm_model:
4
      entrypoint: app.py
5
      base_url: ${policy_base_url}
6
      api_key: ${policy_api_key}
7
      model: ${policy_model_name}
8
      return_token_id_information: true
9
      uses_reasoning_parser: true

The three interpolations resolve to the job’s vLLM endpoint. List this file first in config_paths.

For native-v1 only the server-type prefix is checked, so responses_api_models/policy_model/configs/policy_model.yaml works too — Gym runs the directory named in the YAML body, not the one the config file sits in.

Custom implementations need {server_type}/{implementation}/ (with app.py and requirements.txt) in the FileSet for native-v1 and wheels-v1. YAML under configs/ does not replace that directory, and a directory without requirements.txt or pyproject.toml is not recognised as a server at all. Details: GRPO Environment Packages.

Validate before uploading, whichever format you built

$
uv run --package nmp-rl pi-to-gym-conversion --validate-only ./my-env-pkg

Despite the command’s name this validates any of the three formats, not just converter output. It prints {"valid": true, "format": "...", "name": "..."} or exits non-zero with the specific violation. The same checks run at submit time, so validating locally catches the failure earlier and for free.

Prompt dataset format

GRPO rows are rollout rows, not prompt/completion pairs and not preference triples. The prompt goes under responses_create_params.input, in OpenAI Responses API shape.

1
{
2
  "task_idx": 0,
3
  "vf_env_id": "ascii-tree",
4
  "responses_create_params": {"input": [{"role": "user", "content": "Draw a binary tree of depth 3."}]},
5
  "agent_ref": {"type": "responses_api_agents", "name": "verifiers_agent"},
6
  "question": "Draw a binary tree of depth 3.",
7
  "answer": "",
8
  "task": "ascii-tree",
9
  "example_id": "ex-0",
10
  "info": {}
11
}

Extra keys are allowed and forwarded to the environment. Only the agent-agnostic fields are schema-checked, so vf_env_id and task_idx are not rejected when missing or mismatched — a wrong vf_env_id surfaces as the environment failing to load, not as a validation error.

Validation runs exactly one rollout per row of validation.jsonl — there is no validation counterpart to num_generations_per_prompt. To score a prompt k times, repeat its row k times.

Upload both FileSets

nemo files upload takes a directory and uploads it recursively, preserving relative paths:

$
ENV=ascii-tree-env
$
nemo files filesets create "$ENV" --workspace default --purpose environment --exist-ok
$
nemo files upload ./ascii-tree-pkg/ "$ENV" --workspace default
$
nemo files list "$ENV" --workspace default
$
$
DATA=ascii-tree-env-dataset
$
nemo files filesets create "$DATA" --workspace default --purpose dataset --exist-ok
$
nemo files upload ./ascii-tree-data/ "$DATA" --workspace default

The trailing slash on the local path matters — it selects the directory’s contents rather than the directory itself. Without it everything nests one level deeper under the directory’s basename, which leaves the manifest off the FileSet root. Confirm with nemo files list before submitting.

--purpose environment is enforced, not cosmetic: submit rejects an environment FileSet whose purpose is anything other than environment or generic.

Relative paths must be preserved. config_paths is matched against the FileSet listing, so a package flattened during upload fails with config_paths reference files that are not in the package.

These are the names --upload would have used: it derives them from the hub slug as <slug>-env and <slug>-env-dataset. Pass --environment-name / --dataset-name to override.

Submit the job

There is no grpo subcommand. GRPO submits through nemo customization rl submit, selected by training.type.

1
{
2
  "model": "default/qwen3-0.6b",
3
  "dataset": "default/ascii-tree-env-dataset",
4
  "environment": "default/ascii-tree-env",
5
  "training": {
6
    "type": "grpo",
7
    "epochs": 1,
8
    "learning_rate": 1e-6,
9
    "max_seq_length": 2048,
10
    "batch_size": 32,
11
    "micro_batch_size": 1,
12
    "num_generations_per_prompt": 8,
13
    "temperature": 1.0,
14
    "parallelism": { "num_nodes": 1, "num_gpus_per_node": 1 }
15
  },
16
  "output": { "name": "qwen3-0.6b-ascii-tree" }
17
}

$
nemo customization rl submit ./job.json --workspace default

training.type is required — it is the union discriminator, and omitting it fails with union_tag_not_found rather than defaulting to DPO. model, dataset, and environment are plain string references.

Read the rl-<hex> job id from the name field of the submit response; rl submit has no --name flag.

Train an adapter instead of full weights

GRPO trains every weight by default. Set finetuning_type to train a LoRA adapter instead — useful when the full-weight run does not fit in memory, or when you want several environments’ worth of behaviour over one shared base deployment.

1
{
2
  "model": "default/qwen3-0.6b",
3
  "dataset": "default/ascii-tree-env-dataset",
4
  "environment": "default/ascii-tree-env",
5
  "training": {
6
    "type": "grpo",
7
    "finetuning_type": "lora",
8
    "lora": { "rank": 16, "alpha": 32, "dropout": 0.0 },
9
    "epochs": 1,
10
    "batch_size": 32,
11
    "num_generations_per_prompt": 8
12
  },
13
  "output": { "name": "qwen3-0.6b-ascii-tree" }
14
}

LoRA fields

Omitting the lora block while setting finetuning_type: "lora" is fine — defaults are filled in. Supplying a lora block alongside all_weights is rejected.

Using the adapter

The adapter hot-reloads onto a READY deployment of the base model that has lora_enabled: true, so there is no new deployment to create before evaluating it. Route inference through the provider gateway (/provider/<name>/-/v1 with model: default--<adapter>); the model-entity path always resolves to the base model.

A full-weight GRPO job instead registers a new model entity, which does need its own deployment.

Key hyperparameters

All GRPO knobs live under training, including policy_backend (a sibling of parallelism, not a field on it).

Advanced clipping and advantage-estimation settings (ratio_clip_c, advantage_clip_low / advantage_clip_high, normalize_rewards, use_leave_one_out_baseline, top_k) are documented in Training Configuration.

Read the results

Monitor GRPO on reward, not loss. The GRPO surrogate loss oscillates near zero and carries no signal about run quality.

status_details also carries two constants stated once when training starts: training_type (grpo or dpo — backend is nemo_rl for both) and rollouts_per_step, so the rollouts generated so far is step × rollouts_per_step.

Troubleshooting

Cluster prerequisites

GRPO fails at submit, before any GPU is claimed, unless the platform operator has configured sandboxed Gym. Both settings live on the platform, not in the job JSON:

If a submit fails on either of the first two, that is a platform configuration gap, not a problem with your package. Neither is settable per job.

Installing OpenSandbox is a separate operator task: OpenSandbox, or OpenSandbox with Kata for the Kata runtime.

Next Steps

• Build the environment FileSet itself in GRPO Environment Packages.
• Monitor reward and the other training curves in Check Customization Job Metrics.
• Compare a trained model against its base in Evaluate Models & Agents.