> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server. # GRPO and Reward Environments > Train a model with GRPO against a NeMo Gym reward environment. GRPO (Group Relative Policy Optimization) trains a model against a **reward environment** instead of labeled completions. For each prompt the platform samples a group of responses, the environment scores them, and the policy is updated toward the responses that scored better than the others in their group. That last part is why GRPO behaves differently from every other customization method here: the learning signal is the *spread* of rewards inside a group. If every response to a prompt earns the same score, that prompt contributes nothing. GRPO runs on the `rl` backend, the same one that runs DPO, and requires a NeMo Platform configured with `platform.runtime: kubernetes` — it provisions a Ray cluster and has no local Docker fallback. ## What you need Unlike SFT or DPO, GRPO takes **two** FileSets: | FileSet | `purpose` | Holds | | ----------- | ------------- | --------------------------------------------------------------------------- | | Environment | `environment` | Code and config that runs a rollout and returns a reward | | Dataset | `dataset` | `training.jsonl` (required) and `validation.jsonl` (optional) — prompt rows | Plus a registered model entity, exactly as for the other backends. Prompt JSONL must **not** live inside the environment package. Schema validation rejects any `.jsonl` in the environment FileSet, so keep the two FileSets separate. ## Choose an environment format The environment package declares its format in a `nemo-environment.yaml` manifest at its root. Three formats are supported. | You have | Format | Ships wheels | `config_paths` live under | | -------------------------------------------------------------------------------------------- | ------------------- | ------------ | ---------------------------------------------------------------------- | | A Gym server tree whose dependencies are already resolvable at spin-up | `native-v1` | No | `responses_api_agents/`, `resources_servers/`, `responses_api_models/` | | Any `verifiers` environment installable as a wheel like a Prime Intellect hub environment | `adapter-wheels-v1` | Yes | `configs/` | | Any other environment code, including a Gym server tree whose dependencies you want vendored | `wheels-v1` | Yes | Anywhere in the package | All three formats run on the same Gym runtime and are supported equally. The format decides two things only: where a package's dependencies come from, and where its `config_paths` may live. ### The manifest Every package declares itself in `nemo-environment.yaml` at its root. The shape is the same in all three formats: ```yaml format: wheels-v1 # or native-v1, adapter-wheels-v1 config_paths: # at least one, relative to the package root - configs/policy_model.yaml - configs/my_env.yaml metadata: name: my-env # required description: Custom reward environment ``` `adapter-wheels-v1` adds one additional block, because the agent harness comes from the training image rather than the package: ```yaml adapter: agent: verifiers_agent # the only harness built into the image agent_type: responses_api_agents ``` Rules that apply to every format: * `nemo-environment.yaml` sits at the package root. * Every `config_paths` entry is relative, contains no `..`, is not a symlink, and exists in the package. * No `.jsonl` anywhere in the package. * `wheels/` is non-empty and contains only `.whl` files, for the two wheels formats. * Unknown manifest keys are rejected. ## How dependencies are installed The format you choose does not change *what runs* — all three start on the same Gym runtime. It changes where dependencies come from, and that is what usually decides whether a job starts at all. There are two installs at job start: 1. Gym builds **one venv per server** from that server's `pyproject.toml` or `requirements.txt`, plus `nemo-gym` at the image version and Gym's pinned `ray[default]` and `openai`. A package's `wheels/` directory is offered here as a *candidate pool* — **a package index is still enabled**, so anything the wheelhouse misses is fetched from it. 2. The job then installs the package's vendored closure into each agent and resources-server venv with `--no-index`. Fully offline, but only for what `wheels/` already carries, and only after step 1 succeeded. | Format | Server dependencies come from | Needs egress at spin-up | | ------------------- | ------------------------------------------------------------------ | ----------------------------------------- | | `native-v1` | A package index, resolved from the server's `requirements.txt` | Yes | | `wheels-v1` | The package's `wheels/`, for anything it vendors | No, **if the closure also covers step 1** | | `adapter-wheels-v1` | The package's `wheels/`, plus the agent harness's own requirements | Yes — see below | A `wheels-v1` job runs with no egress only when `wheels/` carries the server's full requirement closure, `nemo-gym` at exactly the training-image version, and Gym's pinned `ray[default]` and `openai` — the venv is seeded empty, so nothing carries over from the image. Completeness of `wheels/` is what makes an offline run work, not the format name. Full detail: [GRPO Environment Packages](/documentation/customizer-reference/tutorials/grpo-environment-packages#dependencies). `adapter-wheels-v1` requires network access when the job starts. The package `wheels/` directory covers the hub environment; `verifiers_agent` also installs `verifiers` from GitHub. Egress is an operator setting, not a job field: `NMP_RL_SANDBOX_ALLOW_INTERNET`, plus `NMP_RL_SANDBOX_PUBLIC_DNS_ALLOW` for hosts outside the built-in `*.com` / `*.org` allowance. Ask your platform operator which is configured before choosing a format. ### Vendoring a wheel closure Wheels must match the **architecture of the nodes that run GRPO training**, not the machine that builds the package. The training images are published for both `linux/amd64` and `linux/arm64`, so there is no single correct answer — check what your cluster runs and target that: ```bash kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.architecture}' # amd64 or arm64 ARCH=x86_64 # amd64 nodes # ARCH=aarch64 # arm64 nodes ``` The interpreter is **Python 3.13** on every architecture. A closure resolved for the wrong architecture passes `--validate-only`, which checks layout rather than wheel tags, and then fails on the cluster with `has no wheels with a matching platform tag`. ```bash pip download --dest my-env/wheels \ --only-binary=:all: \ --python-version 3.13 \ --platform "manylinux_2_39_$ARCH" \ --platform "manylinux_2_28_$ARCH" \ --platform "manylinux_2_17_$ARCH" \ --platform "manylinux2014_$ARCH" ``` `--platform` is repeated because pip matches these tags literally rather than expanding a compatibility range. Name each tag you need, including older glibc floors. Clear `wheels/` before rebuilding into it. Copies overwrite by filename, so a wheel from an earlier run survives whenever the new closure resolved that project to a different version, and the package then vendors both. ## Build the environment package Pick the subsection matching the format you chose above. Each produces a directory you validate and then upload. ### `native-v1` — a Gym server tree, dependencies resolved at spin-up For `native-v1`, the package is a slice of the Gym source tree with its directory structure preserved, plus a manifest you add at the root: ```text resources_servers/example_single_tool_call/ app.py configs/example_single_tool_call.yaml requirements.txt nemo-environment.yaml ``` ```yaml format: native-v1 config_paths: - resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml metadata: name: example-single-tool-call ``` Gym's own configs point `datasets[].jsonl_fpath` at a file inside the server directory. That file cannot ship in the environment package. Remove the data directory and supply prompts through the dataset FileSet instead. `native-v1` ships no wheels, so its server's dependencies resolve from a package index at spin-up and the job needs egress. If that is not available on your cluster, package the same server tree as `wheels-v1` instead and vendor the closure — see [How dependencies are installed](#how-dependencies-are-installed). ### `wheels-v1` — any environment, dependencies vendored Use the same directory layout as `native-v1` — the server's own tree, copied from a Gym checkout — and add a `wheels/` directory holding the full dependency closure so nothing is fetched at job start. `config_paths` may live anywhere in the package. This is the format to choose when the cluster has no egress. Layout, the closure contents, and a worked example: [GRPO Environment Packages](/documentation/customizer-reference/tutorials/grpo-environment-packages). ### `adapter-wheels-v1` — a `verifiers` environment, via the converter A `verifiers` environment — a Prime Intellect hub package, for example — is the one case with a scripted path. `pi-to-gym-conversion` downloads the package, vendors its full wheel closure, writes the configs and manifest, and builds the prompt JSONL. Run it on a machine with internet access. Training clusters have no hub egress and consume uploaded FileSets only. From a platform install the converter is on `PATH`; from this repository: ```bash uv run --package nmp-rl pi-to-gym-conversion \ --hub-id primeintellect/ascii-tree \ --hub-version 0.1.5 \ --out-dir ./ascii-tree-pkg \ --dataset-dir ./ascii-tree-data \ --validation-fraction 0.1 ``` Pin `--hub-version`. Left unset, the converter takes whatever the index offers at that moment; a later release can narrow `Requires-Python` and fail the download, or install but not run on the training image's Python. The converter writes: ```text ascii-tree-pkg/ nemo-environment.yaml configs/policy_model.yaml # points Gym at the job's own vLLM engine configs/verifiers_agent.yaml # vf_env_id, vf_env_args, max_tokens, temperature wheels/*.whl ascii-tree-data/ training.jsonl validation.jsonl # only with --validation-fraction > 0 ``` Add `--upload` (with `NMP_BASE_URL` set) to create both FileSets and upload them in the same command. ### Every package needs a `policy_model` server Gym server configs reference the model they roll out against by name, conventionally `policy_model`, and Gym resolves every reference when it merges configs — long before any rollout. If nothing in your `config_paths` *defines* that server, spin-up fails with `ServerRefNotFoundError: ... Available responses_api_models: (none)`. The converter writes this file for you. Hand-built packages must add it, in every format: ```yaml policy_model: responses_api_models: vllm_model: entrypoint: app.py base_url: ${policy_base_url} api_key: ${policy_api_key} model: ${policy_model_name} return_token_id_information: true uses_reasoning_parser: true ``` The three interpolations resolve to the job’s vLLM endpoint. List this file first in `config_paths`. | Format | Path | | -------------------------------- | ----------------------------------------------------------- | | `adapter-wheels-v1`, `wheels-v1` | `configs/policy_model.yaml` | | `native-v1` | `responses_api_models/vllm_model/configs/policy_model.yaml` | For `native-v1` only the server-type prefix is checked, so `responses_api_models/policy_model/configs/policy_model.yaml` works too — Gym runs the directory named in the YAML body, not the one the config file sits in. Custom implementations need `{server_type}/{implementation}/` (with `app.py` and `requirements.txt`) in the FileSet for `native-v1` and `wheels-v1`. YAML under `configs/` does not replace that directory, and a directory without `requirements.txt` or `pyproject.toml` is not recognised as a server at all. Details: [GRPO Environment Packages](/documentation/customizer-reference/tutorials/grpo-environment-packages). `pi-to-gym-conversion` currently only vendors wheels for `x86_64` today. On an `arm64` cluster, build the closure yourself with the `pip download` command above and pass it via `--wheels-dir`. ### Validate before uploading, whichever format you built ```bash uv run --package nmp-rl pi-to-gym-conversion --validate-only ./my-env-pkg ``` Despite the command's name this validates any of the three formats, not just converter output. It prints `{"valid": true, "format": "...", "name": "..."}` or exits non-zero with the specific violation. The same checks run at submit time, so validating locally catches the failure earlier and for free. ## Prompt dataset format GRPO rows are **rollout rows**, not prompt/completion pairs and not preference triples. The prompt goes under `responses_create_params.input`, in OpenAI Responses API shape. ```json { "task_idx": 0, "vf_env_id": "ascii-tree", "responses_create_params": {"input": [{"role": "user", "content": "Draw a binary tree of depth 3."}]}, "agent_ref": {"type": "responses_api_agents", "name": "verifiers_agent"}, "question": "Draw a binary tree of depth 3.", "answer": "", "task": "ascii-tree", "example_id": "ex-0", "info": {} } ``` | Field | Required | Meaning | | -------------------------------------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------- | | `responses_create_params` | Yes | The prompt. Messages go under `input`. | | `agent_ref` | Yes | Object routing the row to an agent. `name` is the Gym **instance** in the environment YAML (converter: `verifiers_agent`). | | `vf_env_id` | verifiers environments | Passed to `verifiers.load_environment()`. Set it to the manifest's `metadata.vf_env_id`. | | `task_idx` | verifiers environments | Row index. | | `answer`, `task`, `example_id`, `info`, `question` | No | Passed through to the environment. | Extra keys are allowed and forwarded to the environment. Only the agent-agnostic fields are schema-checked, so `vf_env_id` and `task_idx` are not rejected when missing or mismatched — a wrong `vf_env_id` surfaces as the environment failing to load, not as a validation error. Row shapes are checked inside the training container, not at submit. Submitting only verifies that the dataset FileSet contains `training.jsonl` and that the environment package's manifest is valid, so a malformed row costs a job start. Validate rows locally before uploading. Validation runs exactly one rollout per row of `validation.jsonl` — there is no validation counterpart to `num_generations_per_prompt`. To score a prompt *k* times, repeat its row *k* times. ## Upload both FileSets `nemo files upload` takes a directory and uploads it recursively, preserving relative paths: ```bash ENV=ascii-tree-env nemo files filesets create "$ENV" --workspace default --purpose environment --exist-ok nemo files upload ./ascii-tree-pkg/ "$ENV" --workspace default nemo files list "$ENV" --workspace default DATA=ascii-tree-env-dataset nemo files filesets create "$DATA" --workspace default --purpose dataset --exist-ok nemo files upload ./ascii-tree-data/ "$DATA" --workspace default ``` The trailing slash on the local path matters — it selects the directory's *contents* rather than the directory itself. Without it everything nests one level deeper under the directory's basename, which leaves the manifest off the FileSet root. Confirm with `nemo files list` before submitting. `--purpose environment` is enforced, not cosmetic: submit rejects an environment FileSet whose purpose is anything other than `environment` or `generic`. Relative paths must be preserved. `config_paths` is matched against the FileSet listing, so a package flattened during upload fails with `config_paths reference files that are not in the package`. These are the names `--upload` would have used: it derives them from the hub slug as `-env` and `-env-dataset`. Pass `--environment-name` / `--dataset-name` to override. ## Submit the job There is no `grpo` subcommand. GRPO submits through `nemo customization rl submit`, selected by `training.type`. ```json { "model": "default/qwen3-0.6b", "dataset": "default/ascii-tree-env-dataset", "environment": "default/ascii-tree-env", "training": { "type": "grpo", "epochs": 1, "learning_rate": 1e-6, "max_seq_length": 2048, "batch_size": 32, "micro_batch_size": 1, "num_generations_per_prompt": 8, "temperature": 1.0, "parallelism": { "num_nodes": 1, "num_gpus_per_node": 1 } }, "output": { "name": "qwen3-0.6b-ascii-tree" } } ``` ```bash nemo customization rl submit ./job.json --workspace default ``` `training.type` is required — it is the union discriminator, and omitting it fails with `union_tag_not_found` rather than defaulting to DPO. `model`, `dataset`, and `environment` are plain string references. Read the `rl-` job id from the `name` field of the submit response; `rl submit` has no `--name` flag. ## Train an adapter instead of full weights GRPO trains every weight by default. Set `finetuning_type` to train a LoRA adapter instead — useful when the full-weight run does not fit in memory, or when you want several environments' worth of behaviour over one shared base deployment. ```json { "model": "default/qwen3-0.6b", "dataset": "default/ascii-tree-env-dataset", "environment": "default/ascii-tree-env", "training": { "type": "grpo", "finetuning_type": "lora", "lora": { "rank": 16, "alpha": 32, "dropout": 0.0 }, "epochs": 1, "batch_size": 32, "num_generations_per_prompt": 8 }, "output": { "name": "qwen3-0.6b-ascii-tree" } } ``` | `finetuning_type` | Trains | Produces | | ----------------------- | ------------ | ------------------------------------------------------------ | | `all_weights` (default) | Every weight | A model entity, which needs its own deployment | | `lora` | An adapter | An HF PEFT adapter entity, registered against the base model | | `lora_merged` | — | Rejected. GRPO does not merge adapters at export. | You do not set the output type. It is inferred from `finetuning_type`, so `output` carries only `name` either way. ### LoRA fields | Field | Default | Description | | ----------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `rank` | `16` | LoRA rank. | | `alpha` | `32` | Scaling factor. The effective learning-rate multiplier on the adapter is `alpha / rank`, so changing `rank` alone also changes the effective learning rate. | | `dropout` | `0.0` | Dropout applied after the adapter. | | `target_modules` | `null` | Module names to adapt. | | `exclude_modules` | `null` | Module name patterns to skip. | | `use_triton` | unset → compiler picks | Triton LoRA kernels: `true` at `tensor_parallel_size` 1, `false` above. Leave it unset. Setting it `true` with `tensor_parallel_size` above 1 is rejected at submit, because the kernels cannot accept the sharded adapter weights tensor parallelism produces. | Module selection is either/or. Leaving both `target_modules` and `exclude_modules` unset adapts **every** linear layer. Setting either list turns that behavior off, so `exclude_modules` on its own does not mean "all linear layers except these." Use `target_modules` when you want a specific set, and neither field when you want all of them. Omitting the `lora` block while setting `finetuning_type: "lora"` is fine — defaults are filled in. Supplying a `lora` block alongside `all_weights` is rejected. ### Using the adapter The adapter hot-reloads onto a **READY** deployment of the base model that has `lora_enabled: true`, so there is no new deployment to create before evaluating it. Route inference through the provider gateway (`/provider//-/v1` with `model: default--`); the model-entity path always resolves to the base model. A full-weight GRPO job instead registers a new model entity, which does need its own deployment. ## Key hyperparameters All GRPO knobs live under `training`, including `policy_backend` (a sibling of `parallelism`, not a field on it). | Field | Default | Notes | | ---------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `num_generations_per_prompt` | `8` | Group size. The spread of rewards within a group is the entire learning signal. | | `num_prompts_per_step` | Derived | From `batch_size / num_generations_per_prompt`. Their product must be a multiple of `batch_size`, so prefer a value that divides `batch_size` evenly. | | `temperature` | `1.0` | Must be greater than 0. Greedy sampling makes every response in a group identical and the run a no-op. | | `max_rollout_turns` | `1` | Multi-turn rollouts. Most single-step environments use `1`. | | `overlong_filtering` | `false` | Zero the loss contribution of responses truncated at the generation cap. | | `ref_policy_kl_penalty` | `0.0` | KL coefficient against the reference policy. | | `val_at_start` | `false` | Run validation before the first step, so baseline and result come from one job. | | `policy_backend` | `automodel` | The NeMo-RL worker that trains the model, chosen explicitly and never inferred. Set it on `training` (for example `training.policy_backend`), not under `training.parallelism`. `automodel` supports LoRA, `expert_parallel_size` above 1, and `automodel_kwargs`, and needs Transformer Engine (Hopper or newer). `dtensor` runs stock HuggingFace modules on PyTorch FSDP2 for pre-Hopper GPUs, and supports none of those three. Requesting an `automodel`-only feature under `dtensor` is rejected at submit, with every conflict reported at once. | | `batching_strategy` | `dynamic` | How rollouts are grouped into training micro-batches. `dynamic` fills each micro-batch to a token budget, so short rollouts share a batch instead of each paying for a full-length pad. `static` puts one rollout per slot. `sequence_packing` concatenates rollouts, and is rejected for VLM, multimodal, and context-parallel runs. | | `train_mb_tokens` | Derived | Token budget per micro-batch, read by `dynamic` and `sequence_packing`. Defaults to `max_seq_length × micro_batch_size`. | | `sequence_length_round` | `64` | Round bucketed sequence lengths up to a multiple of this. Read only by `dynamic`. | | `router_aux_loss_coef` | Unset | MoE router auxiliary-loss coefficient, applied as a **top-level** HuggingFace config override. Set `0.0` to drop the load-balancing term during RL. | | `hf_config_overrides` | Unset | Passed to NeMo-RL verbatim and forwarded to the training model as config kwargs and to vLLM as `hf_overrides`. Nested keys are preserved, so use this for models that namespace their config — Qwen3.5 reads `router_aux_loss_coef` under `text_config`. | | `automodel_kwargs` | Unset | Passed through to Automodel. `{"force_hf": true}` loads stock HuggingFace modules when a model's custom backbone is not compatible with the parallelizer. Requires `policy_backend: automodel`. | Setting `router_aux_loss_coef` **and** a `router_aux_loss_coef` key inside `hf_config_overrides` is rejected — they write the same top-level key. Keep one. Use `hf_config_overrides` when the model nests it, because a top-level key the model does not read is absorbed silently and the aux loss stays on, surfacing only as degraded accuracy many steps in. Advanced clipping and advantage-estimation settings (`ratio_clip_c`, `advantage_clip_low` / `advantage_clip_high`, `normalize_rewards`, `use_leave_one_out_baseline`, `top_k`) are documented in [Training Configuration](/documentation/customizer-reference/manage-customization-jobs/training-configuration). ## Read the results Monitor GRPO on **reward**, not loss. The GRPO surrogate loss oscillates near zero and carries no signal about run quality. | Metric | Read it for | | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `train_reward` | Mean reward over the step's rollouts. The headline. | | `val_accuracy` | The validation pass's mean reward. NeMo-RL names it `accuracy`; it is not an accuracy in the classifier sense, and there is no `val_reward`. | | `train_truncation_rate` | Rising truncation is a common reason reward stops improving | | `train_baseline_reward/pct_mixed` | Share of prompt groups whose responses disagreed. Falling toward zero means there is no gradient left to learn from | | `train_approx_entropy` | Entropy collapse — responses degenerate while reward still looks acceptable | | `train_token_mult_prob_error`, `train_sampling_importance_ratio` | Drift between the sampling policy and the training policy. Both sit near 1 on a healthy run. | | `train_timing/total_step_time` | Wall clock per training step, in seconds | `status_details` also carries two constants stated once when training starts: `training_type` (`grpo` or `dpo` — `backend` is `nemo_rl` for both) and `rollouts_per_step`, so the rollouts generated so far is `step × rollouts_per_step`. `train_total_reward/stddev`, `/p25` and `/p75` describe reward spread, but they are close to uninformative when the environment returns a binary 0/1 reward — which most verifier environments do. On a Bernoulli reward, `stddev` is a deterministic function of the mean and adds nothing to `train_reward`, and the quartiles collapse to three states: `[0, 0]` below mean reward 0.25, `[0, 1]` between 0.25 and 0.75, `[1, 1]` above. A `p25` pinned at 0 for a whole run is correct, not a bug. Use `baseline_reward/pct_0` / `pct_1` / `pct_mixed` as the spread for binary rewards; keep the dispersion series for continuous or multi-component rewards. ## Troubleshooting | Message | Fix | | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Missing nemo-environment.yaml at environment root` | Place the manifest at the FileSet root, not in a subdirectory | | `config_paths reference files that are not in the package` | Re-upload preserving relative paths | | `Prompt JSONL must not live in the environment package` | Move rows to the dataset FileSet | | `native-v1 config_paths must be under (...)` | Move the config under a Gym server directory, or switch format | | `adapter-wheels-v1 config_paths should live under configs/` | Move the config under `configs/` | | `must carry a non-empty wheels/ directory` | Vendor the dependency closure | | `adapter.agent ... is not built into the training image` | Use a supported harness, or switch format | | `ServerRefNotFoundError: ... Available responses_api_models: (none)` | A config references a model server nothing defines — include `configs/policy_model.yaml` | | `Environment fileset ... has purpose 'dataset'; expected purpose='environment'` | Recreate the fileset with `--purpose environment` | | `AlmostServerError`, after an "Almost-Servers Detected" banner | A server block failed validation — most often a `domain` that is missing, empty, or not in the closed set. Set a valid `domain` on every `resources_servers` block | | `Missing pyproject.toml or requirements.txt for uv venv setup in server dir` | A custom implementation directory has no install marker. Add `requirements.txt` | | The job runs but scores an environment you did not ship | Same missing install marker, on a directory whose name matches a Gym built-in — the built-in was used instead. Add `requirements.txt`, or rename the implementation | | `ModuleNotFoundError` at the first rollout | The server's venv is missing an import: an empty or incomplete `requirements.txt`, or a wheelhouse that did not cover the venv build | | `your requirements are unsatisfiable` | `wheels/` vendors two versions of one project; clear it and rebuild the closure | | `OpenSandbox is not yet available on this cluster` | Operator setting. The cluster has not enabled sandboxed Gym — see [Cluster prerequisites](#cluster-prerequisites) | | `Sandboxed GRPO requires the job-storage PVC claim name` | Operator setting, same section | | `has no wheels with a matching platform tag` | The closure was built for the wrong interpreter or architecture — rebuild per [Vendoring a wheel closure](#vendoring-a-wheel-closure) | | Spin-up hangs, then fails on a network or resolver error | The environment's venv build cannot reach an index. See [How dependencies are installed](#how-dependencies-are-installed) | ### Cluster prerequisites GRPO fails at submit, before any GPU is claimed, unless the platform operator has configured sandboxed Gym. Both settings live on the platform, not in the job JSON: | Setting | Why | | ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `NMP_SANDBOX_CLUSTER_CAPABLE=true` (Helm: `sandboxClusterCapable`) | Declares that OpenSandbox is installed. The platform chart does **not** install it. GRPO defaults to sandboxed Gym (`NMP_RL_SANDBOXED_GYM_DEFAULT`) and fails closed without this | | `NMP_RL_JOB_STORAGE_PVC_CLAIM` | The Gym sandbox mounts the job-storage claim itself to read the downloaded environment and dataset, and only learns the claim by name | | `NMP_RL_SANDBOX_ALLOW_INTERNET` | Needed whenever an environment's venv build has to reach a package index — which is every `native-v1` package, and every `adapter-wheels-v1` one | | `NMP_RL_SANDBOX_PUBLIC_DNS_ALLOW` | Extra hosts, consulted only when the above is on. The built-in allowance covers `*.com` and `*.org`, so a host on another TLD (for example `hub.primeintellect.ai`) must be named | If a submit fails on either of the first two, that is a platform configuration gap, not a problem with your package. Neither is settable per job. Installing OpenSandbox is a separate operator task: [OpenSandbox](/documentation/self-managed-deployment/setup/helm/opensandbox), or [OpenSandbox with Kata](/documentation/self-managed-deployment/setup/helm/opensandbox-kata) for the Kata runtime. ## Next Steps * Build the environment FileSet itself in [GRPO Environment Packages](/documentation/customizer-reference/tutorials/grpo-environment-packages). * Monitor reward and the other training curves in [Check Customization Job Metrics](/documentation/customizer-reference/tutorials/metrics). * Compare a trained model against its base in [Evaluate Models & Agents](/documentation/evaluate-models).