> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# Quickstart

> Run your first agent evaluation with NeMo Gym.

> **Info**
>
> See [Installation](/get-started/installation) if you need to install NeMo Gym.

> **Tip**
>
> Working with an AI coding assistant? NeMo Gym ships [Agent Skills](/contribute/agent-skills) — vetted workflows for tasks like adding benchmarks, debugging rollouts, and editing docs.

## Configure Your Model

Create an `env.yaml` file in the project root with your model endpoint credentials:

```yaml
policy_base_url: https://api.openai.com/v1
policy_api_key: <your-openai-api-key>
policy_model_name: gpt-4.1-2025-04-14
```

> **Note**
>
> This quickstart uses OpenAI. NeMo Gym supports local and hosted inference — see [Configure Model](/model-server) for vLLM, Fireworks, OpenRouter, and others.

## Run Evaluation

Run your agent on a set of tasks and score the results. This example uses a simple tool calling agent [`simple_agent`](https://github.com/NVIDIA-NeMo/Gym/tree/main/responses_api_agents/simple_agent) with the [`mcqa`](https://github.com/NVIDIA-NeMo/Gym/tree/main/resources_servers/mcqa) (multiple-choice Q\&A) environment and its included example data.

**1. Start servers**

NeMo Gym uses local servers to coordinate your model, agent, and task verification. Start them first:

```bash
gym env start \
    --resources-server mcqa \
    --model-type openai_model
```

You should see three server instances starting:

```text
[1] mcqa (resources_servers/mcqa)
[2] mcqa_simple_agent (responses_api_agents/simple_agent)
[3] policy_model (responses_api_models/openai_model)
```

**2. Evaluate your agent**

In a new terminal, run your agent on a single task to verify everything works:

```bash
source .venv/bin/activate

gym eval run --no-serve \
    --agent mcqa_simple_agent \
    --input resources_servers/mcqa/data/example.jsonl \
    --output results/mcqa_rollouts.jsonl \
    --limit 5 \
    --num-repeats 1
```

You should see a progress bar followed by aggregate metrics:

```text
Collecting rollouts: 100%|██████| 5/5 [01:22<00:00, 16.44s/it]

Key metrics for mcqa_simple_agent:
{
    "mean/reward": 0.8,
    "pass@1[avg-of-1]/accuracy": 80.0,
    "pass@1/accuracy": 80.0
}
Finished rollout collection! View results at:
Fully materialized inputs: results/mcqa_rollouts_materialized_inputs.jsonl
Rollouts: results/mcqa_rollouts.jsonl
Aggregate metrics: results/mcqa_rollouts_aggregate_metrics.json
```

> **Warning**
>
> If `gym eval run` fails with a `500 Internal Server Error` on the `/run` endpoint, the most common cause is an invalid or missing API key. Verify that `policy_api_key` and `policy_base_url` in your `env.yaml` are correct.
>
> If `gym env start` instead stops with `model endpoint(s) never answered`, nothing is listening at the endpoint it names. See [Model Endpoint Unreachable](/troubleshooting/configuration#model-endpoint-unreachable).

For per-task pass rates, see `gym eval profile` in the [CLI Reference](/reference/cli-commands).

## Explore

Now that you have a working setup, explore what's available.

NeMo Gym ships with environments across many domains. You can use these existing environments in addition to building your own.

```bash
gym list benchmarks
```

```text
Available benchmarks in NeMo Gym (76)
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark name   ┃ Domain             ┃ Agent name                        ┃ Num repeats ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ aime24           │ math               │ aime24_math_with_judge_simple_ag… │ 32          │
│ aime25           │ math               │ aime25_math_with_judge_simple_ag… │ 32          │
│ browsecomp       │ agent              │ browsecomp_tavily_search_simple_… │ 1           │
│ gpqa             │ knowledge          │ gpqa_mcqa_simple_agent            │ 8           │
│ ifbench          │ instruction_fol…   │ ifbench_benchmark_simple_agent    │ 5           │
| ...              | ...                | ...                               | ...         |
│ tau2             │ agent              │ tau2_benchmark_agent              │ 8           │
│ xstest           │ safety             │ xstest_benchmark_simple_agent     │ 4           │
└──────────────────┴────────────────────┴───────────────────────────────────┴─────────────┘
```

This lists benchmarks with pre-configured agents. For the full set of environments (including training environments), see the [Available Environments](https://github.com/NVIDIA-NeMo/Gym#-available-environments) table.

Every CLI command supports `-h` or `--help` for detailed usage information:

```bash
gym --help
gym env start --help
```

## Next Steps

#### [Browse Environments](https://github.com/NVIDIA-NeMo/Gym#-available-environments)

Browse available environments for evaluation and training.

#### [Agents](/agent-server)

Explore available agent harnesses and learn how to integrate your own agent.

#### [Training](/tutorials/training-tutorials)

Improve your agent or model with RL or fine-tuning.

#### [Build Custom Environments](/environment-tutorials)

Create your own evaluation or training environments.