Ray Deployment | NVIDIA NeMo Evaluator

Distribute evaluations across a Ray cluster for teams using NeMo Gym’s Ray infrastructure.

Setup

$ pip install -e ".[ray]"

CLI

$ export NVIDIA_API_KEY="your-api-key-here"
$ 
$ ray job submit --working-dir . -- python -m nemo_evaluator.engine.ray_launcher \
>     --benchmark gsm8k --shards 8 --repeats 5 \
>     --model-url https://integrate.api.nvidia.com/v1/chat/completions \
>     --model-id nvidia/nemotron-3-nano-30b-a3b \
>     --api-key $NVIDIA_API_KEY \
>     --output-dir ./eval_results/ray

Architecture

Each shard runs as a @ray.remote task. The head node collects all results and merges them locally.

Python API

1 import ray
2 from nemo_evaluator.engine.ray_launcher import run_shard
3 
4 ray.init()
5 
6 shards = 8
7 futures = [
8     run_shard.remote(
9         benchmark="gsm8k",
10         shard_idx=i,
11         total_shards=shards,
12         model_url="https://inference-api.nvidia.com/v1",
13         model_id="azure/openai/gpt-5.2",
14         n_repeats=5,
15     )
16     for i in range(shards)
17 ]
18 
19 results = ray.get(futures)
20 # results is a list of bundle dicts, one per shard

Resource requirements

Evaluation is CPU+network bound (no GPU needed for the evaluator itself):

1 @ray.remote(num_cpus=2, memory=2 * 1024 * 1024 * 1024)
2 def run_shard(...):
3     ...

Adjust based on dataset size and concurrency. The ModelClient default is 8 concurrent requests.