> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.orchestration.ray_serve_gateway

Ray Serve gateway that launches multiple vLLM instances and routes requests across them.

Selected automatically (see `api.effective_ray_serve`) whenever an instance's TP/PP footprint
would need to span multiple Slurm nodes, or via `use_ray_serve: true`. Joins the Ray cluster
already bootstrapped by the sbatch script and defines one Ray Serve deployment with
`number_of_instances` replicas; each replica launches its own `vllm serve --distributed-executor-backend ray` subprocess and proxies requests to it. Ray Serve's own
`max_replicas_per_node` and HTTP proxy handle node placement and routing, replacing what used to
be hand-rolled here. Deliberately not using `ray.serve.llm`: it conflicts with vLLM's own
RayDistributedExecutor over nested placement groups (ray-project/ray#59064).

## Module Contents

### Classes

| Name                                                                     | Description                                                                              |
| ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- |
| [`VLLMInstance`](#nemo_gym-orchestration-ray_serve_gateway-VLLMInstance) | One Ray Serve replica = one vLLM instance, proxying every request to its own subprocess. |

### Functions

| Name                                                                                         | Description                                                                                  |
| -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| [`build_instance_command`](#nemo_gym-orchestration-ray_serve_gateway-build_instance_command) | The `vllm serve` command each replica runs for its own instance.                             |
| [`free_local_port`](#nemo_gym-orchestration-ray_serve_gateway-free_local_port)               | An OS-assigned free TCP port on this node, since colocated replicas can't share a fixed one. |
| [`main`](#nemo_gym-orchestration-ray_serve_gateway-main)                                     | -                                                                                            |
| [`max_replicas_per_node`](#nemo_gym-orchestration-ray_serve_gateway-max_replicas_per_node)   | How many instance drivers may share one physical node's GPU capacity; None if unknown.       |
| [`parse_args`](#nemo_gym-orchestration-ray_serve_gateway-parse_args)                         | -                                                                                            |

### Data

[`HEALTH_PATH`](#nemo_gym-orchestration-ray_serve_gateway-HEALTH_PATH)

[`HEALTH_POLL_INTERVAL_S`](#nemo_gym-orchestration-ray_serve_gateway-HEALTH_POLL_INTERVAL_S)

[`HEALTH_TIMEOUT_S`](#nemo_gym-orchestration-ray_serve_gateway-HEALTH_TIMEOUT_S)

[`app`](#nemo_gym-orchestration-ray_serve_gateway-app)

[`logger`](#nemo_gym-orchestration-ray_serve_gateway-logger)

### API

```python
class nemo_gym.orchestration.ray_serve_gateway.VLLMInstance(
    model: str,
    tensor_parallel_size: int,
    pipeline_parallel_size: int,
    trust_remote_code: bool,
    served_model_name: str | None = None,
    extra_args: str = ''
)
```

One Ray Serve replica = one vLLM instance, proxying every request to its own subprocess.

**`_base_url`** `= f'http://localhost:{port}'`

---

**`_proc`** `= subprocess.Popen(cmd, env=env)`

---

**`_session`** `= aiohttp.ClientSession()`

---

```python
nemo_gym.orchestration.ray_serve_gateway.VLLMInstance._wait_until_healthy() -> None
```

```python
nemo_gym.orchestration.ray_serve_gateway.VLLMInstance.check_health() -> None
```

```python
nemo_gym.orchestration.ray_serve_gateway.VLLMInstance.health() -> fastapi.Response
```

async

```python
nemo_gym.orchestration.ray_serve_gateway.VLLMInstance.proxy(
    request: fastapi.Request,
    path: str
) -> fastapi.Response
```

async

```python
nemo_gym.orchestration.ray_serve_gateway.build_instance_command(
    model: str,
    tensor_parallel_size: int,
    pipeline_parallel_size: int,
    trust_remote_code: bool,
    port: int,
    served_model_name: str | None = None,
    extra_args: str = ''
) -> list[str]
```

The `vllm serve` command each replica runs for its own instance.

```python
nemo_gym.orchestration.ray_serve_gateway.free_local_port() -> int
```

An OS-assigned free TCP port on this node, since colocated replicas can't share a fixed one.

```python
nemo_gym.orchestration.ray_serve_gateway.main(
    argv: list[str] | None = None
) -> None
```

```python
nemo_gym.orchestration.ray_serve_gateway.max_replicas_per_node(
    tensor_parallel_size: int,
    pipeline_parallel_size: int,
    gpus_per_node: int | None
) -> int | None
```

How many instance drivers may share one physical node's GPU capacity; None if unknown.

```python
nemo_gym.orchestration.ray_serve_gateway.parse_args(
    argv: list[str] | None = None
) -> argparse.Namespace
```

```python
nemo_gym.orchestration.ray_serve_gateway.HEALTH_PATH = '/health'
```

```python
nemo_gym.orchestration.ray_serve_gateway.HEALTH_POLL_INTERVAL_S = 5.0
```

```python
nemo_gym.orchestration.ray_serve_gateway.HEALTH_TIMEOUT_S = 900.0
```

```python
nemo_gym.orchestration.ray_serve_gateway.app = FastAPI()
```

```python
nemo_gym.orchestration.ray_serve_gateway.logger = logging.getLogger(__name__)
```