nemo_gym.orchestration.ray_serve_gateway

Ray Serve gateway that launches multiple vLLM instances and routes requests across them.

Selected automatically (see api.effective_ray_serve) whenever an instance’s TP/PP footprint would need to span multiple Slurm nodes, or via use_ray_serve: true. Joins the Ray cluster already bootstrapped by the sbatch script and defines one Ray Serve deployment with number_of_instances replicas; each replica launches its own vllm serve --distributed-executor-backend ray subprocess and proxies requests to it. Ray Serve’s own max_replicas_per_node and HTTP proxy handle node placement and routing, replacing what used to be hand-rolled here. Deliberately not using ray.serve.llm: it conflicts with vLLM’s own RayDistributedExecutor over nested placement groups (ray-project/ray#59064).

Module Contents

Classes

Name	Description
`VLLMInstance`	One Ray Serve replica = one vLLM instance, proxying every request to its own subprocess.

Functions

Name	Description
`build_instance_command`	The `vllm serve` command each replica runs for its own instance.
`free_local_port`	An OS-assigned free TCP port on this node, since colocated replicas can’t share a fixed one.
`main`	-
`max_replicas_per_node`	How many instance drivers may share one physical node’s GPU capacity; None if unknown.
`parse_args`	-

Data

HEALTH_PATH

HEALTH_POLL_INTERVAL_S

HEALTH_TIMEOUT_S

app

logger

API

class nemo_gym.orchestration.ray_serve_gateway.VLLMInstance(
    model: str,
    tensor_parallel_size: int,
    pipeline_parallel_size: int,
    trust_remote_code: bool,
    served_model_name: str | None = None,
    extra_args: str = ''
)

One Ray Serve replica = one vLLM instance, proxying every request to its own subprocess.

_base_url

= f'http://localhost:{port}'

_proc

= subprocess.Popen(cmd, env=env)

_session

= aiohttp.ClientSession()

nemo_gym.orchestration.ray_serve_gateway.VLLMInstance._wait_until_healthy() -> None

nemo_gym.orchestration.ray_serve_gateway.VLLMInstance.check_health() -> None

nemo_gym.orchestration.ray_serve_gateway.VLLMInstance.health() -> fastapi.Response

async

nemo_gym.orchestration.ray_serve_gateway.VLLMInstance.proxy(
    request: fastapi.Request,
    path: str
) -> fastapi.Response

async

nemo_gym.orchestration.ray_serve_gateway.build_instance_command(
    model: str,
    tensor_parallel_size: int,
    pipeline_parallel_size: int,
    trust_remote_code: bool,
    port: int,
    served_model_name: str | None = None,
    extra_args: str = ''
) -> list[str]

The vllm serve command each replica runs for its own instance.

nemo_gym.orchestration.ray_serve_gateway.free_local_port() -> int

An OS-assigned free TCP port on this node, since colocated replicas can’t share a fixed one.

nemo_gym.orchestration.ray_serve_gateway.main(
    argv: list[str] | None = None
) -> None

nemo_gym.orchestration.ray_serve_gateway.max_replicas_per_node(
    tensor_parallel_size: int,
    pipeline_parallel_size: int,
    gpus_per_node: int | None
) -> int | None

How many instance drivers may share one physical node’s GPU capacity; None if unknown.

nemo_gym.orchestration.ray_serve_gateway.parse_args(
    argv: list[str] | None = None
) -> argparse.Namespace

nemo_gym.orchestration.ray_serve_gateway.HEALTH_PATH = '/health'

nemo_gym.orchestration.ray_serve_gateway.HEALTH_POLL_INTERVAL_S = 5.0

nemo_gym.orchestration.ray_serve_gateway.HEALTH_TIMEOUT_S = 900.0

nemo_gym.orchestration.ray_serve_gateway.app = FastAPI()

nemo_gym.orchestration.ray_serve_gateway.logger = logging.getLogger(__name__)