Inference Server

InferenceServer serves one or more local models behind an OpenAI-compatible endpoint on your Ray cluster. Choose a typed backend configuration for either Ray Serve or NVIDIA Dynamo; the client-facing endpoint and lifecycle API stay the same. Both backends support the serving capabilities described in this guide, so benchmark your workload and choose the backend based on measured performance.

Prerequisites

Install the inference-server extra:

$
uv pip install "nemo-curator[inference_server]"

Local GPU serving through this extra targets x86_64 Linux. The vLLM, ai-dynamo, and NIXL dependencies are not installed by the extra on aarch64 or macOS.

The tested 26.07 stack uses:

• CUDA 12.9.1 in the NeMo Curator container.
• PyTorch 2.10.
• Ray Serve 2.55.1 or later.
• vLLM 0.18.x for inference_server (vllm<0.19).
• ai-dynamo 1.1.0 and NIXL 0.10.0 or later for the Dynamo backend.

Start or connect to a Ray cluster before creating the server. The examples below use RayClient; an externally managed Ray cluster works as well.

Ray Serve Quickstart

Ray Serve is the default backend. Omitting backend= is equivalent to passing RayServeServerConfig().

1
from openai import OpenAI
2
3
from nemo_curator.core.client import RayClient
4
from nemo_curator.core.serve import InferenceServer, RayServeModelConfig
5
6
ray_client = RayClient(num_cpus=8, num_gpus=1)
7
ray_client.start()
8
9
model = RayServeModelConfig(
10
    model_identifier="HuggingFaceTB/SmolLM2-135M-Instruct",  # pragma: allowlist secret
11
    deployment_config={
12
        "autoscaling_config": {
13
            "min_replicas": 1,
14
            "max_replicas": 1,
15
        },
16
    },
17
    engine_kwargs={
18
        "tensor_parallel_size": 1,
19
        "max_model_len": 2048,
20
    },
21
)
22
23
with InferenceServer(models=[model]) as server:
24
    client = OpenAI(base_url=server.endpoint, api_key="unused")
25
    response = client.chat.completions.create(
26
        model="HuggingFaceTB/SmolLM2-135M-Instruct",  # pragma: allowlist secret
27
        messages=[{"role": "user", "content": "Say hello in one word."}],
28
        max_tokens=16,
29
    )
30
    print(response.choices[0].message.content)

The server waits for every configured model to appear at /v1/models before start() returns. server.endpoint contains the correct host and port for the selected backend.

Shared Server API

InferenceServer accepts a list of model configurations and one matching server configuration.

Only one InferenceServer can be active in a Python process at a time. Stop the current server before starting another.

Use a context manager for automatic cleanup:

1
with InferenceServer(models=[model]) as server:
2
    print(server.endpoint)

For explicit lifecycle control:

1
server = InferenceServer(models=[model])
2
server.start()
3
try:
4
    print(server.endpoint)
5
finally:
6
    server.stop()

Ray Serve Configuration

RayServeModelConfig

Use model_name when weights come from a local path but clients should use a stable API name:

1
model = RayServeModelConfig(
2
    model_identifier="/models/gemma-3-27b-it",
3
    model_name="google/gemma-3-27b-it",
4
    deployment_config={
5
        "autoscaling_config": {"min_replicas": 1, "max_replicas": 2},
6
    },
7
    engine_kwargs={"tensor_parallel_size": 4},
8
)

Multiple Ray Serve Models

Each model can use a distinct deployment and runtime environment:

1
from nemo_curator.core.serve import InferenceServer, RayServeModelConfig
2
3
models = [
4
    RayServeModelConfig(
5
        model_identifier="HuggingFaceTB/SmolLM2-135M-Instruct",  # pragma: allowlist secret
6
        model_name="writer",
7
        deployment_config={
8
            "autoscaling_config": {"min_replicas": 1, "max_replicas": 2},
9
        },
10
        engine_kwargs={"tensor_parallel_size": 1},
11
    ),
12
    RayServeModelConfig(
13
        model_identifier="HuggingFaceTB/SmolLM-135M-Instruct",  # pragma: allowlist secret
14
        model_name="reviewer",
15
        deployment_config={
16
            "autoscaling_config": {"min_replicas": 1, "max_replicas": 1},
17
        },
18
        engine_kwargs={"tensor_parallel_size": 1},
19
    ),
20
]
21
22
server = InferenceServer(models=models)
23
server.start()

Clients select writer or reviewer in the OpenAI request’s model field.

Dynamo Aggregated Serving

Aggregated mode runs prefill and decode in the same vLLM worker. num_replicas creates static replicas; Dynamo does not use Ray Serve autoscaling configuration.

1
from nemo_curator.core.serve import (
2
    DynamoServerConfig,
3
    DynamoVLLMModelConfig,
4
    InferenceServer,
5
)
6
7
model = DynamoVLLMModelConfig(
8
    model_identifier="HuggingFaceTB/SmolLM2-135M-Instruct",  # pragma: allowlist secret
9
    mode="aggregated",
10
    num_replicas=2,
11
    engine_kwargs={
12
        "tensor_parallel_size": 1,
13
        "max_model_len": 2048,
14
    },
15
)
16
17
server = InferenceServer(
18
    models=[model],
19
    backend=DynamoServerConfig(),
20
    health_check_timeout_s=600,
21
)
22
server.start()

For aggregated models, a tensor-parallel replica can span multiple nodes when the tensor-parallel size divides evenly across available nodes. NeMo Curator prefers a single-node placement and otherwise uses equal GPU bundles on distinct nodes.

Dynamo Disaggregated Serving

Disaggregated mode runs independent prefill and decode workers. Configure both roles explicitly.

1
from nemo_curator.core.serve import (
2
    DynamoRoleConfig,
3
    DynamoServerConfig,
4
    DynamoVLLMModelConfig,
5
    InferenceServer,
6
)
7
8
model = DynamoVLLMModelConfig(
9
    model_identifier="HuggingFaceTB/SmolLM2-135M-Instruct",  # pragma: allowlist secret
10
    mode="disagg",
11
    engine_kwargs={
12
        "max_model_len": 2048,
13
        "tensor_parallel_size": 1,
14
    },
15
    prefill=DynamoRoleConfig(
16
        num_replicas=2,
17
        engine_kwargs={"tensor_parallel_size": 2},
18
    ),
19
    decode=DynamoRoleConfig(
20
        num_replicas=1,
21
        engine_kwargs={"tensor_parallel_size": 1},
22
    ),
23
)
24
25
server = InferenceServer(
26
    models=[model],
27
    backend=DynamoServerConfig(),
28
    health_check_timeout_s=600,
29
)
30
server.start()

Role-level engine_kwargs shallow-merge over the model-level values. In this example, prefill uses tensor parallelism of 2 and decode uses 1; both inherit max_model_len.

Disaggregated serving uses the NIXL connector for KV transfer by default. kv_transfer_config and kv_events_config are managed by NeMo Curator and are not constructor parameters.

Dynamo Configuration Reference

DynamoVLLMModelConfig

All models in one server must have unique model_name values. Dynamo also rejects names that sanitize to the same component slug.

DynamoRoleConfig

DynamoServerConfig

Dynamo Routing

1
from nemo_curator.core.serve import DynamoRouterConfig, DynamoServerConfig
2
3
backend = DynamoServerConfig(
4
    router=DynamoRouterConfig(
5
        mode="kv",
6
        kv_events=False,
7
        router_kwargs={"dyn_chat_processor": "vllm"},
8
    ),
9
)

kv_events is valid only with KV routing. When you explicitly set mode="kv", kv_events=False uses approximate tree-based tracking. When mode=None auto-selects KV routing for a disaggregated model, NeMo Curator also enables exact event-backed routing even though the configured default is False. If a publishing role explicitly enables vLLM’s hybrid KV-cache manager, automatic routing instead keeps events disabled; explicitly requesting event-backed routing with that configuration raises an error.

Additional router_kwargs are forwarded to the Dynamo frontend. Do not put router_mode or router_kv_events in that dictionary; use the typed fields instead. Boolean false values are emitted as --no-* flags.

For multimodal OpenAI content arrays, the Dynamo frontend can require:

1
DynamoRouterConfig(
2
    router_kwargs={"dyn_chat_processor": "vllm"},
3
)

The worker also needs its model-specific multimodal settings, such as limit_mm_per_prompt in engine_kwargs and enable_multimodal in dynamo_kwargs.

Runtime Environments and Subprocess Variables

Both model types inherit runtime_env from BaseModelConfig:

1
model = DynamoVLLMModelConfig(
2
    model_identifier="my-org/my-model",
3
    runtime_env={
4
        "uv": {"packages": ["my-model-plugin==1.2.0"]},
5
        "env_vars": {"HF_HOME": "/models/hf-cache"},
6
    },
7
)

NeMo Curator merges pip or uv package lists and environment variables instead of replacing the backend’s required packages. Dynamo automatically adds its tested ai-dynamo[vllm] actor environment and pins the actor’s Ray version to the cluster version. On a new cluster node, the first actor can take several minutes to create this cached environment.

Use DynamoServerConfig.subprocess_env for variables that must reach the Dynamo frontend and worker subprocesses:

1
backend = DynamoServerConfig(
2
    subprocess_env={"DYN_TCP_REQUEST_TIMEOUT": "180"},
3
)

Resource Placement

Before starting Dynamo, NeMo Curator checks the total requested GPUs and fails early when the cluster is too small.

• Aggregated GPU count is num_replicas * tensor_parallel_size per model.
• Disaggregated GPU count is the sum of each role’s num_replicas * tensor_parallel_size.
• Aggregated tensor-parallel groups prefer one node, then use an equal multi-node split.
• Each disaggregated role must fit on one node.
• Set CURATOR_IGNORE_RAY_HEAD_NODE=1 to keep model placement off the Ray head node when worker nodes are labeled for that policy.

Dynamo creates detached, named placement groups and actors. On startup, it removes stale resources with the same server-name prefix. On shutdown, it reacquires actor handles by name, terminates the subprocess groups, and removes the placement groups.

HAProxy Ingress

The NeMo Curator container includes HAProxy and socat. When both binaries are available, local Ray cluster initialization enables Ray Serve’s HAProxy ingress and assigns a free metrics port. If either binary is unavailable, Ray Serve uses its default Python proxy.

No HAProxy configuration is required in InferenceServer. To confirm the optimized path, check startup logs for Ray Serve HAProxy ingress enabled.

Use with NeMo Curator Clients

Point any OpenAI-compatible client at server.endpoint:

1
from nemo_curator.models.client.openai_client import AsyncOpenAIClient
2
3
client = AsyncOpenAIClient(
4
    base_url=server.endpoint,
5
    api_key="unused",
6
    max_concurrent_requests=10,
7
)

For Dynamo, the endpoint host is the node running the frontend placement-group bundle. Use server.endpoint instead of assuming localhost.

Run Pipelines While Serving

Ray Data and Ray Actor Pool executors honor Ray’s GPU accounting and schedule GPU stages away from GPUs held by either inference backend. Xenna manages GPU assignment independently and is rejected when an active InferenceServer and a GPU pipeline stage would conflict.

1
from nemo_curator.backends.ray_data import RayDataExecutor
2
3
with InferenceServer(models=[model]) as server:
4
    results = pipeline.run(executor=RayDataExecutor())

CPU-only pipelines can use any executor while the server is active.

Migrate from InferenceModelConfig

Before:

1
from nemo_curator.core.serve import InferenceModelConfig, InferenceServer
2
3
model = InferenceModelConfig(
4
    model_identifier="google/gemma-3-27b-it",
5
    deployment_config={
6
        "autoscaling_config": {"min_replicas": 1, "max_replicas": 1},
7
    },
8
    engine_kwargs={"tensor_parallel_size": 4},
9
)
10
server = InferenceServer(models=[model])

After, using Ray Serve:

1
from nemo_curator.core.serve import InferenceServer, RayServeModelConfig
2
3
model = RayServeModelConfig(
4
    model_identifier="google/gemma-3-27b-it",
5
    deployment_config={
6
        "autoscaling_config": {"min_replicas": 1, "max_replicas": 1},
7
    },
8
    engine_kwargs={"tensor_parallel_size": 4},
9
)
10
server = InferenceServer(models=[model])

Or, using static Dynamo replicas:

1
from nemo_curator.core.serve import (
2
    DynamoServerConfig,
3
    DynamoVLLMModelConfig,
4
    InferenceServer,
5
)
6
7
model = DynamoVLLMModelConfig(
8
    model_identifier="google/gemma-3-27b-it",
9
    num_replicas=1,
10
    engine_kwargs={"tensor_parallel_size": 4},
11
)
12
server = InferenceServer(models=[model], backend=DynamoServerConfig())

Do not pass Ray Serve’s deployment_config to DynamoVLLMModelConfig. Use num_replicas or the disaggregated role counts instead.

Troubleshooting

Model Does Not Become Healthy

• Increase health_check_timeout_s for large model downloads or slow actor-environment creation.
• Check the model name returned by /v1/models; local paths often need an explicit model_name alias.
• Confirm the requested tensor-parallel groups fit the cluster topology.
• For Dynamo, inspect subprocess logs under the Ray session’s nemo_curator_dynamo_<id> directory.

Dynamo Cannot Start etcd or NATS

Install the NeMo Curator container dependencies, or supply existing etcd_endpoint and nats_url values in DynamoServerConfig.

Local Multi-GPU Startup Hangs

On PCIe systems without peer-to-peer GPU access, restart Ray or the Python kernel and either set NCCL_P2P_DISABLE=1 before starting the cluster or reduce tensor_parallel_size to 1.

Runtime Environment Times Out

Ensure cluster nodes can reach the package index and share compatible Python, CUDA, and Ray versions. Dynamo actor setup uses a 600-second timeout; prebuild the NeMo Curator container when workers cannot install packages at runtime.

Next Steps

• LLM Client Setup
• NeMo Data Designer
• Execution Backends
• Per-Stage Runtime Environments