nemo_automodel.recipes.llm.serve_dspark_target

Serve a frozen DSpark target to a remote training run (train-inference disaggregation).

It reads the same YAML as train_dspark (recipe_args selects the target and the captured layers), listens on target_server.host / target_server.port (default 0.0.0.0:8001), and runs the target in this single process::

python -m nemo_automodel.recipes.llm.serve_dspark_target
-c examples/speculative/dspark/qwen3_0.6b_dspark_remote.yaml

See “Target Backends” in examples/speculative/README.md for the training side.

Module Contents

Functions

Name	Description
`build_server`	Load the frozen target described by `cfg` and wrap it in the request-handling logic.
`main`	CLI entry point. Parses `-c <config.yaml>`, loads the target, and serves until a client disconnects.

Data

logger

API

nemo_automodel.recipes.llm.serve_dspark_target.build_server(
    cfg,
    host: str,
    nccl_port: int
) -> nemo_automodel.components.speculative.dspark.remote.DSparkTargetModelServer

Load the frozen target described by cfg and wrap it in the request-handling logic.

Parameters:

cfg

A train_dspark config; recipe_args selects the target and the captured layers.

host

str

Bind/advertise address, also the NCCL rendezvous master.

nccl_port

int

TCP rendezvous port for the NCCL data plane.

Returns: DSparkTargetModelServer

The server logic, ready for serve.

Raises:

NotImplementedError: When launched with more than one rank.

nemo_automodel.recipes.llm.serve_dspark_target.main(
    config_path: str | None = None
) -> None

CLI entry point. Parses -c <config.yaml>, loads the target, and serves until a client disconnects.

nemo_automodel.recipes.llm.serve_dspark_target.logger = logging.getLogger(__name__)