> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Writing Unified Backends

Dynamo's unified backend path lets custom engines implement Dynamo's shared
backend lifecycle contract. The engine owns inference; Dynamo owns
runtime registration, request serving, cancellation monitoring, signal handling,
drain, and graceful shutdown.

Use this path for new token-in-token-out engines unless you need a feature that
is still outside the unified contract.

## Choose an Implementation Language

| Path | Use when |
|---|---|
| Python unified backend | Your engine or serving library is Python-first, or you want the quickest path to integrate a custom engine. |
| Rust unified backend | You want a native Rust binary, tighter control of runtime dependencies, or no Python worker runtime. |
| Python workers (lower-level) | You need custom request handling or features not yet covered by the unified backend contract. Use [Python Workers](/dynamo/dev/advanced-customizations/writing-custom-backends/writing-python-workers). |

Both unified implementations follow the same shape:

```text
parse config -> start engine -> stream generated chunks -> abort/drain -> cleanup
```

The framework handles model registration, endpoint serving, cancellation
plumbing, and shutdown behavior around that engine contract.

## What the Unified Contract Covers

The shared contract provides:

- aggregated token-in-token-out inference
- disaggregated serving modes for supported engines
- model registration through Dynamo discovery
- request cancellation
- structured backend errors
- graceful shutdown and drain hooks

Use the lower-level Python worker path when your integration must own endpoint
registration, request handling, or the request payload directly.

After you implement the backend, package it into a runtime image with
[Runtime Containers](/dynamo/dev/advanced-customizations/writing-custom-backends/runtime-containers). For Kubernetes deployment, place the
custom backend in a `DynamoGraphDeployment` and follow the
[Model Deployment](/dynamo/dev/kubernetes/model-deployment/introduction).

<Tabs>
<Tab title="Python" language="python">

## Python Implementation

<Note>The unified backend API is experimental and may change between releases.</Note>

This guide covers the unified backend infrastructure in
[`dynamo.common.backend`](https://github.com/ai-dynamo/dynamo/tree/main/components/src/dynamo/common/backend):
a shared `LLMEngine` abstract base class (ABC), a runtime-owned `Worker`, and a
sample implementation. For the Rust version of the same contract, use the Rust
tab. To own endpoint registration and request handling directly, use
[Writing Python Workers](/dynamo/dev/advanced-customizations/writing-custom-backends/writing-python-workers).

This guide walks through building a Python backend for an inference
engine that plugs into Dynamo's distributed runtime via
`dynamo.common.backend`. A "unified backend" is a Python entry point
that implements the shared `LLMEngine` ABC and lets the framework own
runtime lifecycle (signal handling, model registration, graceful
shutdown, cancellation monitoring) — your code just owns inference.

Your backend lives in its own package and **does not need to be part
of the dynamo repository**. It depends on `ai-dynamo` from PyPI (or
the git source) and imports `dynamo.common.backend`. The steps below
assume you're starting a fresh package in your own repo.

The reference example is the **sample engine** at
[`sample_engine.py`](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/common/backend/sample_engine.py)
— a complete, runnable implementation. Read it alongside this guide.

**Where to look for what:**

- This guide — step-by-step walkthrough for someone starting a new
  backend from scratch.
- [`LLMEngine` ABC docstrings](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/common/backend/engine.py)
  — authoritative method-by-method contract.
- [Package README](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/common/backend/README.md)
  — in-tree reference: `GenerateRequest` / `GenerateChunk` field
  definitions, cancellation contract, full `DynamoException` table,
  and file index.

### Python: What you are building

A backend is two things:

1. **An engine class** that subclasses `LLMEngine` — owns the model,
   accepts preprocessed token requests, streams output chunks.
2. **A `main.py` entry point** — a three-line shim that hands the
   engine class to `run()` from `dynamo.common.backend.run`, which
   drives the lifecycle.

The `dynamo.common.backend` package handles everything else: signal
handling, distributed runtime setup, model registration with
discovery, the serving loop, graceful shutdown, cancellation
monitoring, and error chain wrapping. (The lifecycle state machine
actually lives in Rust; `dynamo.common.backend.Worker` is a thin
Python shim over it.)

```text
from_args  →  start()  →  generate() / abort()  →  drain()  →  cleanup()
   |            |               |                     |           |
parse argv,  start engine,  serve requests       pre-cleanup    release
return       return            (concurrent)       drain          resources
engine       metadata
```

### Python prerequisites

- Python 3.11 or newer. `dynamo` uses `typing.Required`, which is 3.11+.
- NATS and etcd reachable for end-to-end runs. The dynamo repo's
  `deploy/docker-compose.yml` brings up both in one command if you
  don't already have them running.
- `uv` or `pip` for installing dependencies.
- Familiarity with `async` Python (`asyncio`, async generators) and
  `argparse`.

### Python Step 1: Create the package

```text
my-backend/
├── pyproject.toml
└── src/
    └── my_backend/
        ├── __init__.py
        ├── engine.py
        └── main.py
```

Minimal `pyproject.toml`:

```toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "my-backend"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
    # ai-dynamo bundles dynamo.common.backend. Pin to the release whose
    # LLMEngine contract you tested against — the surface is still beta
    # and may change between releases.
    "ai-dynamo>=1.2.0",
]

[project.optional-dependencies]
dev = ["pytest>=8", "pytest-asyncio>=0.23"]

[project.scripts]
my-backend = "my_backend.main:main"
```

For a bleeding-edge dependency on the dynamo source tree, install
the runtime wheel from a clone:

```bash
git clone https://github.com/ai-dynamo/dynamo.git
pip install 'maturin[patchelf]'
cd dynamo/lib/bindings/python && maturin build --release --out /tmp/wheels
pip install /tmp/wheels/*.whl       # ai-dynamo-runtime
pip install /path/to/dynamo         # ai-dynamo (components/ tree)
```

[Maturin](https://github.com/PyO3/maturin) is the Rust-Python bindings build tool. The `patchelf` extra lets maturin patch native extension library paths during the build.

Building the wheel needs a Rust toolchain plus `clang`, `cmake`,
`protobuf-compiler`, and `libssl-dev`.

### Python Step 2: Subclass `LLMEngine`

In `src/my_backend/engine.py`, declare a class that subclasses
`LLMEngine` and owns whatever state your engine needs. Construction
must be cheap and side-effect-free — heavy work goes in `start()`.

```python
# src/my_backend/engine.py
from __future__ import annotations

import argparse
import asyncio
from collections.abc import AsyncGenerator

from dynamo._core import Context
from dynamo.common.backend import (
    EngineConfig,
    GenerateChunk,
    GenerateRequest,
    LLMEngine,
    WorkerConfig,
)


class MyBackend(LLMEngine):
    def __init__(self, model_name: str, max_tokens: int = 16):
        self.model_name = model_name
        self.max_tokens = max_tokens
        # Heavy state (engine handles, schedulers, KV allocators) is
        # left None here and initialized in start().
        self._engine = None
```

`GenerateRequest` and `GenerateChunk` are `TypedDict`s describing the
shared shape — see Step 4 for the fields.

### Python Step 3: Implement `from_args`

`from_args` is a classmethod factory that parses CLI args and returns
`(engine, WorkerConfig)`. The engine is constructed but **not
started**.

```python
@classmethod
async def from_args(
    cls, argv: list[str] | None = None
) -> tuple[MyBackend, WorkerConfig]:
    parser = argparse.ArgumentParser(prog="my-backend")
    parser.add_argument("--model-name", default="my-model")
    parser.add_argument("--max-tokens", type=int, default=16)
    # Runtime / discovery flags — every unified backend needs these.
    parser.add_argument("--namespace", default="dynamo")
    parser.add_argument("--component", default="backend")
    parser.add_argument("--endpoint", default="generate")
    parser.add_argument("--endpoint-types", default="chat,completions")
    parser.add_argument("--discovery-backend", default="etcd")
    parser.add_argument("--request-plane", default="tcp")
    parser.add_argument("--event-plane", default=None)
    args = parser.parse_args(argv)

    engine = cls(model_name=args.model_name, max_tokens=args.max_tokens)
    worker_config = WorkerConfig(
        namespace=args.namespace,
        component=args.component,
        endpoint=args.endpoint,
        model_name=args.model_name,
        served_model_name=args.model_name,
        endpoint_types=args.endpoint_types,
        discovery_backend=args.discovery_backend,
        request_plane=args.request_plane,
        event_plane=args.event_plane,
    )
    return engine, worker_config
```

`from_args` is `async` to match the ABC; you can `await` from it if
your CLI parsing reads config from a file or hits an API. Most
backends don't need to.

For backends that already have a `DynamoRuntimeConfig`-shaped
config object (e.g. one derived from an existing engine's own
config), prefer the
`WorkerConfig.from_runtime_config(runtime_cfg, model_name=...)`
helper — it pulls the shared discovery / request-plane / parser
fields off the config in one line.

### Python Step 4: Implement `LLMEngine` methods

The ABC has three required methods (`start`, `generate`, `cleanup`)
plus two with default no-op implementations (`abort`, `drain`).

#### Python: `start()`

Start the engine and return `EngineConfig` metadata. After this
returns, `generate()` MUST be ready for concurrent calls.

```python
async def start(self, worker_id: int) -> EngineConfig:
    # ... load weights, build scheduler, warm up CUDA, etc.
    # Heavy: may take minutes. Emit logger.info checkpoints so
    # operators see progress (Worker logs around start() but not
    # inside it).
    self._engine = await heavy_init(self.model_name)

    return EngineConfig(
        model=self.model_name,
        served_model_name=self.model_name,
        context_length=8192,
        kv_cache_block_size=16,    # None if no block-structured KV
        total_kv_blocks=1024,
        max_num_seqs=64,
        max_num_batched_tokens=8192,
    )
```

`worker_id` is an opaque per-worker identifier — most engines ignore
it. Backends needing a stable cluster-wide key (e.g. a
disaggregation machine ID) should derive from it instead of
hashing host/pid or asking operators for a CLI override.

Every `EngineConfig` field except `model` is optional. `None` means
"don't advertise"; KV-aware routing falls back to round-robin when KV
fields are unset.

#### Python: `generate()`

An async generator that yields `GenerateChunk` dicts for a single
request. Called concurrently for multiple in-flight requests.

**Contract** (chunk shape is defined by the `GenerateChunk` TypedDict
— see
[Request / Response Types](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/common/backend/README.md#request--response-types)
in the package README for the field reference):

- Every chunk carries `token_ids` and `index` (use `0` for single
  choice).
- The final chunk additionally carries `finish_reason` and
  `completion_usage`.
- The framework's cancellation monitor calls `engine.abort(context)`
  when the client disconnects or cancels; your loop should also poll
  `context.is_stopped()` between yields and exit cleanly with a
  `finish_reason="cancelled"` chunk.

```python
async def generate(
    self, request: GenerateRequest, context: Context
) -> AsyncGenerator[GenerateChunk, None]:
    prompt_tokens = list(request.get("token_ids", []))
    prompt_len = len(prompt_tokens)

    stop_conditions = request.get("stop_conditions") or {}
    max_new = stop_conditions.get("max_tokens") or self.max_tokens

    def _usage(completion_tokens: int) -> dict[str, int]:
        return {
            "prompt_tokens": prompt_len,
            "completion_tokens": completion_tokens,
            "total_tokens": prompt_len + completion_tokens,
        }

    for i in range(max_new):
        if context.is_stopped():
            yield {
                "token_ids": [],
                "index": 0,
                "finish_reason": "cancelled",
                "completion_usage": _usage(i),
            }
            return

        token_id = await self._next_token(prompt_tokens)

        chunk: GenerateChunk = {"token_ids": [token_id], "index": 0}
        if i == max_new - 1:
            chunk["finish_reason"] = "length"
            chunk["completion_usage"] = _usage(max_new)
        yield chunk
```

Finish reason normalization (`"abort"` → `"cancelled"`, etc.) is
handled by the Rust layer — emit whatever your engine uses
natively.

#### Python: Custom logits processors (optional)

Override `logits_processor_spec()` to expose backend-neutral activation data.
Resolve and cache the specification during startup, call
`logits_processors_for_request()` for each request, and construct fresh
inference-library processor instances from the returned entries. The worker
does not realize these entries automatically.

Import `LogitsProcessorSpec`, `ForcedTokenSequenceSpec`,
`PythonProcessorSpec`, and the serialization helpers from
`dynamo.common.backend.engine`. A generation-only specification runs on
aggregated and decode workers but not prefill or encode workers.
`ForcedTokenSequenceSpec` can cross a JSON request boundary;
`PythonProcessorSpec` contains a live factory and cannot be serialized.

#### Python: Engine management (optional)

Advertise lifecycle operations with `supported_controls()` and implement each
operation in `engine_control(name, body)`. The worker registers advertised
names at `POST /engine/control/<name>`.

Use the separate `supported_updates()` and `engine_update(name, body)` methods
for operations that mutate engine-managed assets. These register at
`POST /engine/update/<name>`. Both capability sets are empty by default, and
request and response bodies are JSON objects.

Override `on_endpoint_ready(endpoint)` when a later management operation needs
the serving endpoint. The worker invokes this hook once after the endpoint
exists and before discovery registration or request serving.

#### Python: `abort(context)` — optional

Called by the framework only when the client disconnects or the
request is cancelled. NOT called on silent stream drops. Override to
release engine-side resources (KV slots, scheduler entries, remote
schedulers):

```python
async def abort(self, context: Context) -> None:
    request_id = context.id()
    await self._engine.cancel(request_id)
```

For cleanup that must run on every drop path — including silent
drops — use a `try/finally` or a context manager inside `generate`,
not `abort`. The sample engine doesn't override `abort` because it
has no engine-side state to release; the default is a no-op.

#### Python: `drain()` — optional

Runs once before shutdown, after the discovery unregister +
grace-period sleep, while NATS/etcd are still alive. Use it for
backend-side draining that must complete before transport teardown
(e.g. in-flight NIXL KV transfers on prefill workers). Default is
no-op.

#### Python: `cleanup()`

Two real requirements, both pinned by the Rust-side conformance kit:

- **Null-safe against partial `start()` failure.** If `start()` raises
  partway through, fields you allocate incrementally may still be
  `None`. `cleanup()` must guard each resource (`if self._engine is
  not None: …`) so the post-failure call doesn't crash on
  half-initialized state.
- **Idempotent.** A second call after a successful first must return
  cleanly without re-entering teardown.

The Rust `Worker` drives both: it calls `cleanup()` after `start()`
returns Ok on shutdown, and the conformance kit (`run_conformance`)
additionally calls `cleanup()` on a never-started engine and twice in a
row, failing your tests with `CleanupWithoutStartFailed` /
`SecondCleanupFailed` if either invariant breaks. The guarded
single-shot pattern below covers both:

```python
async def cleanup(self) -> None:
    if self._engine is not None:
        await self._engine.shutdown()
        self._engine = None
```

#### Python: Metrics and Prometheus (optional)

Unified backends have two metrics surfaces.

Use `register_prometheus(metrics)` to bridge vendor-prefixed Prometheus
families into the worker's `/metrics` output. The framework owns the
`metrics` handle; do not retain it after the method returns.

```python
from dynamo.common.backend.metrics import register_global_registry

async def register_prometheus(self, metrics):
    register_global_registry(metrics, engine_prefix="myengine:")
```

Use `component_metrics_dp_ranks()` plus
`attach_snapshot_publisher(publisher)` when the engine can push per-rank
`ComponentSnapshot` values for `dynamo_component_*` gauges and the
router's `kv_used_blocks` signal:

```python
from dynamo.common.backend.publisher import ComponentSnapshot

def component_metrics_dp_ranks(self):
    return [0]

def attach_snapshot_publisher(self, publisher):
    self._snapshot_publisher = publisher

def _on_stats(self, used_blocks, total_blocks):
    self._snapshot_publisher.publish(
        0,
        ComponentSnapshot(
            kv_used_blocks=used_blocks,
            kv_total_blocks=total_blocks,
            gpu_cache_usage=used_blocks / total_blocks if total_blocks else 0.0,
            dp_rank=0,
        ),
    )
```

Keep the rank list stable for the engine lifetime. `Worker` invokes
`attach_snapshot_publisher()` only when the rank list is non-empty and
`WorkerConfig.enable_kv_routing` is enabled. `register_prometheus()` still
runs when `enable_kv_routing=False`.

A typical engine pushes per-rank snapshots from its own stats callback
(a stat logger, scheduler hook, or dedicated poll thread) and bridges
its vendor-prefixed metric families through `register_prometheus()`.

#### Python: KV event publishing (optional)

Unified backends declare KV event sources; the framework constructs and owns
the `KvEventPublisher` instances. Do not instantiate `KvEventPublisher`
directly from a unified `LLMEngine`. Instead, implement
`kv_event_sources()` and return one source for each data-parallel rank hosted
by the worker.

Rust backends use the equivalent `LLMEngine::kv_event_sources()` trait method;
see [Rust Step 4](#rust-step-4-implement-the-llmengine-trait) and the
[`LLMEngine` trait](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/src/engine.rs).

Use `ZmqSource` when the engine already emits Dynamo-compatible KV events on a
ZMQ socket:

```python
from dynamo.common.backend.publisher import ZmqSource

async def kv_event_sources(self):
    return [
        ZmqSource(endpoint="tcp://127.0.0.1:5557", dp_rank=0),
        ZmqSource(endpoint="tcp://127.0.0.1:5558", dp_rank=1),
    ]
```

Use `PushSource` when the engine needs a live publisher object and drives
`publish_stored()` / `publish_removed()` from its own event thread:

```python
from dynamo.common.backend.publisher import PushSource

def _on_kv_publisher_ready(self, publisher):
    self._kv_publisher = publisher
    self._start_kv_event_thread()

async def kv_event_sources(self):
    return [PushSource(on_ready=self._on_kv_publisher_ready, dp_rank=0)]
```

KV event publishers require `EngineConfig.llm.kv_cache_block_size`. If the
engine declares sources but does not return a block size, `Worker` skips KV
event publishers because the router cannot map token IDs to cache blocks.
`WorkerConfig.enable_kv_routing=False` is the operator-level kill switch; when
it is disabled, the worker does not call `kv_event_sources()`.

Keep rank ownership stable for the engine lifetime. DP-capable engines should
also advertise the same rank shape in `EngineConfig.llm.data_parallel_size` and
`data_parallel_start_rank` so router-forced `dp_rank` values line up with the
published event streams.

`PushSource` engines own cleanup of their event producer. Stop publisher
threads or tasks in `cleanup()` before returning, and do not publish after
cleanup begins.

### Python Step 5: Write `main.py`

Three lines.

```python
# src/my_backend/main.py
from dynamo.common.backend.run import run
from .engine import MyBackend


def main() -> None:
    run(MyBackend)


if __name__ == "__main__":
    main()
```

`run` installs signal handlers, builds the distributed runtime,
calls `engine.start(worker_id)` with a runtime-allocated identifier,
registers the model with discovery, serves the endpoint, and runs the
graceful-shutdown orchestrator on SIGTERM/SIGINT.

Pair this with the `[project.scripts]` entry from Step 1's
`pyproject.toml` so `my-backend ...` works as a console command.

### Python Step 6: Errors and logging

**Errors**: the framework wraps non-`DynamoException` errors raised
from `generate()` (or lifecycle methods) as `Unknown`. For typed
error reporting, raise a `DynamoException` subclass directly from
[`dynamo.llm.exceptions`](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/common/backend/README.md#error-handling)
— it propagates unchanged through the Rust bridge:

```python
from dynamo.llm.exceptions import InvalidArgument

async def generate(self, request, context):
    if not request.get("token_ids"):
        raise InvalidArgument("empty prompt")
    ...
```

The package README has the full table of exception types and which
lifecycle phase raises which one. Engine-init failures should raise
`EngineShutdown` from `start()`. Cleanup shouldn't normally raise —
log and swallow if a subsystem fails.

**Logging**: keep levels consistent across unified backends so
operators see the same surface regardless of which engine they're
running:

- `logger.info` — lifecycle milestones (engine init complete,
  serving started, engine shutdown).
- `logger.debug` — per-request events (request abort, cancellation).
- `logger.warning` — recoverable problems (empty outputs, unexpected
  finish reasons).
- `logger.error` — unrecoverable failures only.

The framework also configures `dynamo.runtime.logging` for you; you
just call `logger = logging.getLogger(__name__)` at the top of your
module and use it.

### Python Step 7: Test your engine

Install the dev extras (`pytest`, `pytest-asyncio`) declared in Step 1:

```bash
pip install -e ".[dev]"
```

The sample engine has a unit-test
[suite](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/common/backend/tests/test_engine.py)
that you can copy as a starting point. The shape of a useful test:

```python
import pytest

from my_backend import MyBackend


class _StubContext:
    def __init__(self, stopped: bool = False) -> None:
        self._stopped = stopped

    def is_stopped(self) -> bool:
        return self._stopped

    def stop(self) -> None:
        self._stopped = True


@pytest.mark.asyncio
async def test_generate_emits_terminal_chunk():
    engine = MyBackend(model_name="m", max_tokens=3)
    await engine.start(worker_id=0)
    try:
        chunks = [
            chunk
            async for chunk in engine.generate(
                {"token_ids": [1, 2, 3]}, _StubContext()
            )
        ]
        assert chunks[-1]["finish_reason"] in ("stop", "length")
        assert chunks[-1]["completion_usage"]["completion_tokens"] == 3
    finally:
        await engine.cleanup()


@pytest.mark.asyncio
async def test_generate_observes_cancellation():
    engine = MyBackend(model_name="m", max_tokens=1000)
    await engine.start(worker_id=0)
    try:
        ctx = _StubContext()
        collected = []
        async for chunk in engine.generate({"token_ids": [1]}, ctx):
            collected.append(chunk)
            if len(collected) >= 2:
                ctx.stop()
        assert collected[-1]["finish_reason"] == "cancelled"
    finally:
        await engine.cleanup()
```

Cover the happy path, cancellation, and any backend-specific edge
cases (stop tokens, max-tokens cap, empty prompt). Three to five
focused tests is plenty — the framework already pins the lifecycle
state machine and cancellation contract with Rust-side tests in
`lib/backend-common`.

### Python Step 8: Run it locally

Three moving parts need to come up: NATS + etcd (discovery and the
event/request planes), the Dynamo frontend (HTTP → backend
discovery), and your backend.

```bash
pip install -e .

# Ensure NATS + etcd are reachable (NATS_SERVER, ETCD_ENDPOINTS).
# --model-name must be a valid HuggingFace repo (or local path); the
# framework fetches the tokenizer + chat template from it on startup.
# Pick a small public repo for smoke tests.
my-backend --model-name Qwen/Qwen3-0.6B --namespace dynamo

# In another shell, start the Dynamo frontend:
python -m dynamo.frontend --http-port 8000
```

Then send a request:

```bash
curl http://localhost:8000/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{
          "model": "Qwen/Qwen3-0.6B",
          "messages": [{"role": "user", "content": "hello"}],
          "max_tokens": 32
        }'
```

A successful response has non-empty `choices[0].message.content`
and a `finish_reason` of `stop` or `length`.
`jq -e '.choices[0].finish_reason'` is a good one-liner for a CI
smoke test.

If your backend looks silent, set `DYN_LOG=info` (or
`DYN_LOG=debug,dynamo=debug` for finer scoping) before launching —
the framework configures `tracing` from `DYN_LOG`.

### Python reference: sample engine

[`sample_engine.py`](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/common/backend/sample_engine.py)
is the canonical minimal reference. Run it as-is:

```bash
python -m dynamo.common.backend.sample_main --model-name test-model
```

It generates rotating token IDs with no ML dependencies, so it's a
useful stand-in for AIPerf / end-to-end pipeline smoke tests. Lift
these patterns:

- `from_args` parses CLI args and returns `(engine, WorkerConfig)`
  with no awaits.
- `start()` returns an `EngineConfig` whose KV fields are
  illustrative but not load-bearing (no real KV cache).
- `generate()` polls `context.is_stopped()` between yields and
  emits a `cancelled` terminal on observation.
- `cleanup()` is a no-op because the engine holds no resources.

### Python checklist

Before shipping:

- [ ] `LLMEngine` subclassed; `from_args` returns
      `(engine, WorkerConfig)`.
- [ ] `start()` returns `EngineConfig` with at least a non-empty
      `model`.
- [ ] `generate()` polls `context.is_stopped()` between yields and
      emits a `"cancelled"` terminal on observation.
- [ ] Final chunk has `finish_reason` and `completion_usage`.
- [ ] Typed `DynamoException` subclasses used for error reporting
      where the category matters.
- [ ] `cleanup()` releases all engine resources.
- [ ] Logging levels match the standards in Step 6.

### Python see also

- [`LLMEngine` ABC](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/common/backend/engine.py)
  — authoritative contract.
- [Package README](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/common/backend/README.md)
  — lifecycle, error model, and request/response contract.
- [Sample engine](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/common/backend/sample_engine.py)
  — example user guide.
- Rust tab on this page — the Rust counterpart, same contract,
  lower-level.

</Tab>
<Tab title="Rust" language="rust">

## Rust Implementation

<Note>The unified backend API is experimental and may change between releases.</Note>

This guide covers the unified backend infrastructure in
[`dynamo-backend-common`](https://github.com/ai-dynamo/dynamo/tree/main/lib/backend-common):
a shared `LLMEngine` contract, a runtime-owned `Worker`, and a mocker
implementation.

This guide walks through building a Rust unified backend for an
inference engine that plugs into Dynamo's distributed runtime. A
unified backend is a standalone Rust binary that owns its engine and
serves requests via the shared
[`LLMEngine`](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/src/engine.rs) contract in
[`dynamo-backend-common`](https://github.com/ai-dynamo/dynamo/tree/main/lib/backend-common) — no Python
worker runtime required. For the Python version of the same contract
use the Python tab on this page.

Your backend lives in its own crate and **does not need to be part of
the dynamo repository**. It pulls `dynamo-backend-common` in as a normal
git or path dependency. The steps below assume you're starting a fresh
crate in your own repo; an optional note in Step 1 covers the in-tree
variant for contributors landing a backend inside `ai-dynamo/dynamo`.

For a Python engine, use the Python tab on this page. To own endpoint
registration and request handling directly, use
[Writing Python Workers](/dynamo/dev/advanced-customizations/writing-custom-backends/writing-python-workers).

The reference example is the **mocker backend** at
[`lib/backend-common/examples/mocker`](https://github.com/ai-dynamo/dynamo/tree/main/lib/backend-common/examples/mocker)
— a small, complete, pure-Rust implementation. Read it alongside this
guide.

**Where to look for what:**

- This guide — step-by-step walkthrough for someone starting a new
  backend from scratch.
- [`LLMEngine` trait doc comments](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/src/engine.rs)
  — authoritative method-by-method contract.
- [Crate README](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/README.md) — in-tree
  reference: architecture, file index, disaggregation contract,
  error taxonomy, conformance kit.
- [`backend-common` design notes](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/CLAUDE.md)
  — rationale and invariants.

### Rust: What you are building

A backend is two things:

1. **An engine type** that implements the `LLMEngine` trait — owns the
   model, accepts preprocessed token requests, streams output tokens.
2. **A `main.rs` entry point** — a three-line shim that hands the
   engine to `dynamo_backend_common::run`, which drives the lifecycle.

The `dynamo-backend-common` crate handles everything else: signal
handling, model registration with discovery, the serving loop, graceful
shutdown, metrics, cancellation plumbing, and the debug-mode contract
validator.

Engines work directly with `PreprocessedRequest` and `LLMEngineOutput`
— the same types used by Dynamo's preprocessing, routing, and frontend.
No Python-shaped translation layer.

```text
construct  →  start()  →  generate() / abort()  →  drain() →  cleanup()
   |            |               |                      |          |
parse args   start engine,  serve requests       pre-cleanup   release
return       return            (concurrent)       drain        resources
engine       metadata
```

### Rust prerequisites

- Rust 1.85 or newer (the dynamo workspace is edition 2024). The
  toolchain pin in Step 1 locks this in for you; older toolchains will
  fail with `feature edition2024 is required` deep inside the build.
- NATS and etcd reachable for end-to-end runs. The dynamo repo's
  `deploy/docker-compose.yml` brings up both in one command if you
  don't already have them running.
- Familiarity with `async` Rust, `tokio`, and `clap`. The trait uses
  `async_trait`, and the framework expects a `tokio` runtime.

### Rust Step 1: Create the crate

Your backend is a standalone Rust binary crate. It can live in its own
repository — the dynamo repo is **not** required to be your parent
workspace. Pick whatever layout you prefer:

```text
my-backend/
├── Cargo.toml
└── src/
    ├── main.rs
    └── engine.rs        # (or my_engine.rs — whatever you call it)
```

`cargo new --bin my-backend` is the fastest starting point; add
`src/engine.rs` yourself afterwards.

#### Rust: Getting the `dynamo-backend-common` crate

`dynamo-backend-common` lives in the
[ai-dynamo/dynamo](https://github.com/ai-dynamo/dynamo) repository and
is not on crates.io. Depend on it via git:

```toml
[package]
name = "my-backend"
version = "0.1.0"
edition = "2024"

[[bin]]
name = "my-backend"
path = "src/main.rs"

[dependencies]
# Replace <SHA> with the dynamo commit you want to build against.
# `branch = "main"` works too but moves under you on every rebuild.
dynamo-backend-common = { git = "https://github.com/ai-dynamo/dynamo.git", rev = "<SHA>" }

anyhow = "1"
async-stream = "0.3"
async-trait = "0.1"
clap = { version = "4", features = ["derive", "env"] }
futures = "0.3"
# Must match the version pinned by dynamo-runtime — it relies on
# tokio_unstable runtime metrics that change shape across releases.
tokio = { version = "=1.48.0", features = ["full"] }
tracing = "0.1"

[dev-dependencies]
dynamo-backend-common = { git = "https://github.com/ai-dynamo/dynamo.git", rev = "<SHA>", features = ["testing"] }
```

The `testing` feature pulls in the conformance kit used in Step 7.

Pick a SHA with:

```bash
git ls-remote https://github.com/ai-dynamo/dynamo.git main
```

> **No release tags yet.** `dynamo-backend-common` landed after the last
> tagged release (`v1.1.1`), so `tag = "v1.1.1"` won't resolve the
> crate. Track `main` or pin to a specific SHA until a release tag ships
> that includes the crate.

#### Rust: Two build-time requirements you cannot skip

These are easy to miss and surface as confusing compile errors deep
inside `dynamo-runtime`:

1. **`tokio_unstable` cfg flag.** `dynamo-runtime` uses tokio's
   unstable runtime-metrics API. Create `.cargo/config.toml` in your
   crate root:

   ```toml
   [build]
   rustflags = ["--cfg", "tokio_unstable"]
   ```

   Without it, you'll see errors like `method blocking_queue_depth not
   found on RuntimeMetrics` while compiling `dynamo-runtime`.

2. **Rust toolchain pin.** Match dynamo's toolchain so workspace-edition
   crates compile. Create `rust-toolchain.toml`:

   ```toml
   [toolchain]
   channel = "1.96.1"
   ```

   Older toolchains fail with `feature edition2024 is required`.

> **Tip — local development**: while iterating against an unreleased
> change in `dynamo-backend-common`, point the dep at a local clone:
> `dynamo-backend-common = { path = "/path/to/dynamo/lib/backend-common" }`.
> Switch back to the git dep before publishing your crate.

If you'd rather develop *inside* the dynamo workspace as a new
sub-crate, drop your crate under `dynamo/lib/` and use
`dynamo-backend-common = { workspace = true }` instead. The trait
contract is identical, and the `.cargo/config.toml` plus toolchain
pin in the dynamo repo cover the two requirements above for you.

### Rust Step 2: Define your engine struct

In `src/engine.rs` (or whatever you named it), declare a struct that
owns whatever state your engine needs. Anything you allocate inside
`start()` later must live behind interior mutability so the trait's
`&self` methods can reach it.

```rust
use async_trait::async_trait;
use dynamo_backend_common::engine::GenerateContext;
use dynamo_backend_common::{
    BackendError, CommonArgs, DynamoError, EngineConfig, ErrorType, FinishReason, LLMEngine,
    LLMEngineOutput, LLMEngineOutputExt, LlmRegistration, PreprocessedRequest, WorkerConfig,
    chunk, usage,
};
use futures::stream::BoxStream;
use tokio::sync::RwLock;

pub struct MyBackend {
    model: String,
    inner: RwLock<Option<Inner>>,   // allocated in start()
}

// Replace this with whatever your engine owns — handle, scheduler,
// client, channel sender, etc. Fields go here. Truly stateless
// engines can skip `Inner` and `inner` entirely.
struct Inner {}
```

`async-trait` lets the trait use `async fn` (still required for
object-safety with `Arc<dyn LLMEngine>`); `async-stream`'s `stream!`
macro lets the `generate` body yield items from inside an `async` block.

The mocker example uses `OnceCell` for `Inner`; `RwLock<Option<_>>`
also works — pick whichever fits your shutdown semantics.

### Rust Step 3: Wire up CLI arguments

Every backend's CLI shares a common base (`--namespace`, `--component`,
`--endpoint`, etc.) provided by `CommonArgs`. Flatten that into your
engine's `Args` struct and add your engine-specific flags.

```rust
#[derive(clap::Parser, Debug)]
#[command(
    name = env!("CARGO_BIN_NAME"),
    about = "My Dynamo Rust backend."
)]
struct Args {
    #[command(flatten)]
    common: CommonArgs,

    /// HF repo or local model directory.
    #[arg(value_name = "MODEL")]
    model: String,

    /// Public-facing model name advertised to clients.
    #[arg(long)]
    served_model_name: Option<String>,

    // ... engine-specific flags here.
}
```

Define an inherent `from_args` constructor that parses the args and
returns both the engine and a `WorkerConfig`. **`from_args` is not on
the trait** — it stays inherent so the trait can remain object-safe
(`Arc<dyn LLMEngine>` must work).

The snippet below calls a tiny `invalid_arg` helper that builds a
typed `BackendError::InvalidArgument`. Its full definition lives in
Step 6 — for now, mentally substitute "any function that returns a
`DynamoError` with category `InvalidArgument`."

```rust
impl MyBackend {
    pub fn from_args(argv: Option<Vec<String>>) -> Result<(Self, WorkerConfig), DynamoError> {
        let args = match argv {
            Some(a) => <Args as clap::Parser>::try_parse_from(a),
            None => <Args as clap::Parser>::try_parse(),
        }
        .map_err(|e| invalid_arg(e.to_string()))?;

        let engine = Self {
            model: args.model.clone(),
            inner: RwLock::new(None),
        };

        let config = WorkerConfig {
            namespace: args.common.namespace,
            component: args.common.component,
            endpoint: args.common.endpoint,
            endpoint_types: args.common.endpoint_types,
            custom_jinja_template: args.common.custom_jinja_template,
            // Pass `--disaggregation-mode` from `CommonArgs` through to the
            // Worker — without this line the worker silently registers as
            // Aggregated regardless of what the operator passed.
            disaggregation_mode: args.common.disaggregation_mode,
            model_name: args.model,
            served_model_name: args.served_model_name,
            ..Default::default()
        };

        Ok((engine, config))
    }
}
```

`WorkerConfig::default()` sets `model_input` to `ModelInput::Tokens`,
which is the input mode supported by the `LLMEngine` trait. Use
`RawEngine` and `Worker::new_raw` for engines that accept raw text or
tensor inputs.

If your engine branches on the disaggregation role inside `generate`
(prefill vs decode), keep the same `DisaggregationMode` on your engine
struct so the runtime registration (`WorkerConfig`) and the per-request
dispatch stay in lockstep.

### Rust Step 4: Implement the `LLMEngine` trait

The trait has three required methods (`start`, `generate`, `cleanup`)
plus two with default implementations you can override (`abort`, `drain`).

#### Rust: `start()`

Start the engine and return `EngineConfig` metadata. After this
returns, the engine MUST be ready for concurrent `generate()` calls.
Use interior mutability for anything you initialize here.

```rust
async fn start(&self, _worker_id: u64) -> Result<EngineConfig, DynamoError> {
    tracing::info!(model = %self.model, "starting my backend");

    // ... start your engine (may take minutes for real backends — emit
    // tracing::info! checkpoints so operators see progress).
    let inner = init_engine(&self.model).await?;
    *self.inner.write().await = Some(inner);

    Ok(EngineConfig {
        model: self.model.clone(),
        served_model_name: Some(self.model.clone()),
        // Token-pipeline metadata goes in the `llm` sub-record (RawEngines
        // leave it None).
        llm: Some(LlmRegistration {
            context_length: Some(8192),
            kv_cache_block_size: Some(64),     // None if no block-structured KV
            total_kv_blocks: Some(16384),
            max_num_seqs: Some(256),
            max_num_batched_tokens: Some(8192),
            ..Default::default()
        }),
        ..Default::default()
    })
}
```

`worker_id` is an opaque per-worker identifier — most engines ignore
it with `_worker_id`. Backends needing a stable cluster-wide key
(e.g. a disaggregation machine ID) should derive from it.

Every `EngineConfig` field except `model` is optional. `None` means
"don't advertise"; KV-aware routing falls back to round-robin when KV
fields are unset. Engines wrapping an external runtime can read these
values from the live engine after it comes up, instead of hard-coding
them. The `..Default::default()` is load-bearing: `EngineConfig`
sometimes grows new fields (e.g. `bootstrap_host`/`bootstrap_port`
for disaggregation) and the default keeps existing engines compiling.

#### Rust: `generate()`

Yield a stream of `Result<LLMEngineOutput, DynamoError>` items for a
single request. Called concurrently for multiple in-flight requests.

`ctx: GenerateContext` is a thin wrapper that `Deref`s to
`dyn AsyncEngineContext`, so the cancellation methods (`stopped()`,
`is_stopped()`, `id()`) you'd expect are still there. The wrapper
additionally exposes `notify_first_token()` for decode-mode requests
— most engines can ignore it; the framework auto-fires on the first
non-empty chunk.

**Contract** (the [debug-mode validator](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/src/validate.rs)
panics on violations):

- Exactly one **terminal item** must be the last item yielded. A
  terminal is either an `Ok(chunk)` with `finish_reason` set, or an
  `Err(DynamoError)`. No items may be yielded after a terminal.
- Non-terminal chunks use `chunk::token(id)` and leave `finish_reason`
  unset.
- The returned stream is `'static`: clone or move any state from
  `&self` or `request` into the stream body before constructing it.

Terminal chunks come from one of four `LLMEngineOutput` constructors,
optionally chained with the `LLMEngineOutputExt` setters
(`.with_tokens(...)`, `.with_usage(...)`):

- `LLMEngineOutput::stop()` — natural completion (e.g. you reached your
  echo limit, the engine hit a stop string).
- `LLMEngineOutput::length()` — `max_tokens` cap reached.
- `LLMEngineOutput::cancelled()` — you observed `ctx.stopped()`.
- `LLMEngineOutput::error(msg)` — message-only error terminal (loses
  the typed `BackendError` variant — yield `Err(DynamoError)` instead
  when the category matters).

Non-terminal chunks use `chunk::token(id)` (single-token convenience).

A streaming-`generate` template:

```rust
async fn generate(
    &self,
    request: PreprocessedRequest,
    ctx: GenerateContext,
) -> Result<BoxStream<'static, Result<LLMEngineOutput, DynamoError>>, DynamoError> {
    // Destructure once and move fields into the stream — no extra clones
    // (the stream is 'static and outlives `request`).
    let PreprocessedRequest { token_ids, stop_conditions, .. } = request;
    let prompt_tokens = token_ids.len() as u32;
    let mut output_rx = self.submit_to_engine(token_ids, stop_conditions).await?;

    Ok(Box::pin(async_stream::stream! {
        let mut completion_tokens = 0_u32;
        loop {
            tokio::select! {
                biased;

                // Cancellation: emit FinishReason::Cancelled terminal.
                _ = ctx.stopped() => {
                    yield Ok(LLMEngineOutput::cancelled()
                        .with_usage(usage(prompt_tokens, completion_tokens)));
                    break;
                }

                // Next item from the engine.
                next = output_rx.recv() => {
                    let Some(engine_output) = next else {
                        yield Ok(LLMEngineOutput::error(
                            "engine stream ended without a terminal".into()
                        ));
                        break;
                    };

                    // Translate your engine's per-step output into LLMEngineOutput.
                    // For a terminal step set `finish_reason`; otherwise leave it None.
                    let mut out = LLMEngineOutput {
                        token_ids: engine_output.tokens,
                        finish_reason: engine_output.terminal_reason,
                        ..Default::default()
                    };
                    completion_tokens += out.token_ids.len() as u32;

                    if out.finish_reason.is_some() {
                        out = out.with_usage(usage(prompt_tokens, completion_tokens));
                        yield Ok(out);
                        break;
                    }
                    yield Ok(out);
                }
            }
        }
    }))
}
```

`biased` is load-bearing for the channel-receiving pattern above:

1. When cancellation and a pending token are both ready, yield the
   cancellation, not one more token.
2. During cleanup the stream sees both `ctx.stopped()` and
   `rx.recv() -> None` simultaneously; `biased` picks the clean
   cancellation path instead of erroring on a closed channel. The
   mocker's [stream body](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/examples/mocker/src/engine.rs)
   spells this out.

If your engine doesn't have a receiver — e.g. you're computing tokens
inline like a deterministic echo backend — the body collapses to a
plain loop that polls cancellation between yields:

```rust
Ok(Box::pin(async_stream::stream! {
    for (i, token_id) in tokens_to_emit.iter().enumerate() {
        tokio::select! {
            biased;
            _ = ctx.stopped() => {
                yield Ok(LLMEngineOutput::cancelled()
                    .with_usage(usage(prompt_tokens, i as u32)));
                return;
            }
            _ = tokio::time::sleep(delay), if !delay.is_zero() => {}
        }
        if i == tokens_to_emit.len() - 1 {
            yield Ok(LLMEngineOutput::stop()
                .with_tokens(vec![*token_id])
                .with_usage(usage(prompt_tokens, (i + 1) as u32)));
        } else {
            yield Ok(chunk::token(*token_id));
        }
    }
}))
```

No channel-close race to worry about; `biased` is still cheap and
recommended for consistency.

**Cancellation rules**:

- The stream **must** poll `ctx.is_stopped()` (or `await ctx.stopped()`)
  between yields.
- On cancellation, emit a terminal with `FinishReason::Cancelled` — not
  `Length` or `Stop`. The conformance kit treats any other terminal
  after cancellation as ignoring the signal.

**Typed errors vs. string errors**:

```rust
// Typed (preferred): preserves BackendError variant end-to-end.
yield Err(DynamoError::builder()
    .error_type(ErrorType::Backend(BackendError::InvalidArgument))
    .message("bad request")
    .build());

// String: convenient, loses the typed variant.
yield Ok(LLMEngineOutput::error("something went wrong".into()));
```

Use typed errors when the failure category matters to the caller. Use
string errors when it doesn't.

#### Rust: `abort()` and per-request cleanup

`abort` is called by the framework **only** when `ctx.stopped()` or
`ctx.killed()` fires — i.e. an explicit client/operator cancel. It is
NOT called when the stream is silently dropped (TCP reset, consumer
timeout without cancellation).

**For cleanup that must run on any drop path** (releasing a scheduler
slot, freeing a request handle), use RAII inside the `generate` stream
body:

```rust
struct RequestGuard { /* ... */ }
impl Drop for RequestGuard {
    fn drop(&mut self) {
        // Always runs when the stream is dropped, however that happens.
    }
}

Ok(Box::pin(async_stream::stream! {
    let _guard = RequestGuard { /* ... */ };
    // ... your stream body
}))
```

The mocker's `ActiveRequestGuard` is the canonical example.

Use `abort` only for out-of-band notifications (e.g. telling a remote
scheduler to stop computing for this request).

#### Rust: `drain()` and `cleanup()`

- `drain()` runs once before shutdown, after the discovery
  unregister + grace-period sleep, while NATS/etcd are still alive.
  Use it for backend-side draining that must complete before the
  transport layer goes away (e.g. in-flight NIXL KV transfers on
  prefill workers). Default is no-op.
- `cleanup()` is called once on shutdown. Release all engine
  resources. The framework guarantees `cleanup()` runs exactly once after
  `start()` is invoked, whether `start()` succeeds or returns an error. It is
  not called when `start()` was never invoked. Release resources allocated
  during construction through `Drop` rather than relying on `cleanup()`.

Make `cleanup()` idempotent and tolerant of being called from a
half-initialized state. An engine that tears down NCCL groups in
`cleanup()` can hang on a second attempt.

### Rust Step 5: Write `main.rs`

Three lines. That's it.

```rust
use std::sync::Arc;

mod engine;

fn main() -> anyhow::Result<()> {
    let (engine, config) = engine::MyBackend::from_args(None)?;
    dynamo_backend_common::run(Arc::new(engine), config)
}
```

`run` installs signal handlers, builds the distributed runtime, calls
`engine.start()`, registers the model with discovery, serves the
endpoint, and runs the full graceful-shutdown orchestrator on
SIGTERM/SIGINT.

### Rust Step 6: Errors and logging

**Errors**: every error returned from `start`, `generate`, `cleanup`,
and `from_args` uses `ErrorType::Backend(BackendError::X)`. From the
frontend's perspective, anything bubbling up through the backend has
"originated from the backend" — engine code vs. framework code is not
distinguished. Top-level `ErrorType::X` variants are reserved for
non-backend paths.

A small helper module per backend keeps the call sites clean:

```rust
pub(crate) fn invalid_arg(msg: impl Into<String>) -> DynamoError {
    DynamoError::builder()
        .error_type(ErrorType::Backend(BackendError::InvalidArgument))
        .message(msg)
        .build()
}
```

Common nested categories: `InvalidArgument`, `CannotConnect`,
`EngineShutdown`, `StreamIncomplete`, `Cancelled`, `ResponseTimeout`,
`Disconnected`, `ConnectionTimeout`, `Unknown`.

**Logging**: keep levels consistent across Rust backends so operators
see the same surface everywhere.

- `tracing::info!` for lifecycle milestones (engine started, cleanup
  complete). `Worker` already logs "Serving {model} on …" and "Engine
  cleanup complete" — add your own only for events those don't cover.
- `tracing::debug!` for per-request events (cancellation, abort).
- `tracing::warn!` for recoverable problems.
- `tracing::error!` only for unrecoverable failures.

### Rust Step 7: Run the conformance kit

Before merging, prove your engine satisfies the contract. The
conformance kit is one call:

```rust
#[tokio::test]
async fn my_engine_passes_conformance() {
    // `run_conformance` takes a factory closure rather than a built
    // engine — the kit constructs a second pristine engine for its
    // "cleanup-without-start" check.
    dynamo_backend_common::testing::run_conformance(|| {
        MyBackend::new(/* your defaults */).expect("construct")
    })
    .await
    .expect("conformance");
}
```

The kit runs `start`/`generate`/`cleanup` directly against your engine
— no external service is involved. If your engine needs a real GPU,
remote model server, or other heavyweight resource to construct, gate
the test with `#[ignore]` and require an explicit opt-in env var.

What it asserts:

| Check | Failure mode |
|---|---|
| `start()` returns a non-empty `EngineConfig.model` | `EmptyModelInConfig` |
| Single `generate()` ends in a terminal chunk | `NoTerminalChunk` |
| No chunks after the terminal | `ChunkAfterTerminal` |
| Interleaved `generate()` calls all succeed | `ConcurrentGenerateFailed` |
| Mid-stream cancel terminates within 2s | `CancellationNotObserved` |
| Cancelled stream's terminal is `FinishReason::Cancelled` | `CancellationIgnored` |
| `cleanup()` succeeds twice (idempotent) | `SecondCleanupFailed` |
| `cleanup()` on a never-started engine succeeds | `CleanupWithoutStartFailed` |

For tests that don't need a real engine, use `testing::mock_context()`
or `testing::cancelling_context(after)` to drive `generate` manually.

### Rust Step 8: Run it locally

Three moving parts need to come up: NATS + etcd (discovery and the
event/request planes), the Dynamo Python frontend (HTTP → backend
discovery), and your backend.

The fastest path is to copy the **mocker example's
[`docker-compose.yml`](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/examples/mocker/docker-compose.yml)
and [`Dockerfile.frontend`](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/examples/mocker/Dockerfile.frontend)**,
swap in your image, and run `docker compose up --build`. That brings
up NATS + etcd + the Python frontend (built from the dynamo workspace
at the same SHA as your backend) + your backend, all on one network.

For a non-Docker dev loop:

```bash
cargo build --release

# Ensure NATS + etcd are reachable (NATS_SERVER, ETCD_ENDPOINTS).
./target/release/my-backend Qwen/Qwen3-0.6B \
    --namespace dynamo \
    --component backend \
    --endpoint generate

# In another shell, start the Python frontend from the dynamo repo:
python -m dynamo.frontend --http-port 8000
```

Then send a request:

```bash
curl http://localhost:8000/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{
          "model": "Qwen/Qwen3-0.6B",
          "messages": [{"role": "user", "content": "hello"}],
          "max_tokens": 32
        }'
```

A successful response has non-empty `choices[0].message.content` and a
`finish_reason` of `stop` or `length`. `jq -e '.choices[0].finish_reason'`
is a good one-liner for a CI smoke test.

`run` initializes `tracing` from the `DYN_LOG` env var (defaults to
`info`); set `DYN_LOG=debug` or
`DYN_LOG=info,dynamo_backend_common=trace` for more detail. `RUST_LOG`
is not honored — `DYN_LOG` replaces it.

### Rust reference: mocker backend

[`lib/backend-common/examples/mocker`](https://github.com/ai-dynamo/dynamo/tree/main/lib/backend-common/examples/mocker)
is the canonical small-but-complete reference. Lift these patterns:

- Single shared scheduler driving many concurrent streams via a
  fan-out task and per-request `mpsc` channels.
- `ActiveRequestGuard` for RAII cleanup that runs on any stream drop.
- `biased` select with `ctx.stopped()` first, channel second — the
  shutdown-race fix discussed in Step 4.
- `cleanup()` signals every active stream via `ctx.stop_generating()`
  so each yields a clean `Cancelled` terminal instead of an error from
  channel-close.

### Rust checklist

Before shipping:

- [ ] `LLMEngine` implemented; `from_args` is inherent (not on the trait).
- [ ] All errors use `ErrorType::Backend(BackendError::X)`.
- [ ] `generate` polls `ctx.is_stopped()` between yields and emits
      `FinishReason::Cancelled` on cancel.
- [ ] Per-request cleanup uses RAII guards, not just `abort`.
- [ ] `cleanup` is idempotent.
- [ ] Conformance kit runs green: `testing::run_conformance(|| ...)`.
- [ ] Logging levels match the standards in Step 6.

### Rust see also

- [Crate README](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/README.md) — in-tree
  reference (architecture, file index, contracts at a glance).
- [`LLMEngine` trait](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/src/engine.rs) — authoritative contract.
- [Design notes](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/CLAUDE.md) — rationale and
  invariants.
- [`Worker`](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/src/worker.rs) — runtime
  lifecycle internals (signal handling, graceful shutdown, model
  registration).
- [Conformance kit](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/src/testing.rs) —
  `run_conformance`, `mock_context`, `cancelling_context`.
- [Mocker backend](../../knowledge-base/modular-components/backends/mocker/rust-backend.md) — example user guide.
- [Python sibling](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/common/backend/README.md)
  — Python ABC layered over this crate.

</Tab>
</Tabs>