> This page is for version dev.
> For other versions, use one of these documentation indexes:
> - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt
> - dev: https://docs.nvidia.com/dynamo/dev/llms.txt
> - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt
> - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt
> - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt
> - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt
> - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt
> - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt
> - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt
> - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt
> - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt
> - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt
> - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt
> - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt
> - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt
> - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt
> - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt
> - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt
> - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt
> - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt
> - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt

> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Muse-Spark-Open-816B-A42B

This page covers serving Muse-Spark-Open-816B-A42B with SGLang (recommended) or experimental vLLM on GB200 and GB300.

Choose your deployment target

Framework

SGLang Recommended

<input type="radio" id="recipe-engine-vllm" name="recipe-engine" value="vllm" />

vLLM Experimental

GPU

GB200

<input type="radio" id="recipe-sku-gb300" name="recipe-sku" value="gb300" />

GB300

Deployment mode

Aggregated

<input type="radio" id="recipe-variant-disagg" name="recipe-variant" value="disagg" />

Disaggregated

<input type="radio" id="recipe-variant-kv-router" name="recipe-variant" value="kv-router" />

KV-aware routing

**Model** Muse-Spark-Open-816B-A42B **Precision** NVFP4 + FP8 KV cache **Framework** SGLang **GPUs** 4x GB200, TP4/EP4

**Model** Muse-Spark-Open-816B-A42B **Precision** NVFP4 + FP8 KV cache **Framework** SGLang **Topology** 1P1D, TP4/EP4 per worker **KV transfer** NIXL

**Model** Muse-Spark-Open-816B-A42B **Precision** NVFP4 + FP8 KV cache **Framework** SGLang **GPUs** 8x GB200, 2 TP4/EP4 replicas **Routing** KV-aware

**Model** Muse-Spark-Open-816B-A42B **Precision** NVFP4 + FP8 KV cache **Framework** SGLang **GPUs** 4x GB300, TP4/EP4

**Model** Muse-Spark-Open-816B-A42B **Precision** NVFP4 + FP8 KV cache **Framework** SGLang **Topology** 1P1D, TP4/EP4 per worker **KV transfer** NIXL

**Model** Muse-Spark-Open-816B-A42B **Precision** NVFP4 + FP8 KV cache **Framework** SGLang **GPUs** 8x GB300, 2 TP4/EP4 replicas **Routing** KV-aware

**Model** Muse-Spark-Open-816B-A42B **Precision** NVFP4 + FP8 KV cache **Framework** vLLM (experimental) **GPUs** 4x GB200, TP4/EP4 **Topology** Aggregated

**Model** Muse-Spark-Open-816B-A42B **Precision** NVFP4 + FP8 KV cache **Framework** vLLM (experimental) **GPUs** 4x GB300, TP4/EP4 **Topology** Aggregated

> **Warning**
>
> Video requests fail with the referenced Dynamo image because it does not include `decord2`, the package that provides SGLang's `decord` video decoder. Frontend decoding (`DYN_SGL_FRONTEND_DECODING=true`) is not supported for image or video inputs in these disaggregated recipes. Select an aggregated SGLang recipe for video requests.

## Prerequisites

* A Kubernetes cluster with the Dynamo platform installed and GB200 GPUs available. Use 4 GPUs (one node) for aggregated serving and 8 GPUs (two nodes) for either the disaggregated or KV-aware routed target.
* Access to the Muse-Spark-Open-816B-A42B checkpoint and tokenizer. The public model identifier is `meta-models/Muse-Spark-Open-816B-A42B-NVFP4`.
* Access to the SGLang runtime image referenced by the deployment manifest.
* ComputeDomain support with Dynamic Resource Allocation (DRA) enabled.

- A Kubernetes cluster with the Dynamo platform installed and GB300 GPUs available. Use 4 GPUs for aggregated serving and 8 GPUs for either the disaggregated or KV-aware routed target.
- Access to the Muse-Spark-Open-816B-A42B checkpoint and tokenizer. The public model identifier is `meta-models/Muse-Spark-Open-816B-A42B-NVFP4`.
- Access to the SGLang runtime image referenced by the deployment manifest.
- For disaggregated serving, ComputeDomain support with Dynamic Resource Allocation (DRA) enabled.

* A Kubernetes cluster with the Dynamo platform installed and 4 GB200 GPUs available.
* Access to the Muse-Spark-Open-816B-A42B checkpoint and tokenizer. The public model identifier is `meta-models/Muse-Spark-Open-816B-A42B-NVFP4`.
* Access to the vLLM runtime image referenced by the deployment manifest.

- A Kubernetes cluster with the Dynamo platform installed and 4 GB300 GPUs available.
- Access to the Muse-Spark-Open-816B-A42B checkpoint and tokenizer. The public model identifier is `meta-models/Muse-Spark-Open-816B-A42B-NVFP4`.
- Access to the vLLM runtime image referenced by the deployment manifest.

## Deploy

### 1. Create the namespace and Hugging Face secret

```bash
export NAMESPACE=your-namespace
kubectl create namespace ${NAMESPACE}
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token" \
  -n ${NAMESPACE}
```

### 2. Create shared storage

> **Warning**
>
> Edit **`recipes/muse-spark-open-816b-a42b/model-cache/model-cache.yaml`** and set **`storageClassName`** to a ReadWriteMany storage class available on the target cluster.

Then create the `shared-model-cache` PVC:

```bash
kubectl apply -f recipes/muse-spark-open-816b-a42b/model-cache/model-cache.yaml -n ${NAMESPACE}
```

### 3. Download the model

The model-download manifest is preconfigured with the public model identifier. Apply it to populate the PVC and wait for the download to finish:

```bash
kubectl apply -f recipes/muse-spark-open-816b-a42b/model-cache/model-download.yaml -n ${NAMESPACE}
kubectl wait --for=condition=Complete job/model-download \
  -n ${NAMESPACE} --timeout=10800s
```

All worker manifests mount the `shared-model-cache` PVC at `/model`.

### 4. Deploy the selected target

```bash
kubectl apply -f recipes/muse-spark-open-816b-a42b/sglang/agg-gb200-agentic/deploy.yaml -n ${NAMESPACE}
```

This base omits provider-specific networking. Configure [RDMA](/dynamo/dev/kubernetes/installation/rdma-setup/overview) for long-context requests using a [cluster Kustomization](https://github.com/ai-dynamo/dynamo/blob/main/recipes/templates/kustomize/README.md).

```bash
kubectl apply -f recipes/muse-spark-open-816b-a42b/sglang/disagg-gb200-agentic/deploy.yaml -n ${NAMESPACE}
```

```bash
kubectl apply -f recipes/muse-spark-open-816b-a42b/sglang/agg-gb200-agentic-kv-router/deploy.yaml -n ${NAMESPACE}
```

```bash
kubectl apply -f recipes/muse-spark-open-816b-a42b/sglang/agg-gb300-agentic/deploy.yaml -n ${NAMESPACE}
```

This base omits provider-specific networking. Configure [RDMA](/dynamo/dev/kubernetes/installation/rdma-setup/overview) for long-context requests using a [cluster Kustomization](https://github.com/ai-dynamo/dynamo/blob/main/recipes/templates/kustomize/README.md).

```bash
kubectl apply -f recipes/muse-spark-open-816b-a42b/sglang/disagg-gb300-agentic/deploy.yaml -n ${NAMESPACE}
```

```bash
kubectl apply -f recipes/muse-spark-open-816b-a42b/sglang/agg-gb300-agentic-kv-router/deploy.yaml -n ${NAMESPACE}
```

The vLLM recipes use aggregated TP4/EP4 serving on 4 GPUs and mount the same `shared-model-cache` PVC.

```bash
kubectl apply -f recipes/muse-spark-open-816b-a42b/vllm/agg-gb200-agentic/deploy.yaml -n ${NAMESPACE}
```

```bash
kubectl apply -f recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml -n ${NAMESPACE}
```

## Smoke Test

Wait for the selected deployment to become ready, then forward its frontend Service:

```bash
kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-sglang-aggregate-gb200 \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-sglang-aggregate-gb200-frontend 8000:8000 -n ${NAMESPACE}
```

```bash
kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-sglang-disagg-gb200 \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-sglang-disagg-gb200-frontend 8000:8000 -n ${NAMESPACE}
```

```bash
kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-sglang-aggregate-gb200-kv \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-sglang-aggregate-gb200-kv-frontend 8000:8000 -n ${NAMESPACE}
```

```bash
kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-sglang-aggregate-gb300 \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-sglang-aggregate-gb300-frontend 8000:8000 -n ${NAMESPACE}
```

```bash
kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-sglang-disagg-gb300 \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-sglang-disagg-gb300-frontend 8000:8000 -n ${NAMESPACE}
```

```bash
kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-sglang-aggregate-gb300-kv \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-sglang-aggregate-gb300-kv-frontend 8000:8000 -n ${NAMESPACE}
```

```bash
kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-vllm-aggregate-gb200 \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-vllm-aggregate-gb200-frontend 8000:8000 -n ${NAMESPACE}
```

```bash
kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-vllm-aggregate-gb300 \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-vllm-aggregate-gb300-frontend 8000:8000 -n ${NAMESPACE}
```

With either frontend forwarded, send a deterministic chat completion using the served model name `meta-models/Muse-Spark-Open-816B-A42B-NVFP4`:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"meta-models/Muse-Spark-Open-816B-A42B-NVFP4","messages":[{"role":"user","content":"Reply with exactly: READY"}],"temperature":0,"max_tokens":64}'
```

## Benchmark

Use [perf.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/muse-spark-open-816b-a42b/perf/perf.yaml) to run AIPerf against the selected deployment. The agentic trace has approximately 64K input tokens, 400 output tokens, and 90% KV-cache reuse.

### Select the workload

The selector provides the endpoint and workload settings for your recipe:

```bash
export ENDPOINT=muse-spark-sglang-aggregate-gb200-frontend:8000
export CONCURRENCY=12
export REQUEST_COUNT=3541
```

```bash
export ENDPOINT=muse-spark-sglang-aggregate-gb200-kv-frontend:8000
export CONCURRENCY=22
export REQUEST_COUNT=3541
```

```bash
export ENDPOINT=muse-spark-sglang-disagg-gb200-frontend:8000
export CONCURRENCY=16
export REQUEST_COUNT=3541
```

```bash
export ENDPOINT=muse-spark-sglang-aggregate-gb300-frontend:8000
export CONCURRENCY=16
export REQUEST_COUNT=3541
```

```bash
export ENDPOINT=muse-spark-sglang-aggregate-gb300-kv-frontend:8000
export CONCURRENCY=20
export REQUEST_COUNT=3541
```

```bash
export ENDPOINT=muse-spark-sglang-disagg-gb300-frontend:8000
export CONCURRENCY=16
export REQUEST_COUNT=3541
```

```bash
export ENDPOINT=muse-spark-vllm-aggregate-gb200-frontend:8000
export CONCURRENCY=24
export REQUEST_COUNT=500
```

```bash
export ENDPOINT=muse-spark-vllm-aggregate-gb300-frontend:8000
export CONCURRENCY=24
export REQUEST_COUNT=500
```

These settings replay the first 500 requests for functional validation, rather than the full trace.

### Dataset

Place the [benchmark trace](https://media.githubusercontent.com/media/ai-dynamo/dynamo/main/recipes/kimi-k2.6/perf/traces/64k_400_90kv_agent_new_noschedule_short_15perc.jsonl) at `/model-cache/agentic.jsonl` on the `shared-model-cache` PVC. The Job replays it with `--custom-dataset-type mooncake_trace`.

### Run the benchmark

Restart workers between independent runs to reset KV-cache and routing state. After the previous Job completes, run from the repository root with the selected settings:

```bash
kubectl delete job muse-spark-aiperf -n ${NAMESPACE} --ignore-not-found
kubectl set env --local -f recipes/muse-spark-open-816b-a42b/perf/perf.yaml \
  ENDPOINT="${ENDPOINT}" CONCURRENCY="${CONCURRENCY}" \
  REQUEST_COUNT="${REQUEST_COUNT}" -o yaml | \
  kubectl create -f - -n ${NAMESPACE}
kubectl wait --for=condition=Complete job/muse-spark-aiperf \
  -n ${NAMESPACE} --timeout=10800s
```

The Job runs five sequential warmup requests before replaying the trace.

## Expected Performance

Disaggregated results were measured with cluster-specific RDMA networking.

The table lists the measured operating point for each of the six recipe targets.

<table>
  <thead>
    <tr><th>Recipe</th><th>GPUs</th><th>Workload</th><th>Concurrency</th><th>Requests</th><th>Passing</th><th>Failing</th><th>Output throughput (tok/s)</th><th>Output throughput/GPU (tok/s/GPU)</th><th>Output throughput/user p50 (tok/s/user)</th><th>TTFT p50 (ms)</th></tr>
  </thead>

  <tbody>
    <tr data-sku="gb200" data-variant="agg">
      <td>sglang/agg-gb200-agentic</td>

      <td>4</td>

      <td>trace</td>

      <td>12</td>

      <td>3,541</td>

      <td>3,526</td>

      <td>15</td>

      <td>514.30</td>

      <td>128.58</td>

      <td>57.43</td>

      <td>221.70</td>
    </tr>

    <tr data-sku="gb200" data-variant="disagg">
      <td>sglang/disagg-gb200-agentic</td>

      <td>8</td>

      <td>trace</td>

      <td>16</td>

      <td>3,541</td>

      <td>3,526</td>

      <td>15</td>

      <td>768.50</td>

      <td>96.07</td>

      <td>58.77</td>

      <td>1,335.20</td>
    </tr>

    <tr data-sku="gb200" data-variant="kv-router">
      <td>sglang/agg-gb200-agentic-kv-router</td>

      <td>8</td>

      <td>trace</td>

      <td>22</td>

      <td>3,541</td>

      <td>3,526</td>

      <td>15</td>

      <td>934.20</td>

      <td>116.78</td>

      <td>54.51</td>

      <td>239.20</td>
    </tr>

    <tr data-sku="gb300" data-variant="agg">
      <td>sglang/agg-gb300-agentic</td>

      <td>4</td>

      <td>trace</td>

      <td>16</td>

      <td>3,541</td>

      <td>3,526</td>

      <td>15</td>

      <td>635.80</td>

      <td>158.95</td>

      <td>50.97</td>

      <td>177.90</td>
    </tr>

    <tr data-sku="gb300" data-variant="disagg">
      <td>sglang/disagg-gb300-agentic</td>

      <td>8</td>

      <td>trace</td>

      <td>16</td>

      <td>3,541</td>

      <td>3,526</td>

      <td>15</td>

      <td>835.80</td>

      <td>104.48</td>

      <td>62.12</td>

      <td>289.40</td>
    </tr>

    <tr data-sku="gb300" data-variant="kv-router">
      <td>sglang/agg-gb300-agentic-kv-router</td>

      <td>8</td>

      <td>trace</td>

      <td>20</td>

      <td>3,541</td>

      <td>3,526</td>

      <td>15</td>

      <td>1,054.70</td>

      <td>131.84</td>

      <td>66.70</td>

      <td>158.40</td>
    </tr>
  </tbody>
</table>

These results cover the first 500 trace requests and establish functional operation. They are not full-trace performance runs and are not directly comparable to the SGLang results.

<table>
  <thead>
    <tr><th>Recipe</th><th>GPUs</th><th>Workload</th><th>Concurrency</th><th>Requests</th><th>Passing</th><th>Failing</th><th>Output throughput (tok/s)</th><th>Output throughput/GPU (tok/s/GPU)</th><th>Output throughput/user p50 (tok/s/user)</th><th>TTFT p50 (ms)</th></tr>
  </thead>

  <tbody>
    <tr data-sku="gb200" data-highlight-sku="gb200">
      <td>vllm/agg-gb200-agentic</td>

      <td>4</td>

      <td>trace (first 500; functional)</td>

      <td>24</td>

      <td>500</td>

      <td>499</td>

      <td>1</td>

      <td>475.20</td>

      <td>118.81</td>

      <td>27.23</td>

      <td>2,339.90</td>
    </tr>

    <tr data-sku="gb300" data-highlight-sku="gb300">
      <td>vllm/agg-gb300-agentic</td>

      <td>4</td>

      <td>trace (first 500; functional)</td>

      <td>24</td>

      <td>500</td>

      <td>499</td>

      <td>1</td>

      <td>564.80</td>

      <td>141.20</td>

      <td>28.84</td>

      <td>292.90</td>
    </tr>
  </tbody>
</table>

## Compare All Targets

<table>
  <thead>
    <tr><th>Target</th><th>Framework</th><th>Topology</th><th>Parallelism</th><th>KV cache</th><th>KV transfer</th><th>GPU count</th><th>Context length</th></tr>
  </thead>

  <tbody>
    <tr data-sku="gb200" data-variant="agg">
      <td>GB200 aggregated</td>

      <td>SGLang</td>

      <td>Aggregated</td>

      <td>TP4/EP4</td>

      <td>FP8</td>

      <td>N/A</td>

      <td>4</td>

      <td>1M</td>
    </tr>

    <tr data-sku="gb200" data-variant="disagg">
      <td>GB200 disaggregated</td>

      <td>SGLang</td>

      <td>1P1D disaggregated</td>

      <td>TP4/EP4 per worker</td>

      <td>FP8</td>

      <td>NIXL</td>

      <td>8</td>

      <td>1M</td>
    </tr>

    <tr data-sku="gb200" data-variant="kv-router">
      <td>GB200 KV-aware routing</td>

      <td>SGLang</td>

      <td>Aggregated, 2 replicas</td>

      <td>2 × TP4/EP4</td>

      <td>FP8</td>

      <td>N/A</td>

      <td>8</td>

      <td>1M</td>
    </tr>

    <tr data-sku="gb300" data-variant="agg">
      <td>GB300 aggregated</td>

      <td>SGLang</td>

      <td>Aggregated</td>

      <td>TP4/EP4</td>

      <td>FP8</td>

      <td>N/A</td>

      <td>4</td>

      <td>1M</td>
    </tr>

    <tr data-sku="gb300" data-variant="disagg">
      <td>GB300 disaggregated</td>

      <td>SGLang</td>

      <td>1P1D disaggregated</td>

      <td>TP4/EP4 per worker</td>

      <td>FP8</td>

      <td>NIXL</td>

      <td>8</td>

      <td>1M</td>
    </tr>

    <tr data-sku="gb300" data-variant="kv-router">
      <td>GB300 KV-aware routing</td>

      <td>SGLang</td>

      <td>Aggregated, 2 replicas</td>

      <td>2 × TP4/EP4</td>

      <td>FP8</td>

      <td>N/A</td>

      <td>8</td>

      <td>1M</td>
    </tr>
  </tbody>
</table>

<table>
  <thead>
    <tr><th>Target</th><th>Framework</th><th>Topology</th><th>Parallelism</th><th>KV cache</th><th>KV transfer</th><th>GPU count</th><th>Context length</th></tr>
  </thead>

  <tbody>
    <tr data-sku="gb200" data-highlight-sku="gb200">
      <td>GB200 aggregated</td>

      <td>vLLM</td>

      <td>Aggregated</td>

      <td>TP4/EP4</td>

      <td>FP8</td>

      <td>N/A</td>

      <td>4</td>

      <td>1M</td>
    </tr>

    <tr data-sku="gb300" data-highlight-sku="gb300">
      <td>GB300 aggregated</td>

      <td>vLLM</td>

      <td>Aggregated</td>

      <td>TP4/EP4</td>

      <td>FP8</td>

      <td>N/A</td>

      <td>4</td>

      <td>1M</td>
    </tr>
  </tbody>
</table>

## Source

* Deployment assets: [Muse-Spark-Open-816B-A42B SGLang manifests](https://github.com/ai-dynamo/dynamo/tree/main/recipes/muse-spark-open-816b-a42b/sglang)
* Setup assets: [model-cache.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/muse-spark-open-816b-a42b/model-cache/model-cache.yaml) and [model-download.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/muse-spark-open-816b-a42b/model-cache/model-download.yaml)
* Benchmark manifest: [perf.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/muse-spark-open-816b-a42b/perf/perf.yaml)

- Deployment assets: [Muse-Spark-Open-816B-A42B vLLM manifests](https://github.com/ai-dynamo/dynamo/tree/main/recipes/muse-spark-open-816b-a42b/vllm)
- Setup assets: [model-cache.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/muse-spark-open-816b-a42b/model-cache/model-cache.yaml) and [model-download.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/muse-spark-open-816b-a42b/model-cache/model-download.yaml)
- Benchmark manifest: [perf.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/muse-spark-open-816b-a42b/perf/perf.yaml)