Muse-Spark-Open-816B-A42B | NVIDIA Dynamo Documentation

This page covers serving Muse-Spark-Open-816B-A42B with SGLang (recommended) or experimental vLLM on GB200 and GB300.

Choose your deployment target

FrameworkSGLang RecommendedvLLM Experimental

GPUGB200GB300

Deployment modeAggregatedDisaggregatedKV-aware routing

Model Muse-Spark-Open-816B-A42BPrecision NVFP4 + FP8 KV cacheFramework SGLangGPUs 4x GB200, TP4/EP4

Model Muse-Spark-Open-816B-A42BPrecision NVFP4 + FP8 KV cacheFramework SGLangTopology 1P1D, TP4/EP4 per workerKV transfer NIXL

Model Muse-Spark-Open-816B-A42BPrecision NVFP4 + FP8 KV cacheFramework SGLangGPUs 8x GB200, 2 TP4/EP4 replicasRouting KV-aware

Model Muse-Spark-Open-816B-A42BPrecision NVFP4 + FP8 KV cacheFramework SGLangGPUs 4x GB300, TP4/EP4

Model Muse-Spark-Open-816B-A42BPrecision NVFP4 + FP8 KV cacheFramework SGLangTopology 1P1D, TP4/EP4 per workerKV transfer NIXL

Model Muse-Spark-Open-816B-A42BPrecision NVFP4 + FP8 KV cacheFramework SGLangGPUs 8x GB300, 2 TP4/EP4 replicasRouting KV-aware

Model Muse-Spark-Open-816B-A42BPrecision NVFP4 + FP8 KV cacheFramework vLLM (experimental)GPUs 4x GB200, TP4/EP4Topology Aggregated

Model Muse-Spark-Open-816B-A42BPrecision NVFP4 + FP8 KV cacheFramework vLLM (experimental)GPUs 4x GB300, TP4/EP4Topology Aggregated

Video requests fail with the referenced Dynamo image because it does not include decord2, the package that provides SGLang’s decord video decoder. Frontend decoding (DYN_SGL_FRONTEND_DECODING=true) is not supported for image or video inputs in these disaggregated recipes. Select an aggregated SGLang recipe for video requests.

Prerequisites

A Kubernetes cluster with the Dynamo platform installed and GB200 GPUs available. Use 4 GPUs (one node) for aggregated serving and 8 GPUs (two nodes) for either the disaggregated or KV-aware routed target.
Access to the Muse-Spark-Open-816B-A42B checkpoint and tokenizer. The public model identifier is meta-models/Muse-Spark-Open-816B-A42B-NVFP4.
Access to the SGLang runtime image referenced by the deployment manifest.
ComputeDomain support with Dynamic Resource Allocation (DRA) enabled.

A Kubernetes cluster with the Dynamo platform installed and GB300 GPUs available. Use 4 GPUs for aggregated serving and 8 GPUs for either the disaggregated or KV-aware routed target.
Access to the Muse-Spark-Open-816B-A42B checkpoint and tokenizer. The public model identifier is meta-models/Muse-Spark-Open-816B-A42B-NVFP4.
Access to the SGLang runtime image referenced by the deployment manifest.
For disaggregated serving, ComputeDomain support with Dynamic Resource Allocation (DRA) enabled.

A Kubernetes cluster with the Dynamo platform installed and 4 GB200 GPUs available.
Access to the Muse-Spark-Open-816B-A42B checkpoint and tokenizer. The public model identifier is meta-models/Muse-Spark-Open-816B-A42B-NVFP4.
Access to the vLLM runtime image referenced by the deployment manifest.

A Kubernetes cluster with the Dynamo platform installed and 4 GB300 GPUs available.
Access to the Muse-Spark-Open-816B-A42B checkpoint and tokenizer. The public model identifier is meta-models/Muse-Spark-Open-816B-A42B-NVFP4.
Access to the vLLM runtime image referenced by the deployment manifest.

Deploy

1. Create the namespace and Hugging Face secret

export NAMESPACE=your-namespace
kubectl create namespace ${NAMESPACE}
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token" \
  -n ${NAMESPACE}

2. Create shared storage

Edit recipes/muse-spark-open-816b-a42b/model-cache/model-cache.yaml and set storageClassName to a ReadWriteMany storage class available on the target cluster.

Then create the shared-model-cache PVC:

kubectl apply -f recipes/muse-spark-open-816b-a42b/model-cache/model-cache.yaml -n ${NAMESPACE}

3. Download the model

The model-download manifest is preconfigured with the public model identifier. Apply it to populate the PVC and wait for the download to finish:

kubectl apply -f recipes/muse-spark-open-816b-a42b/model-cache/model-download.yaml -n ${NAMESPACE}
kubectl wait --for=condition=Complete job/model-download \
  -n ${NAMESPACE} --timeout=10800s

All worker manifests mount the shared-model-cache PVC at /model.

4. Deploy the selected target

kubectl apply -f recipes/muse-spark-open-816b-a42b/sglang/agg-gb200-agentic/deploy.yaml -n ${NAMESPACE}

This base omits provider-specific networking. Configure RDMA for long-context requests using a cluster Kustomization.

kubectl apply -f recipes/muse-spark-open-816b-a42b/sglang/disagg-gb200-agentic/deploy.yaml -n ${NAMESPACE}

kubectl apply -f recipes/muse-spark-open-816b-a42b/sglang/agg-gb200-agentic-kv-router/deploy.yaml -n ${NAMESPACE}

kubectl apply -f recipes/muse-spark-open-816b-a42b/sglang/agg-gb300-agentic/deploy.yaml -n ${NAMESPACE}

This base omits provider-specific networking. Configure RDMA for long-context requests using a cluster Kustomization.

kubectl apply -f recipes/muse-spark-open-816b-a42b/sglang/disagg-gb300-agentic/deploy.yaml -n ${NAMESPACE}

kubectl apply -f recipes/muse-spark-open-816b-a42b/sglang/agg-gb300-agentic-kv-router/deploy.yaml -n ${NAMESPACE}

The vLLM recipes use aggregated TP4/EP4 serving on 4 GPUs and mount the same shared-model-cache PVC.

kubectl apply -f recipes/muse-spark-open-816b-a42b/vllm/agg-gb200-agentic/deploy.yaml -n ${NAMESPACE}

kubectl apply -f recipes/muse-spark-open-816b-a42b/vllm/agg-gb300-agentic/deploy.yaml -n ${NAMESPACE}

Smoke Test

Wait for the selected deployment to become ready, then forward its frontend Service:

kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-sglang-aggregate-gb200 \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-sglang-aggregate-gb200-frontend 8000:8000 -n ${NAMESPACE}

kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-sglang-disagg-gb200 \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-sglang-disagg-gb200-frontend 8000:8000 -n ${NAMESPACE}

kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-sglang-aggregate-gb200-kv \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-sglang-aggregate-gb200-kv-frontend 8000:8000 -n ${NAMESPACE}

kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-sglang-aggregate-gb300 \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-sglang-aggregate-gb300-frontend 8000:8000 -n ${NAMESPACE}

kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-sglang-disagg-gb300 \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-sglang-disagg-gb300-frontend 8000:8000 -n ${NAMESPACE}

kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-sglang-aggregate-gb300-kv \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-sglang-aggregate-gb300-kv-frontend 8000:8000 -n ${NAMESPACE}

kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-vllm-aggregate-gb200 \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-vllm-aggregate-gb200-frontend 8000:8000 -n ${NAMESPACE}

kubectl wait --for=condition=Ready dynamographdeployment/muse-spark-vllm-aggregate-gb300 \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/muse-spark-vllm-aggregate-gb300-frontend 8000:8000 -n ${NAMESPACE}

With either frontend forwarded, send a deterministic chat completion using the served model name meta-models/Muse-Spark-Open-816B-A42B-NVFP4:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"meta-models/Muse-Spark-Open-816B-A42B-NVFP4","messages":[{"role":"user","content":"Reply with exactly: READY"}],"temperature":0,"max_tokens":64}'

Benchmark

Use perf.yaml to run AIPerf against the selected deployment. The agentic trace has approximately 64K input tokens, 400 output tokens, and 90% KV-cache reuse.

Select the workload

The selector provides the endpoint and workload settings for your recipe:

export ENDPOINT=muse-spark-sglang-aggregate-gb200-frontend:8000
export CONCURRENCY=12
export REQUEST_COUNT=3541

export ENDPOINT=muse-spark-sglang-aggregate-gb200-kv-frontend:8000
export CONCURRENCY=22
export REQUEST_COUNT=3541

export ENDPOINT=muse-spark-sglang-disagg-gb200-frontend:8000
export CONCURRENCY=16
export REQUEST_COUNT=3541

export ENDPOINT=muse-spark-sglang-aggregate-gb300-frontend:8000
export CONCURRENCY=16
export REQUEST_COUNT=3541

export ENDPOINT=muse-spark-sglang-aggregate-gb300-kv-frontend:8000
export CONCURRENCY=20
export REQUEST_COUNT=3541

export ENDPOINT=muse-spark-sglang-disagg-gb300-frontend:8000
export CONCURRENCY=16
export REQUEST_COUNT=3541

export ENDPOINT=muse-spark-vllm-aggregate-gb200-frontend:8000
export CONCURRENCY=24
export REQUEST_COUNT=500

export ENDPOINT=muse-spark-vllm-aggregate-gb300-frontend:8000
export CONCURRENCY=24
export REQUEST_COUNT=500

These settings replay the first 500 requests for functional validation, rather than the full trace.

Dataset

Place the benchmark trace at /model-cache/agentic.jsonl on the shared-model-cache PVC. The Job replays it with --custom-dataset-type mooncake_trace.

Run the benchmark

Restart workers between independent runs to reset KV-cache and routing state. After the previous Job completes, run from the repository root with the selected settings:

kubectl delete job muse-spark-aiperf -n ${NAMESPACE} --ignore-not-found
kubectl set env --local -f recipes/muse-spark-open-816b-a42b/perf/perf.yaml \
  ENDPOINT="${ENDPOINT}" CONCURRENCY="${CONCURRENCY}" \
  REQUEST_COUNT="${REQUEST_COUNT}" -o yaml | \
  kubectl create -f - -n ${NAMESPACE}
kubectl wait --for=condition=Complete job/muse-spark-aiperf \
  -n ${NAMESPACE} --timeout=10800s

The Job runs five sequential warmup requests before replaying the trace.

Expected Performance

Disaggregated results were measured with cluster-specific RDMA networking.

The table lists the measured operating point for each of the six recipe targets.

Recipe	GPUs	Workload	Concurrency	Requests	Passing	Failing	Output throughput (tok/s)	Output throughput/GPU (tok/s/GPU)	Output throughput/user p50 (tok/s/user)	TTFT p50 (ms)
sglang/agg-gb200-agentic	4	trace	12	3,541	3,526	15	514.30	128.58	57.43	221.70
sglang/disagg-gb200-agentic	8	trace	16	3,541	3,526	15	768.50	96.07	58.77	1,335.20
sglang/agg-gb200-agentic-kv-router	8	trace	22	3,541	3,526	15	934.20	116.78	54.51	239.20
sglang/agg-gb300-agentic	4	trace	16	3,541	3,526	15	635.80	158.95	50.97	177.90
sglang/disagg-gb300-agentic	8	trace	16	3,541	3,526	15	835.80	104.48	62.12	289.40
sglang/agg-gb300-agentic-kv-router	8	trace	20	3,541	3,526	15	1,054.70	131.84	66.70	158.40

These results cover the first 500 trace requests and establish functional operation. They are not full-trace performance runs and are not directly comparable to the SGLang results.

Recipe	GPUs	Workload	Concurrency	Requests	Passing	Failing	Output throughput (tok/s)	Output throughput/GPU (tok/s/GPU)	Output throughput/user p50 (tok/s/user)	TTFT p50 (ms)
vllm/agg-gb200-agentic	4	trace (first 500; functional)	24	500	499	1	475.20	118.81	27.23	2,339.90
vllm/agg-gb300-agentic	4	trace (first 500; functional)	24	500	499	1	564.80	141.20	28.84	292.90

Compare All Targets

Target	Framework	Topology	Parallelism	KV cache	KV transfer	GPU count	Context length
GB200 aggregated	SGLang	Aggregated	TP4/EP4	FP8	N/A	4	1M
GB200 disaggregated	SGLang	1P1D disaggregated	TP4/EP4 per worker	FP8	NIXL	8	1M
GB200 KV-aware routing	SGLang	Aggregated, 2 replicas	2 × TP4/EP4	FP8	N/A	8	1M
GB300 aggregated	SGLang	Aggregated	TP4/EP4	FP8	N/A	4	1M
GB300 disaggregated	SGLang	1P1D disaggregated	TP4/EP4 per worker	FP8	NIXL	8	1M
GB300 KV-aware routing	SGLang	Aggregated, 2 replicas	2 × TP4/EP4	FP8	N/A	8	1M

Target	Framework	Topology	Parallelism	KV cache	KV transfer	GPU count	Context length
GB200 aggregated	vLLM	Aggregated	TP4/EP4	FP8	N/A	4	1M
GB300 aggregated	vLLM	Aggregated	TP4/EP4	FP8	N/A	4	1M

Source

Deployment assets: Muse-Spark-Open-816B-A42B SGLang manifests
Setup assets: model-cache.yaml and model-download.yaml
Benchmark manifest: perf.yaml

Deployment assets: Muse-Spark-Open-816B-A42B vLLM manifests
Setup assets: model-cache.yaml and model-download.yaml
Benchmark manifest: perf.yaml