Solar Open2 250B

Profiles

Prerequisites

• Kubernetes cluster with B200 nodes and the Dynamo operator installed
• A ReadWriteMany storage class. model-cache.yaml requests 1200Gi, which holds the checkpoint alongside benchmark traces and artifacts; the checkpoint itself is about 153 GB.
• A Hugging Face token, as a secret named hf-token-secret, used by the download Job to fetch the checkpoint:

kubectl create secret generic hf-token-secret --from-literal=HF_TOKEN=<token>

• For the disaggregated profile, an RDMA/InfiniBand device plugin exposing rdma/ib on the GPU nodes. If your plugin advertises a different resource name, substitute it in the worker resource requests.

Prepare the model cache

Edit storageClassName in model-cache.yaml first.

kubectl apply -f recipes/solar-open2-250b/model-cache/model-cache.yaml
kubectl apply -f recipes/solar-open2-250b/model-cache/model-download.yaml
kubectl wait --for=condition=complete job/model-download --timeout=2h

The checkpoint is approximately 153 GB. Both profiles share this cache.

Deploy

Aggregated:

kubectl apply -f recipes/solar-open2-250b/vllm/agg-b200-chat/deploy.yaml
kubectl wait --for=condition=Ready \
  dynamographdeployment/solar-open2-250b-vllm-b200-agg-chat --timeout=30m

Disaggregated:

kubectl apply -f recipes/solar-open2-250b/vllm/disagg-b200-chat/deploy.yaml
kubectl wait --for=condition=Ready \
  dynamographdeployment/solar-open2-250b-vllm-b200-disagg-chat --timeout=30m

Smoke test

Port-forward the frontend of the profile you deployed. For the aggregated profile:

kubectl port-forward svc/solar-open2-250b-vllm-b200-agg-chat-frontend 8000:8000 &

For the disaggregated profile:

kubectl port-forward svc/solar-open2-250b-vllm-b200-disagg-chat-frontend 8000:8000 &

Then, against either:

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"solar-open2-250b",
       "messages":[{"role":"user","content":"What is 12 times 7?"}],
       "max_tokens":128}'

Solar Open2 is a reasoning model. Short generations may return the answer in reasoning_content with content empty; allow a larger max_tokens to see a final answer in content.

Benchmark

Both performance Jobs read a mooncake trace from the shared-model-cache PVC. The trace is not part of the model download, so stage it first. The published figures use nim_turbo_8k_1k_70kv_chat_new_noschedule_short_15perc.jsonl (1805 requests, mean ISL 37.4k, mean OSL 1008), which ships in this repository under recipes/nemotron-3-ultra/perf/traces/.

Copy it in through any running pod that mounts the PVC. Note that the worker pods mount that volume at /shared-model-cache, while the benchmark Job mounts the same volume at /model-cache, so the file staged below is what TRACE_FILE refers to:

TRACE=recipes/nemotron-3-ultra/perf/traces/nim_turbo_8k_1k_70kv_chat_new_noschedule_short_15perc.jsonl
# The traces are stored in Git LFS. A plain clone leaves a small pointer file in
# place of the data, so fetch the real contents first and check the size: the file
# is roughly 900 KB, not a few hundred bytes.
git lfs install
git lfs pull --include="$TRACE"
ls -l "$TRACE"
# Whichever profile you deployed:
DGD=solar-open2-250b-vllm-b200-agg-chat      # or solar-open2-250b-vllm-b200-disagg-chat
# The workers mount the cache PVC; the frontend does not. Match the worker
# components by name rather than excluding the frontend, so that pods carrying no
# component-type label are not selected either.
POD=$(kubectl get pods \
  -l "nvidia.com/dynamo-graph-deployment-name=$DGD,nvidia.com/dynamo-component-type in (worker,prefill,decode)" \
  --field-selector status.phase=Running \
  -o jsonpath='{.items[0].metadata.name}')
kubectl exec "$POD" -- mkdir -p /shared-model-cache/traces
kubectl cp "$TRACE" "$POD:/shared-model-cache/traces/"

Set TRACE_FILE in the Job to match whichever trace you stage.

Then run the Job for the profile under test:

kubectl apply -f recipes/solar-open2-250b/perf/perf.yaml          # aggregated
kubectl apply -f recipes/solar-open2-250b/perf/perf-disagg.yaml   # disaggregated

Each Job runs the trace once and writes its summary to ARTIFACT_DIR. To run it again, delete the Job first and re-apply; Jobs are immutable, so re-applying over an existing one fails.

Runtime image

Both manifests pin a purpose-built runtime image by digest. It extends the Dynamo 1.4.1 vLLM runtime with the Solar Open2 architecture and parser support and a Dynamo frontend compatibility patch. The changes are Python-only.

The stock Dynamo 1.4.1 vLLM runtime does not serve this model — it has no Solar Open2 model definition — so the pinned image is required rather than optional. Pin by digest, not by tag.

Configuration notes

Both profiles

• NVFP4 weights, FP8 KV cache. Both manifests set --kv-cache-dtype fp8; the disaggregated profile additionally sets --calculate-kv-scales. FP8 halves KV bytes relative to BF16, which roughly doubles the KV pool and raises the realised prefix-cache hit rate.
• MoE backend. --moe-backend flashinfer_trtllm with --enable-flashinfer-autotune.
• Attention. FlashInfer is selected automatically. Note that FlashAttention is not compatible with an FP8 KV cache on this model and will fail at startup if forced.
• Hybrid KV pages. Solar Open2 has 12 grouped-query attention layers and 36 KDA linear-attention layers. The effective KV page is a hybrid page of roughly 1072 tokens at --block-size 16, not the block size itself. Tools that assume block size equals page size need adjusting.
• Speculative decoding is not supported on this checkpoint and is not enabled.

Aggregated profile

• Two worker replicas at tensor parallel size 2, fronted by a KV-aware router (--router-mode kv, --router-kv-events) with a ZMQ KV-event publisher on each worker. KV-aware routing outperforms round-robin on prefix-reuse workloads.
• Expert parallelism is left off. It measured as neutral for aggregated serving on this checkpoint, so the simpler configuration is used.

Disaggregated profile

• Separate prefill and decode workers connected by NixlConnector, with kv_role: kv_producer on prefill and kv_role: kv_consumer on decode.
• The topology is one prefill worker and two decode workers — TP4 with expert parallelism for prefill (4 GPUs), TP2 for each decode worker (4 GPUs), 8 GPUs in total.
• VLLM_SSM_CONV_STATE_LAYOUT=DS is required. Solar Open2’s KDA layers register as SSM layers, and their convolution state must cross the KV-transfer boundary. Without this variable the engine asserts during initialization and the workers will not start.
• Each worker requests rdma/ib equal to its GPU count, four on prefill and two on each decode worker. This is required, not optional: without it the device plugin does not inject /dev/infiniband, UCX falls back to TCP, and nothing logs an error. Verify inside a running worker with kubectl exec <pod> -c main -- ls /dev/infiniband and expect uverbsN, umadN and rdma_cm. One HCA per GPU is correct; requesting more measured worse. If your RDMA device plugin advertises a different resource name than rdma/ib, substitute it in both the requests and the limits.
• Transport is configured with NCCL_IB_DISABLE=0, UCX_TLS=^cuda_ipc, UCX_RNDV_SCHEME=get_zcopy and UCX_MAX_RNDV_RAILS=4. UCX_MAX_RNDV_RAILS tracks the number of HCAs per worker; adjust it if your nodes differ. UCX_NET_DEVICES is deliberately left unset; pinning it produced NIXL_ERR_BACKEND.
• Per-user output throughput runs close to the floor. The disaggregated profile clears the 50 tok/s target with little margin, and individual runs on contended hardware may land at or below it.
• The frontend uses round-robin routing, not KV-aware routing. This is deliberate. KV-aware routing improves the aggregated profile, but with a single prefill worker there is nothing for a prefix-aware router to choose between. KV-aware and load-aware routing were both measured on this topology and neither improved on round-robin.

Known limitations

• The disaggregated profile delivers slightly lower per-GPU throughput than the 4-GPU aggregated profile, at roughly an order of magnitude higher time-to-first-token. Prefer the aggregated profile unless independent prefill/decode scaling is the priority.
• The 1M context length requires sufficient KV capacity; reduce --max-model-len if deploying on a smaller GPU allocation.
• Speculative decoding is not available on this checkpoint.

API compatibility

Some OpenAI-compatible surfaces do not behave as advertised. These originate in the Dynamo frontend, are not specific to this model, and no recipe setting affects them.

• Logprobs. Requesting logprobs on /v1/chat/completions returns HTTP 500.
• Responses API. store=true succeeds but the response is not retrievable afterwards (GET /v1/responses/{id} returns 404). Unsupported parameters such as frequency_penalty are accepted and silently ignored.
• Completions. stop_token_ids with more than three IDs returns HTTP 500 rather than a 400.

Reasoning budget and structured output

The recipe sets reasoning_effort: high. Generations are long: on GPQA Diamond the median output is roughly 6,400 tokens, the 99th percentile 131,000, and the longest observed 209,000. A small max_tokens therefore truncates the model mid-reasoning and returns finish_reason=length with no final content, which shows up most often on structured-output requests where the JSON never arrives in message.content.

Omit max_tokens and let the 1M context window bound the request, or set it well above the reasoning length the workload needs. Values in the low thousands are not sufficient for this model.

Measured performance

Chat workload, 15% subset of the internal nim_turbo 8k/1k 70kv trace (1805 requests, mean ISL 37.4k tokens, mean OSL 1008 tokens). SLA: p50 TTFT below 5 s and p50 user output throughput at or above 50 tok/s.

Figures are measured with the profile’s benchmark manifest (perf.yaml for aggregated, perf-disagg.yaml for disaggregated). That configuration isolates the warmup phase from profiling; it does not clear server-side KV cache. For runs that must start from a cold cache, restart the worker pods between them.

Source

• Manifests: recipes/solar-open2-250b/
• Model: nota-ai/Solar-Open2-250B-Nota-NVFP4