Model Recipes

Gemma-4-31B

Serve Gemma-4-31B with Dynamo and TensorRT-LLM on B200, GB200, or H200 for long-context agentic workloads.

View as Markdown

Each target below is a Dynamo + TensorRT-LLM deployment of Gemma-4-31B for multimodal, reasoning, and tool-calling workloads.

B200 and GB200 serve the NVFP4 checkpoint with FP8 KV cache and MTP speculative decoding, while H200 serves the BF16 checkpoint. All targets use KV-aware routing and CPU KV-cache offload.

Pick your GPU architecture; every command on this page updates to match.

Choose your deployment target

GPU
Checkpoint nvidia/Gemma-4-31B-IT-NVFP4 + google/gemma-4-31B-it-assistantPrecision NVFP4 weights + FP8 KV cacheGPUs 8x B200: 8 TP1 workersParallelism TP1 per worker, 8 replicasRouting KV-aware
Checkpoint nvidia/Gemma-4-31B-IT-NVFP4 + google/gemma-4-31B-it-assistantPrecision NVFP4 weights + FP8 KV cacheGPUs 8x GB200: 8 TP1 workersParallelism TP1 per worker, 8 replicasRouting KV-aware
Checkpoint google/gemma-4-31B-itPrecision BF16 weights + 16-bit KV cacheGPUs 8x H200: 2 TP4 workersParallelism TP4 per worker, 2 replicasRouting KV-aware

Prerequisites

  • An image-pull secret when required by the cluster. Replace your-image-pull-secret in the deployment manifest with the local secret name.
  • A Kubernetes cluster with the Dynamo platform installed and 8x B200 GPUs available. See the Kubernetes Deployment Guide.
  • A Hugging Face token with access to nvidia/Gemma-4-31B-IT-NVFP4 and google/gemma-4-31B-it-assistant for multi-token prediction (MTP).
  • A Kubernetes cluster with the Dynamo platform installed and 8x GB200 GPUs available. See the Kubernetes Deployment Guide.
  • A Hugging Face token with access to nvidia/Gemma-4-31B-IT-NVFP4 and google/gemma-4-31B-it-assistant for multi-token prediction (MTP).
  • A Kubernetes cluster with the Dynamo platform installed and 8x H200 GPUs available. See the Kubernetes Deployment Guide.
  • A Hugging Face token with access to google/gemma-4-31B-it.

Deploy

1. Create namespace and secret

$export NAMESPACE=your-namespace
$kubectl create namespace ${NAMESPACE}
$kubectl create secret generic hf-token-secret \
>  --from-literal=HF_TOKEN="your-token" \
>  -n ${NAMESPACE}

2. Create storage

Edit recipes/gemma4-31b/model-cache/model-cache.yaml and set storageClassName to a ReadWriteMany storage class available on the target cluster.

If the cluster already provides a shared ReadWriteMany model-cache persistent volume claim (PVC), skip creating shared-model-cache and replace claimName: shared-model-cache in the download, deploy, and performance manifests with the existing claim name.

$kubectl apply -f recipes/gemma4-31b/model-cache/model-cache.yaml -n ${NAMESPACE}

3. Download the model

Edit recipes/gemma4-31b/model-cache/model-download.yaml before creating the Job. The B200 and GB200 downloads are enabled by default. For H200, comment out those commands and uncomment hf download google/gemma-4-31B-it. Download every checkpoint referenced by the selected deploy manifest.

$kubectl apply -f recipes/gemma4-31b/model-cache/model-download.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=7200s

4. Deploy the DGD

$kubectl apply -f recipes/gemma4-31b/trtllm/agg-b200-agentic/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/gemma4-31b/trtllm/agg-gb200-agentic/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/gemma4-31b/trtllm/agg-h200-agentic/deploy.yaml -n ${NAMESPACE}

Smoke Test

Send a test request to verify the deployment serves traffic. First forward the frontend port for your target:

$kubectl port-forward svc/gemma4-31b-agg-b200-agentic-frontend 8000:8000 -n ${NAMESPACE}
$kubectl port-forward svc/gemma4-31b-agg-gb200-agentic-frontend 8000:8000 -n ${NAMESPACE}
$kubectl port-forward svc/gemma4-31b-agg-h200-agentic-frontend 8000:8000 -n ${NAMESPACE}

Then send a chat completion using the model name for your target:

$curl http://localhost:8000/v1/chat/completions \
>  -H 'Content-Type: application/json' \
>  -d '{"model":"nvidia/Gemma-4-31B-IT-NVFP4","messages":[{"role":"user","content":"Write a one-sentence readiness check."}],"max_tokens":64}'
$curl http://localhost:8000/v1/chat/completions \
>  -H 'Content-Type: application/json' \
>  -d '{"model":"nvidia/Gemma-4-31B-IT-NVFP4","messages":[{"role":"user","content":"Write a one-sentence readiness check."}],"max_tokens":64}'
$curl http://localhost:8000/v1/chat/completions \
>  -H 'Content-Type: application/json' \
>  -d '{"model":"google/gemma-4-31B-it","messages":[{"role":"user","content":"Write a one-sentence readiness check."}],"max_tokens":64}'

Benchmark

A single AIPerf trace-replay Job, perf/perf.yaml, covers all three DGDs. It replays a 3,541-request subset of a Mooncake-format agentic trace with a median 64K input sequence length (ISL), median 400-token output sequence length (OSL), and approximately 90% KV-cache reuse. This long-context workload exercises KV-aware routing and CPU KV-cache offload under repeated prefixes. The benchmark pod is co-located with a DGD frontend through podAffinity.

Edit the env block in perf/perf.yaml to target your deployed DGD. Set ENDPOINT, TARGET_MODEL, and CONCURRENCY from the table below, and set the podAffinity value to the matching DGD name. The concurrency values reproduce the Expected Performance numbers.

Targeting a Variant

VariantENDPOINTTARGET_MODELCONCURRENCY
B200 aggregated agenticgemma4-31b-agg-b200-agentic-frontend:8000nvidia/Gemma-4-31B-IT-NVFP4192
GB200 aggregated agenticgemma4-31b-agg-gb200-agentic-frontend:8000nvidia/Gemma-4-31B-IT-NVFP4192
H200 aggregated agenticgemma4-31b-agg-h200-agentic-frontend:8000google/gemma-4-31B-it32

All targets use /model-cache/traces/64k_400_90kv_agent_new_noschedule_short_15perc.jsonl as TRACE_FILE.

Dataset

The Job passes the trace to AIPerf with --custom-dataset-type mooncake_trace. All targets use /model-cache/traces/64k_400_90kv_agent_new_noschedule_short_15perc.jsonl; its SHA-256 is f20d3f2bc83dd1306cda659fbe34e7c4d85ca5497626c98bc0b1c4d2211379d0.

Run the benchmark

Then run the Job:

$kubectl delete job gemma4-31b-bench -n ${NAMESPACE} --ignore-not-found
$kubectl create -f recipes/gemma4-31b/perf/perf.yaml -n ${NAMESPACE}
$kubectl logs -n ${NAMESPACE} -l job-name=gemma4-31b-bench -f
$kubectl wait --for=condition=Complete job/gemma4-31b-bench \
>  -n ${NAMESPACE} --timeout=10800s

For trace staging, concurrency sweeps, and fetching artifacts, see the benchmark README.

Expected Performance

Measured on the 15% agentic trace subset. System throughput is per-GPU output tokens per second; user throughput is the P50 per-request output rate. The row for your selected GPU is highlighted:

WorkloadRecipeGPUConcurrencySystem output tok/s/GPUUser output tok/s (P50)TTFT P50 (ms)
Agentic (15% subset)Aggregated (8 workers)B2001921,096.7557.223,052.24
Agentic (15% subset)Aggregated (8 workers)GB2001921,037.4049.982,893.94
Agentic (15% subset)Aggregated (2 TP4 workers)H20032153.02063.02979.09

Compare All Targets

The B200 and GB200 targets serve the model on TensorRT-LLM with KV-aware routing and EAGLE-style MTP speculative decoding (draft length 3, acceptance length of 2.88). The column for your selected GPU is highlighted:

B200 aggregated agenticGB200 aggregated agenticH200 aggregated agentic
GPU per worker1x B2001x GB2004x H200
ModeAggregatedAggregatedAggregated
FrameworkTensorRT-LLMTensorRT-LLMTensorRT-LLM
PrecisionNVFP4 + FP8 KVNVFP4 + FP8 KVBF16 + 16-bit KV
ParallelismTP1TP1TP4
RoutingKV-awareKV-awareKV-aware
Speculative decodingEAGLE-style MTPEAGLE-style MTPNot enabled; not yet verified
Context length256K256K256K
KV cache offloadingTensorRT-LLM managedTensorRT-LLM managedTensorRT-LLM managed

Limitations

  • MTP support has not been verified on H200 and is not enabled in the H200 deployment.
  • Audio is not supported by the model.
  • For streaming responses with multimodal inputs, the HTTP status code can be misreported. Newer Dynamo versions fix this by setting DYN_HTTP_PRE_COMMIT_ERROR_PEEK_MS, for example to 500.
  • Some video codecs, including AV1, are not included in the runtime image. Extend the image when those codecs are required.
  • media_io_kwargs applies only when frontend decoding is enabled. When frontend decoding is disabled, the frameworks do not receive it correctly.
  • Multimodal items support public URLs only.
  • reasoning_effort only enables or disables reasoning because the Gemma-4-31B chat template does not support finer levels.
  • On the MTP-enabled B200 and GB200 variants, one-model speculative decoding does not support min_p or min_tokens; it rejects frequency_penalty, presence_penalty, and repetition_penalty. Greedy decoding (temperature=0) also rejects n > 1 and best_of > 1.
  • This recipe supports only chat-type endpoints: /v1/chat/completions and /v1/responses.

Source