Is this page helpful?

Model-Specific Deployment Considerations#

Some NIM LLM and VLM containers need extra profile selection or startup settings beyond the standard Get Started path. Use this page for those models.

For generic profile selection, refer to Model Profiles and Selection. For validated GPU and precision combinations, refer to the Support Matrix for NIMs. Check the Release Notes for the requirements that apply to your image version.

GPT-OSS-120B#

The MXFP4 TP1 throughput profile in gpt-oss-120b:2.0.13 is a separate configuration from the default MXFP4 TP1 profile. Select it by its full profile ID:

docker run --rm -it --gpus=all \
  -e NGC_API_KEY \
  -e NIM_MODEL_PROFILE=7de9349cde0a225c2a05f727fbabb8192c067a6d3b1440051730dd9f5c078388 \
  -p 8000:8000 \
  nvcr.io/nim/openai/gpt-oss-120b:2.0.13

For gpt-oss-120b:2.0.13, apply the following settings to the indicated configurations:

Configuration	Required Setting
MXFP4 TP8 on NVIDIA A10G or NVIDIA A100-SXM4-40GB	Add `-e NIM_KVCACHE_PERCENT=0.80` before the container image in the `docker run` command.
MXFP4 LoRA TP4 or TP8 on NVIDIA L40S; MXFP4 LoRA TP8 on NVIDIA RTX PRO 4500 Blackwell Server Edition	Append `--compilation-config '{"cudagraph_mode":"PIECEWISE"}'` after the container image to enable piecewise CUDA graphs.
Any LoRA profile	Use adapters with rank 128 or lower.

The piecewise CUDA graph option is a vLLM backend argument. Refer to the vLLM CUDA graph configuration for its syntax. For Kubernetes, pass backend arguments through the container args field. Refer to Advanced Configuration for configuration precedence and the supported NIM Operator interfaces.

GPT-OSS-20B#

For supported gpt-oss-20b:2.0.13 LoRA profiles, apply the following settings:

Use LoRA adapters with rank 128 or lower.
On NVIDIA A10G, set NIM_MAX_MODEL_LEN below the default 131,072-token context length to a value that fits the KV cache in the available device memory. Select only an A10G LoRA profile listed in the Support Matrix for NIMs.

Gemma 4#

The gemma-4-26b-a4b-it throughput profiles on NVIDIA B200 and NVIDIA GB200 use NVFP4 weights and set kv_cache_dtype=bfloat16 instead of the checkpoint’s fp8_e4m3 setting. The gemma-4-31b-it NVFP4 TP1 throughput profile also uses BF16 KV cache. These settings are included in the profiles; no additional KV-cache override is required when selecting them.

On four NVIDIA A10G GPUs, the gemma-4-26b-a4b-it BF16 TP4 profile 800f3c24e5bdaca88aebd04242be7765c65ac7abbfc9013c404d0ea0ddc9f6fc sets max_model_len=65536. The model’s native 262,144-token context does not fit the available KV-cache memory on that configuration. Use the profile’s 65,536-token context limit when sizing requests.

For example, to run the gemma-4-26b-a4b-it:2.0.13 NVFP4 TP1 throughput profile on a host with an NVIDIA B200 GPU, run the following:

export LOCAL_NIM_CACHE=~/.cache/nim/gemma-4-26b-a4b-it
mkdir -p "$LOCAL_NIM_CACHE"
chmod a+rwxt "$LOCAL_NIM_CACHE"

docker run --rm --gpus all \
  --shm-size=16GB \
  -e NGC_API_KEY \
  -e NIM_MODEL_PROFILE=234e46324452f6124bf432eaa99cc1221ed917b069ee63ebc0c4d2e636f22f3d \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  nvcr.io/nim/google/gemma-4-26b-a4b-it:2.0.13