Is this page helpful?

Model Profiles and Selection#

A model profile is a pre-validated inference configuration that NIM LLM and VLM selects at container startup.

Every model-specific NIM container ships with a model manifest — a catalog of one or more profiles. Each profile is defined by tags for the backend engine, model precision, tensor parallelism size (TP), pipeline parallelism size (PP), and LoRA support.

For model-free NIM, the model manifest is generated at runtime with generic profiles that can deploy the NIM across a wide range of system configurations.

At container startup, NIM selects exactly one profile from the manifest. The selected profile determines which model files are downloaded and how the inference backend is launched.

Profile Naming Convention#

Profile descriptions are generated from the tags in the container’s model manifest. A common base form is:

<backend>-<precision>-tp<N>-pp1[-lora]

Where:

  • <backend> is the inference engine in the container image (vllm or sglang). Each image ships only its own engine’s profiles: the SGLang image lists and loads sglang-* profiles, and the vLLM image lists and loads vllm-* profiles.

  • <precision> is the quantization format (bf16, fp8, mxfp4, or nvfp4)

  • tp<N> is the tensor parallelism degree (number of GPUs)

  • pp1 indicates single-stage pipeline parallelism

  • The -lora suffix indicates the profile supports LoRA adapter loading

For example, vllm-bf16-tp4-pp1-lora uses BF16 precision across four GPUs with LoRA support on the vLLM backend, and sglang-bf16-tp4-pp1 runs the equivalent configuration on the SGLang backend.

Descriptions can include additional tag values, such as a GPU name, a performance variant (throughput or latency), or a minimum-memory value. The base form above is not a template for reconstructing an exact selector.

Copy the complete description printed by list-model-profiles for the image version you plan to deploy, or use the full profile ID. The optimized-profile tables in the Support Matrix for NIMs provide full IDs to distinguish profiles that share the same precision and parallelism.

List Available Profiles#

Before pulling a model-specific NIM image, use the NIM Metadata API to inspect published profiles for a container tag over HTTP. The Choose a Profile for Your Hardware guide explains how to compare GPU targeting, memory requirements, precision, and parallelism, then record a profile ID for deployment with NIM_MODEL_PROFILE. Metadata coverage varies by NIM and tag, and missing fields represent unknown requirements. If metadata is unavailable, use the Support Matrix for NIMs and the list-model-profiles command below to inspect the container on your system.

To list the profiles that a container can run, complete the following steps:

Run the list-model-profiles command:

docker run --rm --gpus=all \
  <nim_llm_image> \
  list-model-profiles

Example output:

MODEL PROFILES
- Compatible with system and runnable:
  - dcec66a50892315842bdc46d5b2d8648fed3fe3d3382437f0a811c56eff8c39c (vllm-bf16-tp1-pp1) [requires >=18 GB/gpu]
  - With LoRA support:
    - d66193b819d2bc2ae40aefcec0da5997b5f9187dd79b8155ec111b16999d18e0 (vllm-bf16-tp1-pp1-feat_lora) [requires >=22 GB/gpu]
- Compatible with system but low memory:
  - a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2 (vllm-bf16-tp1-pp1) [requires >=45 GB/gpu, try --max-model-len=4096 to reduce to >=30 GB/gpu]
- Incompatible with system:
    - 27af459c9caa0f9b34d5e07e5962960df6b0120df2039d06148e0e63595195e5 (vllm-bf16-tp2-pp1)
    - 30d16624c8100d40e6cde3af7f4e4ff6028f776e92efdcf09fcb515ae65662c0 (vllm-bf16-tp4-pp1)
    - 6f888502f35dc189f8c67f3e11174028a4ce42e92868e6a0ca10ef1d84953874 (vllm-bf16-tp8-pp1)

Note

The example above shows output from a vLLM container image. The SGLang container image emits profiles with sglang-* descriptions instead (for example, sglang-bf16-tp1-pp1).

Each profile has:

  • Profile ID: A unique 64-character profile ID.

  • Profile description: A human-readable string constructed by joining tag values with hyphens (for example, vllm-bf16-tp1-pp1 on a vLLM image or sglang-bf16-tp1-pp1 on an SGLang image).

  • Memory annotation: An estimated VRAM requirement per GPU, shown in brackets (for example, [requires >=18 GB/gpu]).

Memory-Based Profile Classification#

NIM estimates the GPU VRAM required by each profile and classifies it into one of three categories based on the available memory on the system:

Category

Meaning

Action

Compatible

Estimated VRAM fits within available GPU memory

Profile is eligible for automatic selection. Backend startup still validates the configuration.

Low memory

The estimated allocation fits only with a shorter context length

NIM can attempt startup with a warning. Explicitly reduce context or choose another profile if needed.

Incompatible

The estimated allocation exceeds available GPU memory even with a shorter context length

Memory estimates remain advisory. Consider higher tensor parallelism or a quantized precision.

If a profile is classified as low memory, the listing output includes a suggestion. For example:

[requires >=45 GB/gpu, try --max-model-len=4096 to reduce to >=30 GB/gpu]

To apply the suggestion, pass the --max-model-len argument:

docker run --rm -it --gpus=all \
  -p 8000:8000 \
  <nim_llm_image> \
  --max-model-len 4096

Note

Reducing --max-model-len limits the maximum sequence length (input + output tokens) per request. Choose a value that fits your use case.

For single-node vLLM and SGLang GPU deployments using tensor and pipeline parallelism, NIM checks the selected profile before downloading model weights. For automatic selection, NIM_MODEL_PROFILE=default, and explicit profile selection, predicted memory shortfalls and unavailable or stale estimates produce warnings, not startup rejections. NIM attempts to download and start the selected profile without automatically reducing context length. Hardware, precision, topology, feature, and configuration validation still apply, as do download failures and backend startup errors.

A declared minimum remains a lower bound when the profile’s memory settings are unchanged. LoRA rank and pool limits do not invalidate that minimum when LoRA is disabled. NIM attempts a fresh estimate when active memory settings change. For vLLM, NIM also checks modeled managed-memory use against the resolved --gpu-memory-utilization budget. When the profile, runtime file, or user fixes utilization, NIM also compares its initial reservation against reported free memory. An explicit --kv-cache-memory-bytes allocation is checked against the estimated context requirement and available physical memory instead of utilization-based cache sizing; the initial reservation check still applies. These checks use estimates; they do not measure every allocation during backend startup. An unresolved estimate does not establish that the backend cannot run the profile. Multi-node deployments and additional data or expert parallelism topologies retain their existing memory checks.

How Profile Selection Works#

NIM uses a priority-ordered selection chain to decide which profile to use. The chain is evaluated top-to-bottom; the first selector that produces a match wins.

Priority

Selector

Trigger

Description

1 (highest)

Default profile selector

NIM_MODEL_PROFILE="default"

Uses backend priority, with a memory-aware compatible fallback if nimlib finds no fitting default. Memory estimates remain advisory.

2

Environment-based profile selector

NIM_MODEL_PROFILE=<id-or-name>

Matches an explicit profile by checksum or description.

3

Memory-aware profile selector

(automatic)

Prefers profiles estimated to fit available GPU memory, with a compatible fallback when none fit or estimates are unavailable. Prefers non-LoRA profiles unless LoRA is enabled.

4 (lowest)

Manifest profile selector

(no env var set)

Uses profile_selection_criteria from the manifest to pick a profile automatically based on hardware compatibility (GPU device, VRAM) and parallelism settings.

The memory-aware selector estimates GPU memory requirements from model weights, KV cache, activations, and overhead. Estimates guide preference but do not exclude an otherwise compatible fallback. Hardware, feature, and parallelism constraints still determine which profiles are eligible.

Select a Profile#

The method you use to select a profile depends on your requirements and environment. You can allow NIM to pick a compatible profile automatically, or you can specify the exact profile by ID or by description.

Automatic Selection (Default)#

If you do not set NIM_MODEL_PROFILE, NIM automatically selects the best compatible profile from the manifest based on your hardware (GPU device, available VRAM, estimated memory requirements, and parallelism constraints).

To start the container with automatic profile selection, run the container without NIM_MODEL_PROFILE:

docker run --rm -it --gpus=all \
  -p 8000:8000 \
  <nim_llm_image>

Intelligent Default Selection#

Setting NIM_MODEL_PROFILE to "default" triggers intelligent default selection. NIM picks the best compatible profile based on the following criteria:

  1. Hardware compatibility (GPU device and parallelism)

  2. Backend priority

  3. LoRA configuration

If nimlib finds no fitting default, NIM retries memory-aware selection with a compatible fallback. Memory estimates guide preference but do not reject that fallback.

To use intelligent default selection, set NIM_MODEL_PROFILE to "default" and start the container:

docker run --rm -it --gpus=all \
  -e NIM_MODEL_PROFILE="default" \
  -p 8000:8000 \
  <nim_llm_image>

Explicit Selection by Profile ID#

Specify the full 64-character profile ID from the manifest of the image you plan to deploy. This selects that exact profile without reconstructing its description. Check the ID again when changing image versions: a profile change can produce a different ID.

To select a profile by ID, set NIM_MODEL_PROFILE to the 64-character profile ID:

docker run --rm -it --gpus=all \
  -e NIM_MODEL_PROFILE="70edb8bb9f8511ce2ea195e3caebcc3c7191dc27fea0c8d4acf9c0d9a69e43cd" \
  -p 8000:8000 \
  <nim_llm_image>

Explicit Selection by Profile Description (Friendly Name)#

If the value of NIM_MODEL_PROFILE is not a valid Profile ID, NIM tries to match it against the profile description — a human-readable string constructed from ordered profile tags.

To select a profile by description, set NIM_MODEL_PROFILE to the profile description:

docker run --rm -it --gpus=all \
  -e NIM_MODEL_PROFILE=vllm-bf16-tp1-pp1 \
  -p 8000:8000 \
  <nim_llm_image>

Note

On an SGLang container image, use the matching sglang- profile description (or its hash) instead, for example sglang-bf16-tp1-pp1.

Tip

Use list-model-profiles to discover the exact profile IDs and descriptions available in your container.

Configuration Precedence#

NIM_MODEL_PROFILE selects a model profile. The following sources override its settings, from highest to lowest priority: direct container arguments, NIM_PASSTHROUGH_ARGS, NIM environment variables that map to backend arguments, and runtime_config.json. For the complete order, refer to Advanced Configuration.

For example, if a profile specifies tp=2 but the user also sets tensor parallelism to 4 (NIM_TENSOR_PARALLEL_SIZE=4, or the canonical flag --tensor-parallel-size 4), the backend launches with TP=4.

Important

When backend arguments override profile settings, the overridden values are resolved before model download. NIM selects and downloads the profile that matches the final resolved configuration, so the downloaded model files always match the launch configuration.

Pass Backend CLI Arguments#

Use NIM environment variables or canonical CLI flags for settings shared by both inference engines. NIM translates these flags into the engine’s arguments. For example, set tensor parallelism to 2 with the environment variable:

docker run --rm -it --gpus=all \
  -e NIM_TENSOR_PARALLEL_SIZE=2 \
  -p 8000:8000 \
  <nim_llm_image>

Passing the canonical flag as a container argument is equivalent:

docker run --rm -it --gpus=all \
  -p 8000:8000 \
  <nim_llm_image> \
  --tensor-parallel-size 2

The following settings work with both vLLM and SGLang:

Setting

NIM Environment Variable

Canonical Flag

Tensor parallelism

NIM_TENSOR_PARALLEL_SIZE

--tensor-parallel-size

Pipeline parallelism

NIM_PIPELINE_PARALLEL_SIZE

--pipeline-parallel-size

For LoRA configuration and profile selection, refer to Fine-Tuning with LoRA. Engine-specific arguments must match the backend in your container image.

Note

For a Kubernetes workload, pass the arguments through the container args field. For a NIM Operator deployment that does not set spec.multiNode, use spec.args. Do not use spec.args for a multi-node NIMService; the current LeaderWorkerSet path does not propagate it to the leader or worker NIM containers. The direct-argument interfaces preserve any NIM_PASSTHROUGH_ARGS value provided by the image. For details, refer to Advanced Configuration.

Changes from NIM LLM 1.x#

The following profile selection mechanisms from NIM LLM version 1.x are no longer supported:

Removed Feature

1.x Example

Custom profile selectors

-e NIM_CUSTOM_SELECTOR_CLASSES="inference:MyCustomSelector"

LLM-based profile selector (backend priority chain)

Automatic backend priority: TensorRT-LLM > vLLM > SGLang

Tag-based profile selector

-e NIM_TAGS_SELECTOR="llm_engine=vllm,tp=1"

Tip

Use NIM_MODEL_PROFILE with a profile ID or description as a replacement for these deprecated mechanisms. For further guidance, refer to the 1.x Migration Guide.

Warning

Do not carry NIM_CUSTOM_SELECTOR_CLASSES or NIM_CUSTOM_NIM_SELECTION_CRITERIA_PROVIDER_CLASS over from a 1.x deployment. The container’s own profile selector is active without either variable. When a value is set, nimlib still validates it against the classes in the container’s inference module and never imports a module the value names. A value that is malformed, such as the dotted my_selector.MyCustomSelector form, or that names a class the module does not define stops the container from starting.

Next Steps#

After you select a profile, download model artifacts and start serving: