Is this page helpful?

Model Profiles and Selection#

A model profile is a pre-validated inference configuration that NIM LLM and VLM selects at container startup.

Every model-specific NIM container ships with a model manifest — a catalog of one or more profiles. Each profile is defined by tags for the backend engine, model precision, tensor parallelism size (TP), pipeline parallelism size (PP), and LoRA support.

For model-free NIM, the model manifest is generated at runtime with generic profiles that can deploy the NIM across a wide range of system configurations.

At container startup, NIM selects exactly one profile from the manifest. The selected profile determines which model files are downloaded and how the inference backend is launched.

Profile Naming Convention#

Profile descriptions are generated from the tags in the container’s model manifest. A common base form is:

<backend>-<precision>-tp<N>-pp1[-lora]

Where:

  • <backend> is the inference engine (vllm, sglang, or trtllm). The backend must match the inference engine in the container image you run. For example, the SGLang container image only loads sglang-* profiles.

  • <precision> is the quantization format (bf16, fp8, mxfp4, or nvfp4)

  • tp<N> is the tensor parallelism degree (number of GPUs)

  • pp1 indicates single-stage pipeline parallelism

  • The -lora suffix indicates the profile supports LoRA adapter loading

For example, vllm-bf16-tp4-pp1-lora uses BF16 precision across four GPUs with LoRA support on the vLLM backend, and sglang-bf16-tp4-pp1 runs the equivalent configuration on the SGLang backend.

Descriptions can include additional tag values, such as a GPU name, a performance variant (throughput or latency), or a minimum-memory value. The base form above is not a template for reconstructing an exact selector.

Copy the complete description printed by list-model-profiles for the image version you plan to deploy, or use the full profile ID. The optimized-profile tables in the Support Matrix for NIMs provide full IDs to distinguish profiles that share the same precision and parallelism.

List Available Profiles#

To list the profiles that a container can run, complete the following steps:

Run the list-model-profiles command:

docker run --rm --gpus=all \
  <nim_llm_image> \
  list-model-profiles

Example output:

MODEL PROFILES
- Compatible with system and runnable:
  - dcec66a50892315842bdc46d5b2d8648fed3fe3d3382437f0a811c56eff8c39c (vllm-bf16-tp1-pp1) [requires >=18 GB/gpu]
  - With LoRA support:
    - d66193b819d2bc2ae40aefcec0da5997b5f9187dd79b8155ec111b16999d18e0 (vllm-bf16-tp1-pp1-feat_lora) [requires >=22 GB/gpu]
- Compatible with system but low memory:
  - a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2 (vllm-bf16-tp1-pp1) [requires >=45 GB/gpu, try --max-model-len=4096 to reduce to >=30 GB/gpu]
- Incompatible with system:
    - 27af459c9caa0f9b34d5e07e5962960df6b0120df2039d06148e0e63595195e5 (vllm-bf16-tp2-pp1)
    - 30d16624c8100d40e6cde3af7f4e4ff6028f776e92efdcf09fcb515ae65662c0 (vllm-bf16-tp4-pp1)
    - 6f888502f35dc189f8c67f3e11174028a4ce42e92868e6a0ca10ef1d84953874 (vllm-bf16-tp8-pp1)

Note

The example above shows output from a vLLM container image. The SGLang container image emits profiles with sglang-* descriptions instead (for example, sglang-bf16-tp1-pp1).

Each profile has:

  • Profile ID: A unique 64-character profile ID.

  • Profile description: A human-readable string constructed by joining tag values with hyphens (for example, vllm-fp16-tp1-pp1).

  • Memory annotation: An estimated VRAM requirement per GPU, shown in brackets (for example, [requires >=18 GB/gpu]).

Memory-Based Profile Classification#

NIM estimates the GPU VRAM required by each profile and classifies it into one of three categories based on the available memory on the system:

Category

Meaning

Action

Compatible

Estimated VRAM fits within available GPU memory

Profile can be selected and deployed

Low memory

Model weights fit, but full context length exceeds available memory

Profile can run with a reduced --max-model-len. The listing includes a suggested value.

Incompatible

Model weights alone exceed available GPU memory

Profile cannot run on this hardware. Consider using a profile with higher tensor parallelism or a quantized precision.

If a profile is classified as low memory, the listing output includes a suggestion. For example:

[requires >=45 GB/gpu, try --max-model-len=4096 to reduce to >=30 GB/gpu]

To apply the suggestion, pass the --max-model-len argument:

docker run --rm -it --gpus=all \
  -p 8000:8000 \
  <nim_llm_image> \
  --max-model-len 4096

Note

Reducing --max-model-len limits the maximum sequence length (input + output tokens) per request. Choose a value that fits your use case.

How Profile Selection Works#

NIM uses a priority-ordered selection chain to decide which profile to use. The chain is evaluated top-to-bottom; the first selector that produces a match wins.

Priority

Selector

Trigger

Description

1 (highest)

Default profile selector

NIM_MODEL_PROFILE="default"

Selects the first hardware-compatible profile using backend priority.

2

Environment-based profile selector

NIM_MODEL_PROFILE=<id-or-name>

Matches an explicit profile by checksum or description.

3

Memory-aware profile selector

(automatic)

Estimates VRAM requirements for each profile and filters out profiles that exceed available GPU memory. Prefers non-LoRA profiles unless LoRA is enabled.

4 (lowest)

Manifest profile selector

(no env var set)

Uses profile_selection_criteria from the manifest to pick a profile automatically based on hardware compatibility (GPU device, VRAM) and parallelism settings.

The memory-aware selector runs automatically as part of the selection chain. It estimates GPU memory requirements for each candidate profile by analyzing model weights, KV cache, activations, and overhead. Profiles that do not fit in available GPU memory are excluded from selection.

Select a Profile#

The method you use to select a profile depends on your requirements and environment. You can allow NIM to pick a compatible profile automatically, or you can specify the exact profile by ID or by description.

Automatic Selection (Default)#

If you do not set NIM_MODEL_PROFILE, NIM automatically selects the best compatible profile from the manifest based on your hardware (GPU device, available VRAM, estimated memory requirements, and parallelism constraints).

To start the container with automatic profile selection, run the container without NIM_MODEL_PROFILE:

docker run --rm -it --gpus=all \
  -p 8000:8000 \
  <nim_llm_image>

Intelligent Default Selection#

Setting NIM_MODEL_PROFILE to "default" triggers intelligent default selection. NIM picks the best compatible profile based on the following criteria:

  1. Hardware compatibility (GPU device, VRAM)

  2. Backend priority

  3. LoRA configuration

To use intelligent default selection, set NIM_MODEL_PROFILE to "default" and start the container:

docker run --rm -it --gpus=all \
  -e NIM_MODEL_PROFILE="default" \
  -p 8000:8000 \
  <nim_llm_image>

Explicit Selection by Profile ID#

Specify the full 64-character profile ID from the manifest of the image you plan to deploy. This selects that exact profile without reconstructing its description. Check the ID again when changing image versions: a profile change can produce a different ID.

To select a profile by ID, set NIM_MODEL_PROFILE to the 64-character profile ID:

docker run --rm -it --gpus=all \
  -e NIM_MODEL_PROFILE="70edb8bb9f8511ce2ea195e3caebcc3c7191dc27fea0c8d4acf9c0d9a69e43cd" \
  -p 8000:8000 \
  <nim_llm_image>

Explicit Selection by Profile Description (Friendly Name)#

If the value of NIM_MODEL_PROFILE is not a valid Profile ID, NIM tries to match it against the profile description — a human-readable string constructed from ordered profile tags.

To select a profile by description, set NIM_MODEL_PROFILE to the profile description:

docker run --rm -it --gpus=all \
  -e NIM_MODEL_PROFILE=vllm-fp16-tp1-pp1 \
  -p 8000:8000 \
  <nim_llm_image>

Note

On an SGLang container image, use the matching sglang- profile description (or its hash) instead, for example sglang-fp16-tp1-pp1.

Tip

Use list-model-profiles to discover the exact profile IDs and descriptions available in your container.

Configuration Precedence#

NIM_MODEL_PROFILE selects a model profile. The following sources override its settings, from highest to lowest priority: direct container arguments, NIM_PASSTHROUGH_ARGS, NIM environment variables that map to backend arguments, and runtime_config.json. For the complete order, refer to Advanced Configuration.

For example, if a profile specifies tp=2 but the user also passes the backend’s tensor-parallelism flag set to 4 (such as the vLLM --tensor-parallel-size 4 argument), the backend launches with TP=4.

Important

When backend arguments override profile settings, the overridden values are resolved before model download. NIM selects and downloads the profile that matches the final resolved configuration, so the downloaded model files always match the launch configuration.

Pass Backend CLI Arguments#

You can also control parallelism and other settings by passing backend CLI arguments directly to the container. This works with both vLLM and SGLang, but the argument names are backend-specific.

To pass a vLLM tensor-parallelism argument, append the backend flag to the container command. The following example uses vLLM arguments:

docker run --rm -it --gpus=all \
  -p 8000:8000 \
  <nim_llm_image> \
  --tensor-parallel-size 2

Common vLLM CLI arguments include the following:

vLLM CLI Argument

Purpose

Default

--tensor-parallel-size

Number of tensor-parallel GPUs

1

--pipeline-parallel-size

Number of pipeline-parallel stages

1

--enable-lora

Enable LoRA adapter support

Disabled

Note

On an SGLang image, the equivalent flag names differ. Refer to the SGLang documentation for the corresponding backend arguments. For a Kubernetes workload, pass the arguments through the container args field. For a NIM Operator deployment that does not set spec.multiNode, use spec.args. Do not use spec.args for a multi-node NIMService; the current LeaderWorkerSet path does not propagate it to the leader or worker NIM containers. The direct-argument interfaces preserve any NIM_PASSTHROUGH_ARGS value provided by the image. For details, refer to Advanced Configuration.

Changes from NIM LLM 1.x#

The following profile selection mechanisms from NIM LLM version 1.x are no longer supported:

Removed Feature

1.x Example

Custom profile selectors

-e NIM_CUSTOM_SELECTOR_CLASSES="my_selector.MyCustomSelector"

LLM-based profile selector (backend priority chain)

Automatic backend priority: TensorRT-LLM > vLLM > SGLang

Tag-based profile selector

-e NIM_TAGS_SELECTOR="llm_engine=vllm,tp=1"

Tip

Use NIM_MODEL_PROFILE with a profile ID or description as a replacement for these deprecated mechanisms. For further guidance, refer to the 1.x Migration Guide.

Next Steps#

After you select a profile, download model artifacts and start serving: