About Model-Free NIM

Model-free NIM lets you run any supported model in NIM LLM and VLM without a model-specific container image.

By default, NIM containers ship with a built-in model manifest that defines which model to serve. In model-free mode, you point a generic NIM container at any supported model, such as a Hugging Face repository, an S3 bucket, or a local directory. NIM then generates a manifest at startup and serves that model.

How It Works

When you start a model-free container, you supply a model location with NIM_MODEL_PATH or with the backend positional argument. NIM generates a runtime manifest for that source, selects a profile, and serves the model.

If you do not set a served name, NIM derives one from the model source by removing the URI scheme and any trailing :<version>, then replacing any : that remains. That derived name is the value to use in the model field of inference requests, and it is what /v1/models reports.

For air-gap redeploys, a network-connected start saves the generated manifest in the container cache (NIM_CACHE_PATH). If manifest generation fails during an offline restart, NIM falls back to that cached manifest. For the full air-gap workflow, refer to Air-Gap Deployment.

Profile listing, URI schemes, and download behavior are documented in the configuration sections that follow. For profile selection details, refer to Model Profiles and Selection. For source-specific download steps, refer to Model Download.

Related Use Cases

Model-free NIM supports the following capabilities:

• Flexible single-container deployments: One container image can pass security review and serve any supported model.

• Day-zero model support: You can serve newly released models without waiting for a model-specific NIM container.

• Custom and fine-tuned models: You can serve your own models from any supported source.

Limitations and Considerations

The following limits apply when you choose a model-free NIM container.

• Unsupported architectures: The backend in the container (vLLM or SGLang) must support the model architecture.

• Remote URI first deploy: A remote NIM_MODEL_PATH (for example, ngc:// or hf://) requires network access and credentials on the first deployment so NIM can generate and cache the runtime manifest.

Note

Regardless of where the model is hosted, if its architecture is unsupported by the inference backend in your container (vLLM or SGLang), it is also unsupported by NIM.

Configuring the Model-Free NIM Container Image

The latest model-free NIM container images are in the NGC Catalog:

• SGLang Model-Free NIM

• vLLM Model-Free NIM

Set NIM_LLM_IMAGE to the image name and tag from the catalog page. Model download commands use this value.

export NIM_LLM_IMAGE="nvcr.io/nim/nvidia/<nim-model-free-image-tag>:<version>"

Configuring the Model

Supply the model path and name using either of these methods. If both are provided, the CLI positional argument takes precedence.

NIM_MODEL_PATH Environment Variable

Use NIM_MODEL_PATH to point the container to the model at runtime:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e HF_TOKEN=<yourtoken> \
  -p 8000:8000 \
  ${NIM_LLM_IMAGE}

Model Path as a Container Argument

Pass the model path as the first positional argument to the container:

docker run --gpus=all \
  -e HF_TOKEN=<yourtoken> \
  -p 8000:8000 \
  ${NIM_LLM_IMAGE} \
  hf://meta-llama/Llama-3.1-8B-Instruct

Default Model Name

If you do not set a name, NIM derives one from the model source by removing the URI scheme and any trailing :<version>, then replacing any : still left. The result is the name to use in the model field of inference requests, and it is what /v1/models reports.

Model source	Derived model name
hf://meta-llama/Llama-3.1-8B-Instruct	meta-llama/Llama-3.1-8B-Instruct
ngc://nim/meta/llama-3_1-8b-instruct:hf-0722	nim/meta/llama-3_1-8b-instruct
s3://my-bucket/my-org/my-model	my-bucket/my-org/my-model
gs://my-bucket/my-org/my-model	my-bucket/my-org/my-model
/opt/nim/models/my-model	/opt/nim/models/my-model
s3://my-bucket/team:v2/my-model	my-bucket/team_v2/my-model

The derived name is the same regardless of which inference backend the image uses.

Note

A derived name never contains :. If the model source has one, NIM replaces it, because some backends read : in the model field as a base-model:lora-adapter separator and would reject every request.

A name you set yourself is used verbatim and is not adjusted. On a backend that reserves :, setting one that contains it fails at startup with a message naming the value, rather than starting and rejecting every request.

NIM_SERVED_MODEL_NAME Environment Variable

Use NIM_SERVED_MODEL_NAME to set the model name used in inference requests:

docker run --gpus=all \
  -e NIM_SERVED_MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct \
  ...

CLI Argument

Use the CLI argument --served-model-name to pass the model name used in inference requests.

docker run --gpus=all \
  ...
  --served-model-name "meta-llama/Llama-3.1-8B-Instruct"

Supported Model Sources

The following table lists the model sources supported by model-free NIM, along with the required URI prefix and authentication method.

Prefix	Source	Example	Authentication
hf://	Hugging Face Hub	hf://meta-llama/Llama-3.1-8B-Instruct	HF_TOKEN
ngc://	NVIDIA NGC	ngc://nim/meta/llama-3.3-70b-instruct:hf	NGC_API_KEY
s3://	AWS S3 / S3-compatible	s3://my-bucket/my-org/my-model	AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (additional S3 variables)
modelscope://	ModelScope Hub	modelscope://LLM-Research/Llama-3.2-1B-Instruct:d3e55134	MODELSCOPE_API_TOKEN
gs://	Google Cloud Storage	gs://my-bucket/my-org/my-model	GOOGLE_APPLICATION_CREDENTIALS (or ADC)
(absolute path)	Local directory	/mnt/models/my-llama	None

For details on downloading models from each source (including URI formats, proxy configuration, and cache management), refer to Model Download.

Configuring Deployment Options

Model-free NIM generates profiles for combinations of tensor parallelism (TP), pipeline parallelism (PP), and LoRA. To select a deployment configuration, use either of the following methods. If both are provided, backend CLI arguments take precedence.

NIM_MODEL_PROFILE Environment Variable

Run list-model-profiles to see available profiles, then select one:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e NIM_MODEL_PROFILE=<profile-id-or-description> \
  -e HF_TOKEN=<yourtoken> \
  -p 8000:8000 \
  ${NIM_LLM_IMAGE}

Generated descriptions take the form <engine>-tp<N>-pp<M>[-feat_lora]-<hash>, where <engine> is vllm or sglang depending on the image (for example vllm-tp2-pp1-<hash> or sglang-tp2-pp1-<hash>). Copy the exact ID or description from list-model-profiles output.

Container Arguments

Use NIM environment variables or canonical CLI flags to configure settings shared by both inference engines:

docker run --gpus=all \
  -e NIM_TENSOR_PARALLEL_SIZE=2 \
  -e HF_TOKEN=<yourtoken> \
  -p 8000:8000 \
  ${NIM_LLM_IMAGE} \
  hf://meta-llama/Llama-3.1-8B-Instruct

Alternatively, pass the canonical flag --tensor-parallel-size 2 after the model path:

docker run --gpus=all \
  -e HF_TOKEN=<yourtoken> \
  -p 8000:8000 \
  ${NIM_LLM_IMAGE} \
  hf://meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 2

NIM translates canonical flags into the selected inference engine’s arguments. Common settings include:

Setting	NIM Environment Variable	Canonical Flag
Tensor parallelism	NIM_TENSOR_PARALLEL_SIZE	--tensor-parallel-size
Pipeline parallelism	NIM_PIPELINE_PARALLEL_SIZE	--pipeline-parallel-size

For LoRA profile selection, refer to Fine-Tuning with LoRA. For backend-specific arguments and configuration precedence, refer to Advanced Configuration.

Listing Profiles

Use the list-model-profiles command to view the profiles generated for a given model:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e HF_TOKEN=<yourtoken> \
  ${NIM_LLM_IMAGE} \
  list-model-profiles

For more details on profile selection, refer to Model Profiles and Selection.

S3-Specific Environment Variables

The following environment variables authenticate and configure S3 and S3-compatible endpoints:

Variable	Required	Purpose
AWS_ACCESS_KEY_ID	Yes	AWS access key
AWS_SECRET_ACCESS_KEY	Yes	AWS secret key
AWS_REGION or AWS_DEFAULT_REGION	Yes	AWS region (for example, us-east-1)
AWS_ENDPOINT_URL	Only for S3-compatible	Custom endpoint (for example, http://localhost:9000 for MinIO)
AWS_S3_USE_PATH_STYLE	Only for S3-compatible	Set to true for MinIO or path-style S3 endpoints

For complete authentication and configuration details for each source, refer to Model Download.

Air-Gap Deployments

How model-free NIM behaves in an air-gap environment depends on the type of NIM_MODEL_PATH value you use.

Local Path (/abs/path/to/model)

NIM generates the manifest from the local directory with no network access or credentials. This is the simplest air-gap workflow — ensure the model directory is pre-staged and mounted into the container.

Remote URI (ngc://, hf://, s3://, and so on)

NIM generates a runtime manifest from the URI and saves a copy in NIM_CACHE_PATH. On restart, NIM attempts to generate a fresh manifest. If that fails, such as when the registry is unavailable, NIM falls back to the cached manifest. For offline use, prepare the cache with the same model and container image and include all required model assets.

This enables the following workflow for air-gap redeployment:

1. First deploy (network-connected): Run the container with credentials and the remote URI. NIM downloads the model, generates the manifest, and saves a copy to NIM_CACHE_PATH.

2. Transfer: Ensure the NIM cache is on a PVC or persistent volume that survives pod restarts, or transfer the cache directory to the air-gap environment.

3. Redeploy (air-gapped): Mount the same cache volume. When manifest generation fails offline, NIM uses the cached manifest and model assets. Do not pass registry credentials.

Tip

Use a separate cache for each model and container image. Before changing either in an air-gapped deployment, prepare and transfer a matching cache on a network-connected host.

For a complete walkthrough, refer to Air-Gap Deployment.

Examples

Use the following examples to run model-free NIM with different model sources and profile selection methods. Each example lists the commands in order.

Hugging Face Model with Profile Selection

Use this example to inspect available profiles for a Hugging Face model, select a profile, or override tensor parallelism.

1. Set the model path:

   export MODEL=hf://meta-llama/Llama-3.2-1B
   

2. List the available profiles:

   docker run --gpus=all \
     -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
     -e HF_TOKEN \
     -e NIM_MODEL_PATH=$MODEL \
     ${NIM_LLM_IMAGE} \
     list-model-profiles
   

Example output:

- Compatible with system and runnable:
  - c214460d2ad7a379660126062912d2aeecaa74a3ce14ab9966cd135de49a73f2 (vllm-tp1-pp1-0bdd169fb413e457cef3feda64108b085f73d16b6252a860e5e9ee85f533de73) [requires >=13 GB/gpu]
- With LoRA support:
  - 289b03eb8c26104f416dd0a1055004e31fd9e4b0f84fe2e59754a3ceb710976a (vllm-tp1-pp1-feat_lora-0bdd169fb413e457cef3feda64108b085f73d16b6252a860e5e9ee85f533de73) [requires >=13 GB/gpu]

Note

The example output above is from a vLLM model-free image. On an SGLang model-free image, the profile descriptions are prefixed sglang- instead (for example, sglang-tp1-pp1-...).

3. Start NIM by using one of the following methods:

• Select a profile with NIM_MODEL_PROFILE, using an ID or description from your image’s list-model-profiles output:

  docker run --gpus=all \
    -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
    -e HF_TOKEN \
    -e NIM_MODEL_PROFILE=<profile-id-or-description> \
    -e NIM_MODEL_PATH=$MODEL \
    -p 8000:8000 \
    ${NIM_LLM_IMAGE}
  

• Override tensor parallelism with NIM_TENSOR_PARALLEL_SIZE=2 or the canonical flag --tensor-parallel-size 2:

  docker run --gpus=all \
    -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
    -e HF_TOKEN \
    -p 8000:8000 \
    ${NIM_LLM_IMAGE} \
    $MODEL --tensor-parallel-size 2
  

Model Hosted in S3 Using Default Profile from Automatic Profile Selection

Use this example to serve a model from S3 and let NIM select the default compatible profile automatically.

1. Set the model path:

   export MODEL=s3://my-bucket/my-org/my-fine-tuned-model
   

2. Start NIM with the required S3 credentials:

   docker run --gpus=all \
     -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
     -e NIM_MODEL_PATH=$MODEL \
     -e AWS_ACCESS_KEY_ID=<key> \
     -e AWS_SECRET_ACCESS_KEY=<secret> \
     -e AWS_REGION=us-east-1 \
     -p 8000:8000 \
     ${NIM_LLM_IMAGE}
   

Local Model with TP=8 Profile Specified

Use this example to serve a local model and select a TP=8 profile explicitly.

1. Set the model path:

   export MODEL=/mnt/models/my-120b-model
   

2. Start NIM with the selected profile:

   docker run --gpus=all \
     -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
     -v /mnt/models:/mnt/models \
     -e NIM_MODEL_PROFILE=<tp8_profile> \
     -e NIM_MODEL_PATH=$MODEL \
     -p 8000:8000 \
     ${NIM_LLM_IMAGE}