Is this page helpful?

Fine-Tuning with LoRA#

LoRA (Low-Rank Adaptation) lets you serve a base model plus one or more fine-tuned adapters without retraining or rebuilding the full model.

NIM LLM supports two LoRA serving modes:

  • Static LoRA: Adapters are discovered from a directory at startup.

  • Dynamic LoRA: Adapters can be loaded and unloaded while the server is running.

flowchart TD A[NIM container starts] --> B{LoRA mode} B -->|Static| C[Read adapters in NIM_PEFT_SOURCE] C --> D[Load valid adapters at startup] D --> E["/v1/models" includes base + adapters] B -->|Dynamic| F[Enable runtime adapter updates] F --> G[Watcher or API load and unload during runtime] G --> H["/v1/models" updates as adapters change]

Initial LoRA Setup#

Before you configure static LoRA or dynamic LoRA, complete the shared setup in this section. These settings tell NIM LLM where to find adapters and ensure that the deployment uses a LoRA-capable profile.

Configure Adapter Discovery#

Complete the following setup for both static LoRA and dynamic LoRA:

  1. Mount a directory (containing LoRA adapters) to the container.

  2. Set NIM_PEFT_SOURCE to that directory.

  3. Optionally, pass native vLLM LoRA flags.

Expected adapter layout under NIM_PEFT_SOURCE:

/opt/nim/loras/
├── adapter_a/
│   ├── adapter_config.json
│   └── adapter_model.safetensors   # or adapter_model.bin
└── adapter_b/
    ├── adapter_config.json
    └── adapter_model.bin

Only valid, readable adapter directories are loaded.

NIM passes the following LoRA-related flags to vLLM:

  • --enable-lora

  • --max-loras

  • --max-cpu-loras

  • --max-lora-rank

Select a LoRA-Capable Profile#

When available for your model, select a -feat_lora profile so the deployment uses LoRA-compatible runtime settings.

Example:

export NIM_MODEL_PROFILE=vllm-fp16-tp1-pp1-feat_lora

If you are not sure which profiles are available in your deployment, query the model and profile metadata from your environment. Then choose the LoRA-capable profile.

Static LoRA#

After you complete the required LoRA setup, NIM LLM uses static LoRA when NIM_PEFT_REFRESH_INTERVAL is not set. In static LoRA mode, the NIM container discovers adapters in NIM_PEFT_SOURCE during startup and loads valid adapters. If you add, remove, or update adapters after startup, restart the NIM container to apply those changes.

Dynamic LoRA#

Use dynamic LoRA when you need to add or remove adapters without restarting the deployment. You can manage adapters through directory monitoring, runtime API calls, or both.

Configure Dynamic LoRA Updates#

After you complete the shared LoRA setup, set NIM_PEFT_REFRESH_INTERVAL (polling interval in seconds) to enable dynamic LoRA through directory monitoring.

When NIM_PEFT_SOURCE and NIM_PEFT_REFRESH_INTERVAL are set, NIM starts the LoRA watcher and enables runtime LoRA updates for vLLM.

flowchart LR A[NIM_PEFT_SOURCE set] --> B[NIM_PEFT_REFRESH_INTERVAL set] B --> C[Watcher polls adapter directory] C --> D{Detected change} D -->|New adapter| E[Load adapter] D -->|Removed adapter| F[Unload adapter] E --> G["/v1/models" reflects loaded adapter] F --> H["/v1/models" no longer lists adapter]

You can also use vLLM runtime endpoints for manual control:

  • POST /v1/load_lora_adapter

  • POST /v1/unload_lora_adapter

Load and Unload Adapters at Runtime#

To manage adapters through the directory watcher, use the following actions:

  • Load: copy a new adapter folder into NIM_PEFT_SOURCE.

  • Unload: remove an adapter folder from NIM_PEFT_SOURCE.

  • Wait one refresh interval for /v1/models to reflect changes.

To manage adapters through the manual API, use the following actions:

  • Load with POST /v1/load_lora_adapter.

  • Unload with POST /v1/unload_lora_adapter.

If you use both the watcher and the manual API together, an adapter that the API removes but that is still present in the directory can be reloaded by the watcher during the next scan.

Serve Multiple Adapters#

You can serve multiple adapters at the same time, subject to GPU memory limits.

Use /v1/models to discover available adapter IDs, then send the adapter ID in the request model field.

Code Examples#

The following example shows a minimal local workflow for serving a fine-tuned Llama model with LoRA. It includes shared setup, static and dynamic startup commands, model discovery, and an inference request that targets a loaded adapter.

Serve a Fine-Tuned Llama Model with LoRA#

To serve a fine-tuned Llama model with LoRA, complete the following steps:

  1. Set up the environment variables and create the LoRA adapter directory.

    # Common setup
export LOCAL_NIM_CACHE=$PWD/.cache
mkdir -p "$LOCAL_NIM_CACHE"
export NGC_API_KEY=<your_ngc_api_key>
export CUDA_VISIBLE_DEVICES=0
export NIM_MODEL_PROFILE=<lora-capable-profile>

# Prepare adapters
mkdir -p "$PWD/loras"

  2. Start the model by using one of the following options:

    • Use static LoRA loading.

      docker run -it --rm --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -v "$PWD/loras:/opt/nim/loras" \
  -p 8000:8000 \
  -e NGC_API_KEY \
  -e NIM_MODEL_PROFILE \
  -e CUDA_VISIBLE_DEVICES \
  -e NIM_PEFT_SOURCE=/opt/nim/loras \
  <nim-llm-image>

    • Use dynamic LoRA loading with the watcher enabled.

      docker run -it --rm --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -v "$PWD/loras:/opt/nim/loras" \
  -p 8000:8000 \
  -e NGC_API_KEY \
  -e NIM_MODEL_PROFILE \
  -e CUDA_VISIBLE_DEVICES \
  -e NIM_PEFT_SOURCE=/opt/nim/loras \
  -e NIM_PEFT_REFRESH_INTERVAL=10 \
  <nim-llm-image>

  3. Verify that the models loaded.

    curl -s localhost:8000/v1/models | jq

  4. Send an inference request to an adapter.

    curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my_lora_adapter",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 64
  }' | jq

LoRA with NVIDIA Dynamo#

This release documents LoRA with NVIDIA Dynamo in single-container mode. The container runs a Dynamo SmartRouter frontend and one GPU worker.

LoRA in Dynamo mode keeps the same NIM_PEFT_SOURCE static/dynamic experience described above, with one difference in the management plane:

  • Adapters are served through Dynamo’s native LoRA API — POST / GET / DELETE /v1/loras — hosted on the worker’s Dynamo system-status server (the NIM_HEALTH_PORT port), not on the frontend.

  • The legacy NIM endpoints POST /v1/load_lora_adapter and POST /v1/unload_lora_adapter are not available in Dynamo mode.

  • The frontend’s /v1/models still lists the base model plus loaded adapters, and you send inference to an adapter by its id in the request model field as in standard NIM.

Setting NIM_PEFT_SOURCE (and NIM_PEFT_REFRESH_INTERVAL for dynamic mode) starts a watcher that loads and unloads adapters through /v1/loras, so the directory-based workflow is unchanged: drop an adapter folder into NIM_PEFT_SOURCE to load it, remove it to unload.

Adapters appear progressively, not all at once (differs from standard NIM). The worker reports Ready as soon as the base model is loaded; the watcher then loads adapters one at a time through /v1/loras. So in static mode the adapters do not all show up in /v1/models the instant the worker is Ready — they appear progressively, and the time for the full set to register scales with the number and size of the adapters. Standard NIM (vanilla vLLM) instead loads every adapter as part of engine startup, so all adapters are already in /v1/models the moment the server reports ready. In Dynamo mode, poll /v1/models (or the worker’s /v1/loras) until the expected adapter count appears rather than assuming they are all present at Ready.

Single container#

Run the container in single-container Dynamo mode (NIM_DYNAMO_SINGLE=1) with NIM_PEFT_SOURCE, the same as standard NIM:

docker run -it --rm --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -v "$PWD/loras:/opt/nim/loras" \
  -p 8000:8000 -p 9090:9090 \
  -e NGC_API_KEY \
  -e NIM_DYNAMO_SINGLE=1 \
  -e NIM_PEFT_SOURCE=/opt/nim/loras \
  -e NIM_PEFT_REFRESH_INTERVAL=10 \
  <dynamo-enabled-nim-image>

  • OpenAI API and /v1/models are on port 8000 (the frontend).

  • LoRA management (/v1/loras) and worker health (/v1/health/ready) are on port 9090 (the worker system-status server). 9090 is the default for NIM_HEALTH_PORT, so no override is needed.

Inspect or drive adapters directly:

curl -s localhost:9090/v1/loras          # list adapters loaded on the worker
curl -s localhost:8000/v1/models | jq    # base + adapters (for inference)

Omit NIM_PEFT_REFRESH_INTERVAL for static (boot-time) loading.

Multi-worker on Kubernetes#

NIM 3.0 does not publish a LoRA DGD recipe or a public multi-worker Kubernetes LoRA procedure. The runtime-management subsection that follows applies only to the single-container deployment shown previously.

Managing Adapters at Runtime#

In a single-container deployment, the worker’s Dynamo system server, not the frontend, serves the LoRA management API (POST, GET, and DELETE on /v1/loras). Reach the worker on NIM_HEALTH_PORT, which defaults to 9090.

POST /v1/loras loads an adapter on the worker that receives the request.

# On the single-container worker system port:
curl -s localhost:9090/v1/loras                                   # list loaded adapters
curl -s -X POST localhost:9090/v1/loras \
  -H 'Content-Type: application/json' \
  -d '{"lora_name":"my-lora","source":{"uri":"file:///opt/nim/loras/my-lora"}}'
curl -s -X DELETE localhost:9090/v1/loras/my-lora                 # unload

Not supported#

The following configurations are not supported:

  • LoRA with the SGLang backend in Dynamo mode (vLLM only).

  • hf:// adapter sources, unless the worker runtime registers a Hugging Face LoRA source; use s3:// or file:/// otherwise.

Troubleshooting#

Adapter never loads (directory not accessible / cannot read <file> / missing required file)#

Applies to both standard NIM and Dynamo mode. The symptom is a warning that repeats every refresh interval, e.g.:

LoRA adapter '<name>': directory not accessible at <NIM_PEFT_SOURCE>/<name>

and the adapter never appears in /v1/models (nor, in Dynamo mode, in /v1/loras).

This is a file-permission problem. NIM containers run as a non-root user (uid 1000 by default), and the LoRA watcher re-validates every adapter directory under NIM_PEFT_SOURCE on each scan. For NIM’s user it requires:

  • the adapter directory — read + execute (r-x), to list and traverse it;

  • adapter_config.json — read (r);

  • the weights file (adapter_model.safetensors or adapter_model.bin) — read (r).

If any check fails the adapter is skipped and the warning repeats on every poll.

The usual cause is that the adapters were staged by a different user than NIM runs as — commonly a root process that wrote them owner-only (0700). The NGC CLI in particular forces 0700 regardless of umask. Group ownership alone doesn’t help without the group permission bits.

Fix — make the adapters readable by NIM’s user, then the watcher loads them on the next poll (no restart):

chmod -R go+rX <adapter directory>          # dirs -> r-x, files -> r, for group + other

Then stage adapters readable to begin with, so no manual chmod is needed:

  • Docker: the host directory you bind-mount at NIM_PEFT_SOURCE, and the adapter files in it, must be readable by uid 1000 — e.g. chmod -R go+rX ./loras before -v "$PWD/loras:/opt/nim/loras".

  • Kubernetes: run the staging job/pod with NIM’s group and a permissive umask — securityContext: { runAsGroup: 1000, fsGroup: 1000 } plus umask 0022 — or append && chmod -R go+rX <dir> to the download step. Two gotchas: runAsGroup requires an explicit runAsUser (use runAsUser: 0 for a root-based downloader), and fsGroup with fsGroupChangePolicy: OnRootMismatch will not re-chown files added after the volume was first mounted.