Is this page helpful?

Fine-Tuning with LoRA#

LoRA (Low-Rank Adaptation) lets you serve a base model plus one or more fine-tuned adapters without retraining or rebuilding the full model.

NIM LLM and VLM supports two LoRA serving modes:

Static LoRA: Adapters are discovered from a directory at startup.
Dynamic LoRA: Adapters can be loaded and unloaded while the server is running.

The following diagram shows how the NIM container selects a LoRA mode and updates /v1/models:

flowchart TD A[NIM container starts] --> B{LoRA mode} B -->|Static| C[Read adapters in NIM_PEFT_SOURCE] C --> D[Load valid adapters at startup] D --> E["/v1/models" includes base + adapters] B -->|Dynamic| F[Enable runtime adapter updates] F --> G[Watcher or API load and unload during runtime] G --> H["/v1/models" updates as adapters change]

Prerequisites#

Before you configure static LoRA or dynamic LoRA, complete the following prerequisites. These settings tell NIM LLM and VLM where to find adapters and select a LoRA-capable profile.

Mount a directory that contains LoRA adapters into the container.
Set NIM_PEFT_SOURCE to that directory.
Optional: Pass native vLLM LoRA flags. NIM passes the following LoRA-related flags to vLLM:
- --enable-lora
- --max-loras
- --max-cpu-loras
- --max-lora-rank
When a LoRA-capable profile is available for your model, select a -feat_lora profile so the deployment uses LoRA-compatible runtime settings.

Example:
```
export NIM_MODEL_PROFILE=vllm-fp16-tp1-pp1-feat_lora
```
If you are not sure which profiles are available in your deployment, query the model and profile metadata from your environment. Then choose the LoRA-capable profile. For more information, refer to Model Profiles and Selection.

Use the following adapter layout under NIM_PEFT_SOURCE:

/opt/nim/loras/
├── adapter_a/
│   ├── adapter_config.json
│   └── adapter_model.safetensors   # or adapter_model.bin
└── adapter_b/
    ├── adapter_config.json
    └── adapter_model.bin

Only valid, readable adapter directories are loaded.

Configure Static LoRA#

NIM LLM and VLM uses static LoRA when NIM_PEFT_REFRESH_INTERVAL is not set. In static LoRA mode, the NIM container discovers adapters in NIM_PEFT_SOURCE during startup and loads valid adapters.

To configure static LoRA, complete the following steps:

Start the NIM container without setting NIM_PEFT_REFRESH_INTERVAL.
Query /v1/models to confirm that the response includes the base model and the loaded adapters.
Restart the NIM container after you add, remove, or update adapters.

Configure Dynamic LoRA#

Use dynamic LoRA when you need to add or remove adapters without restarting the deployment. You can manage adapters through directory monitoring, runtime API calls, or both.

To configure dynamic LoRA, complete the following steps:

Set NIM_PEFT_REFRESH_INTERVAL to the polling interval in seconds.
Start the NIM container with NIM_PEFT_SOURCE and NIM_PEFT_REFRESH_INTERVAL set.

When both variables are set, NIM starts the LoRA watcher and enables runtime LoRA updates for vLLM.

The following diagram shows how the watcher loads and unloads adapters:

flowchart LR A[NIM_PEFT_SOURCE set] --> B[NIM_PEFT_REFRESH_INTERVAL set] B --> C[Watcher polls adapter directory] C --> D{Detected change} D -->|New adapter| E[Load adapter] D -->|Removed adapter| F[Unload adapter] E --> G["/v1/models" reflects loaded adapter] F --> H["/v1/models" no longer lists adapter]
Optional: Use the vLLM runtime endpoints for manual control:
- POST /v1/load_lora_adapter
- POST /v1/unload_lora_adapter

Load and Unload Adapters at Runtime#

To load and unload adapters at runtime, complete the following steps:

To load an adapter through the directory watcher, copy a new adapter folder into NIM_PEFT_SOURCE.
To unload an adapter through the directory watcher, remove the adapter folder from NIM_PEFT_SOURCE.
Wait one refresh interval for /v1/models to reflect changes.
Optional: Load with POST /v1/load_lora_adapter or unload with POST /v1/unload_lora_adapter.

If you use both the watcher and the manual API together, an adapter that the API removes but that is still present in the directory can be reloaded by the watcher during the next scan.

Serve Multiple Adapters#

You can serve multiple adapters at the same time, subject to GPU memory limits.

To serve multiple adapters, complete the following steps:

Query /v1/models to discover available adapter IDs.
Send the adapter ID in the request model field.

Serve a Fine-Tuned Llama Model with LoRA#

The following example shows a minimal local workflow for serving a fine-tuned Llama model with LoRA. It includes shared setup, static and dynamic startup commands, model discovery, and an inference request that targets a loaded adapter.

To serve a fine-tuned Llama model with LoRA, complete the following steps:

Set up the environment variables and create the LoRA adapter directory.

# Common setup
export LOCAL_NIM_CACHE=$PWD/.cache
mkdir -p "$LOCAL_NIM_CACHE"
export NGC_API_KEY=<your_ngc_api_key>
export CUDA_VISIBLE_DEVICES=0
export NIM_MODEL_PROFILE=<lora-capable-profile>

# Prepare adapters
mkdir -p "$PWD/loras"

Start the model by using one of the following options:

Use static LoRA loading.

docker run -it --rm --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -v "$PWD/loras:/opt/nim/loras" \
  -p 8000:8000 \
  -e NGC_API_KEY \
  -e NIM_MODEL_PROFILE \
  -e CUDA_VISIBLE_DEVICES \
  -e NIM_PEFT_SOURCE=/opt/nim/loras \
  <nim-llm-image>

Use dynamic LoRA loading with the watcher enabled.

docker run -it --rm --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -v "$PWD/loras:/opt/nim/loras" \
  -p 8000:8000 \
  -e NGC_API_KEY \
  -e NIM_MODEL_PROFILE \
  -e CUDA_VISIBLE_DEVICES \
  -e NIM_PEFT_SOURCE=/opt/nim/loras \
  -e NIM_PEFT_REFRESH_INTERVAL=10 \
  <nim-llm-image>

Verify that the models loaded.
```
curl -s localhost:8000/v1/models | jq
```

Send an inference request to an adapter.

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my_lora_adapter",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 64
  }' | jq

Next Steps#

After adapters are listed in /v1/models, continue with the following topics:

Model Profiles and Selection for LoRA-capable profiles.
Environment Variables for NIM_PEFT_SOURCE and NIM_PEFT_REFRESH_INTERVAL.