Is this page helpful?

Fine-Tuning with LoRA#

LoRA (Low-Rank Adaptation) lets you serve a base model plus one or more fine-tuned adapters without retraining or rebuilding the full model.

NIM LLM and VLM supports two LoRA serving modes on both the vLLM and SGLang backends:

Static LoRA: Adapters are discovered from a directory at startup.
Dynamic LoRA: Adapters can be loaded and unloaded while the server is running.

The following diagram shows how the NIM container selects a LoRA mode and updates /v1/models:

flowchart TD A[NIM container starts] --> B{LoRA mode} B -->|Static| C[Read adapters in NIM_PEFT_SOURCE] C --> D[Load valid adapters at startup] D --> E["/v1/models" includes base + adapters] B -->|Dynamic| F[Enable runtime adapter updates] F --> G[Watcher or API load and unload during runtime] G --> H["/v1/models" updates as adapters change]

Prerequisites#

Before you configure static LoRA or dynamic LoRA, complete the following prerequisites. These settings tell NIM LLM and VLM where to find adapters and select a LoRA-capable profile.

Mount a directory that contains LoRA adapters into the container.
Set NIM_PEFT_SOURCE to that directory.
Optional: Configure LoRA with NIM’s shared arguments:
- --enable-lora
- --max-loras
- --max-cpu-loras
- --max-lora-rank
When a LoRA-capable profile is available for your model, select a -feat_lora profile so the deployment uses LoRA-compatible runtime settings.

Example:
```
export NIM_MODEL_PROFILE=<lora-capable-profile>
```
Copy the exact ID or description from list-model-profiles. The profile must match your image’s backend (vllm-* or sglang-*). For more information, refer to Model Profiles and Selection.

Use the following adapter layout under NIM_PEFT_SOURCE:

/opt/nim/loras/
├── adapter_a/
│   ├── adapter_config.json
│   └── adapter_model.safetensors   # or adapter_model.bin
└── adapter_b/
    ├── adapter_config.json
    └── adapter_model.bin

Only valid, readable adapter directories are loaded.

NIM translates the shared LoRA arguments to the selected backend:

NIM Argument	vLLM Argument	SGLang Argument
`--enable-lora`	`--enable-lora`	`--enable-lora`
`--max-loras`	`--max-loras`	`--max-loras-per-batch`
`--max-cpu-loras`	`--max-cpu-loras`	`--max-loaded-loras`
`--max-lora-rank`	`--max-lora-rank`	`--max-lora-rank`

On SGLang, --max-cpu-loras limits the total number of loaded adapters. If you set it, use a value at least as large as --max-loras and the number of adapters loaded at startup.

Configure Static LoRA#

NIM LLM and VLM uses static LoRA when NIM_PEFT_REFRESH_INTERVAL is not set. In static LoRA mode, the NIM container discovers adapters in NIM_PEFT_SOURCE during startup and loads valid adapters.

To configure static LoRA, complete the following steps:

Start the NIM container without setting NIM_PEFT_REFRESH_INTERVAL.
Query /v1/models to confirm that the response includes the base model and the loaded adapters.
Restart the NIM container after you add, remove, or update adapters.

Configure Dynamic LoRA#

Use dynamic LoRA when you need to add or remove adapters without restarting the deployment. You can manage adapters through directory monitoring, runtime API calls, or both.

To configure dynamic LoRA, complete the following steps:

Set NIM_PEFT_REFRESH_INTERVAL to the polling interval in seconds.
Start the NIM container with NIM_PEFT_SOURCE and NIM_PEFT_REFRESH_INTERVAL set.

When both variables are set, NIM starts the LoRA watcher and enables runtime adapter updates.

The following diagram shows how the watcher loads and unloads adapters:

flowchart LR A[NIM_PEFT_SOURCE set] --> B[NIM_PEFT_REFRESH_INTERVAL set] B --> C[Watcher polls adapter directory] C --> D{Detected change} D -->|New adapter| E[Load adapter] D -->|Removed adapter| F[Unload adapter] E --> G["/v1/models" reflects loaded adapter] F --> H["/v1/models" no longer lists adapter]
Optional: Use the NIM runtime endpoints for manual control on either backend:
- POST /v1/load_lora_adapter
- POST /v1/unload_lora_adapter

On SGLang, the watcher can start with an empty adapter directory. Set --max-lora-rank high enough for the adapters you plan to load; an adapter that exceeds the startup rank limit cannot be loaded later.

Load and Unload Adapters at Runtime#

To load and unload adapters at runtime, complete the following steps:

To load an adapter through the directory watcher, copy a new adapter folder into NIM_PEFT_SOURCE.
To unload an adapter through the directory watcher, remove the adapter folder from NIM_PEFT_SOURCE.
Wait one refresh interval for /v1/models to reflect changes.
Optional: Load with POST /v1/load_lora_adapter or unload with POST /v1/unload_lora_adapter.

If you use both the watcher and the manual API together, an adapter that the API removes but that is still present in the directory can be reloaded by the watcher during the next scan.

Do not give an adapter directory the same name as the served model (NIM_SERVED_MODEL_NAME). Use distinct names so requests can identify the base model and each adapter.

Serve Multiple Adapters#

You can serve multiple adapters at the same time, subject to GPU memory limits.

To serve multiple adapters, complete the following steps:

Query /v1/models to discover available adapter IDs.
Set the request model field for your backend. On vLLM, use the adapter ID, such as my_lora_adapter. On SGLang, use <served-model-name>:<adapter>, such as my-model:my_lora_adapter.

On SGLang, a bare adapter name can select the base model instead. Include the served model name and the colon separator to select the adapter.

Serve a Fine-Tuned Llama Model with LoRA#

The following example shows a minimal local workflow for serving a fine-tuned Llama model with LoRA. It includes shared setup, static and dynamic startup commands, model discovery, and an inference request that targets a loaded adapter.

To serve a fine-tuned Llama model with LoRA, complete the following steps:

Set up the environment variables and create the LoRA adapter directory.

# Common setup
export LOCAL_NIM_CACHE=$PWD/.cache
mkdir -p "$LOCAL_NIM_CACHE"
export NGC_API_KEY=<your_ngc_api_key>
export CUDA_VISIBLE_DEVICES=0
export NIM_MODEL_PROFILE=<lora-capable-profile>
export NIM_SERVED_MODEL_NAME=my-model

# Prepare adapters
mkdir -p "$PWD/loras"

Start the model by using one of the following options:

Use static LoRA loading.

docker run -it --rm --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -v "$PWD/loras:/opt/nim/loras" \
  -p 8000:8000 \
  -e NGC_API_KEY \
  -e NIM_MODEL_PROFILE \
  -e NIM_SERVED_MODEL_NAME \
  -e CUDA_VISIBLE_DEVICES \
  -e NIM_PEFT_SOURCE=/opt/nim/loras \
  <nim-llm-image>

Use dynamic LoRA loading with the watcher enabled.

docker run -it --rm --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -v "$PWD/loras:/opt/nim/loras" \
  -p 8000:8000 \
  -e NGC_API_KEY \
  -e NIM_MODEL_PROFILE \
  -e NIM_SERVED_MODEL_NAME \
  -e CUDA_VISIBLE_DEVICES \
  -e NIM_PEFT_SOURCE=/opt/nim/loras \
  -e NIM_PEFT_REFRESH_INTERVAL=10 \
  <nim-llm-image>

Verify that the models loaded.
```
curl -s localhost:8000/v1/models | jq
```

Send an inference request to an adapter. Select the tab for your backend. These examples use the my_lora_adapter adapter and the served model name my-model configured above.

vLLM

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my_lora_adapter",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 64
  }' | jq

SGLang

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-model:my_lora_adapter",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 64
  }' | jq

Next Steps#

After adapters are listed in /v1/models, continue with the following topics:

Model Profiles and Selection for LoRA-capable profiles.
Environment Variables for NIM_PEFT_SOURCE and NIM_PEFT_REFRESH_INTERVAL.