LoRA Adapters | NVIDIA Dynamo Documentation

What Are LoRA Adapters

LoRA (Low-Rank Adaptation) serves specialized model variants without duplicating full base weights. In Dynamo you can load adapters at runtime, route inference to workers that hold them, and manage them declaratively on Kubernetes. It provides:

Dynamic loading: Load and unload adapters without restarting workers
Multiple sources: file://, s3://, or hf:// URIs
Automatic caching: Downloaded adapters are cached under DYN_LORA_PATH
Discovery: Loaded adapters appear in /v1/models
KV-aware routing: Route requests to workers with the matching adapter and cached prefix blocks
Kubernetes native: Declarative loading through the DynamoModel CRD

Architecture

Rust core (lib/llm/src/lora/): Downloading, caching, and validation
Python manager (components/src/dynamo/common/lora/): Custom source support
Worker handlers (components/src/dynamo/vllm/handlers.py): Load/unload API and inference integration

Serve a LoRA Adapter

Kubernetes

Local

Prerequisites

A Kubernetes cluster with the Dynamo Platform installed and a vLLM runtime image
A LoRA adapter compatible with your base model
For s3:// sources: AWS credentials in a Kubernetes Secret

Full manifests and MinIO setup: Kubernetes LoRA deployment example.

Deploy a LoRA-enabled worker

Enable LoRA on the worker: set LoRA and system API environment variables, and pass vLLM LoRA flags in args:. DYN_SYSTEM_ENABLED and DYN_SYSTEM_PORT expose load/unload on the worker system port.

Adapted from examples/backends/vllm/deploy/v1beta1/agg_lora.yaml:

1 apiVersion: nvidia.com/v1beta1
2 kind: DynamoGraphDeployment
3 metadata:
4   name: vllm-agg-lora
5 spec:
6   components:
7   - name: Frontend
8     type: frontend
9     replicas: 1
10     podTemplate:
11       spec:
12         containers:
13         - name: main
14           image: ${RUNTIME_IMAGE}
15   - name: VllmDecodeWorker
16     type: decode
17     replicas: 1
18     podTemplate:
19       spec:
20         containers:
21         - name: main
22           image: ${RUNTIME_IMAGE}
23           workingDir: /workspace/examples/backends/vllm
24           envFrom:
25           - secretRef:
26               name: hf-token-secret
27           env:
28           - name: DYN_LORA_ENABLED
29             value: "true"
30           - name: DYN_LORA_PATH
31             value: /tmp/dynamo_loras
32           - name: DYN_SYSTEM_ENABLED
33             value: "true"
34           - name: DYN_SYSTEM_PORT
35             value: "9090"
36           command:
37           - python3
38           - -m
39           - dynamo.vllm
40           args:
41           - --model
42           - Qwen/Qwen3-0.6B
43           - --enable-lora
44           - --max-lora-rank
45           - "64"
46           - --enforce-eager
47           resources:
48             limits:
49               nvidia.com/gpu: "1"

Apply and wait for readiness:

$ kubectl apply -f vllm-agg-lora.yaml -n ${NAMESPACE}
$ kubectl wait --for=condition=Ready dynamographdeployment/vllm-agg-lora \
>   -n ${NAMESPACE} --timeout=600s

Worker environment variables

Variable	Description	Default
`DYN_LORA_ENABLED`	Enable LoRA adapter support	`false`
`DYN_LORA_PATH`	Local cache directory for downloaded LoRAs	`~/.cache/dynamo_loras`
`DYN_SYSTEM_ENABLED`	Expose worker load/unload API	`false`
`DYN_SYSTEM_PORT`	System API port	—
`AWS_ACCESS_KEY_ID`	S3 access key (for `s3://` URIs)	—
`AWS_SECRET_ACCESS_KEY`	S3 secret key (for `s3://` URIs)	—
`AWS_ENDPOINT`	Custom S3 endpoint (MinIO, etc.)	—
`AWS_REGION`	AWS region	`us-east-1`
`AWS_ALLOW_HTTP`	Allow HTTP (non-TLS) connections	`false`

vLLM worker arguments

Argument	Description
`--enable-lora`	Enable LoRA adapter support in vLLM
`--max-lora-rank`	Maximum LoRA rank (must be >= your adapter’s rank)
`--max-loras`	Maximum number of LoRAs loaded simultaneously

Set --max-lora-rank to at least your adapter’s rank. A lower value causes load failures.

S3 or MinIO credentials on the worker

Store secret keys in a Kubernetes Secret and set non-secret endpoint/region inline:

1           env:
2           - name: DYN_LORA_ENABLED
3             value: "true"
4           - name: AWS_ENDPOINT
5             value: http://minio:9000        # for MinIO; omit for AWS S3
6           - name: AWS_REGION
7             value: us-east-1
8           - name: AWS_ALLOW_HTTP
9             value: "true"                    # MinIO/non-TLS only
10           - name: AWS_ACCESS_KEY_ID
11             valueFrom:
12               secretKeyRef:
13                 name: minio-secret
14                 key: AWS_ACCESS_KEY_ID
15           - name: AWS_SECRET_ACCESS_KEY
16             valueFrom:
17               secretKeyRef:
18                 name: minio-secret
19                 key: AWS_SECRET_ACCESS_KEY

Load an adapter

Use a DynamoModel CRD for declarative, cluster-native loading. It discovers worker endpoints for baseModelName, creates a Service, and calls the load API on each pod.

1 apiVersion: nvidia.com/v1alpha1
2 kind: DynamoModel
3 metadata:
4   name: customer-support-lora
5   namespace: dynamo-system
6 spec:
7   modelName: customer-support-adapter-v1
8   baseModelName: Qwen/Qwen3-0.6B  # Must match the worker --model value
9   modelType: lora
10   source:
11     uri: s3://my-models-bucket/loras/customer-support/v1

Verify readiness:

$ kubectl get dynamomodel customer-support-lora
$ # NAME                    TOTAL   READY   AGE
$ # customer-support-lora   2       2       30s

See Managing Models with DynamoModel for the full CRD workflow.

Load imperatively with the worker system API

Port-forward the worker system port and POST to /v1/loras:

$ kubectl port-forward svc/vllm-agg-lora-vllmdecodeworker 9090:9090 -n ${NAMESPACE}

$ curl -X POST http://localhost:9090/v1/loras \
>   -H "Content-Type: application/json" \
>   -d '{
>     "lora_name": "customer-support-lora",
>     "source": {
>       "uri": "s3://my-loras/customer-support-v1"
>     }
>   }'

List loaded adapters with GET /v1/loras. Unload with DELETE /v1/loras/{lora_name}.

Run inference

Port-forward the Frontend and set the request model field to the adapter name (lora_name or DynamoModel modelName):

$ kubectl port-forward svc/vllm-agg-lora-frontend 8000:8000 -n ${NAMESPACE}

$ curl -X POST http://localhost:8000/v1/chat/completions \
>   -H "Content-Type: application/json" \
>   -d '{
>     "model": "customer-support-lora",
>     "messages": [{"role": "user", "content": "Hello!"}],
>     "max_tokens": 100
>   }'

The model field is case-sensitive and must match the loaded adapter name exactly. For disaggregated serving, load the adapter on both prefill and decode workers.

KV Cache-Aware LoRA Routing

How LoRA-aware KV routing works

When KV-aware routing is enabled, the router accounts for LoRA adapter identity when computing block hashes:

Distinct hash spaces per adapter: Blocks cached under adapter A are never confused with adapter B or the base model, even when token sequences match. The adapter name is mixed into the LocalBlockHash computation.
Prefix sharing within the same adapter: Requests targeting the same LoRA adapter reuse KV prefix blocks like base-model requests.
No extra configuration: The LoRA name propagates through KV events (BlockStored) from the engine to the router. The router uses the lora_name field to route requests to workers with matching cached blocks.

This works across the publisher pipeline, the KV consolidator, and the routing query path.

For a local two-worker demo with KV-aware routing, run agg_lora_router.sh and load the adapter on both worker system ports.

Troubleshooting

LoRA fails to load

Check S3 connectivity:

$ aws --endpoint-url=$AWS_ENDPOINT s3 ls s3://my-loras/ --recursive

Check the cache directory:

$ ls -la ~/.cache/dynamo_loras/
$ # or the path set in DYN_LORA_PATH

Check worker logs:

$ kubectl logs deployment/my-worker | grep -i lora

Confirm --max-lora-rank is at least your adapter’s rank.

Model not found after loading

Verify the LoRA name matches exactly (case-sensitive)
List loaded adapters: curl http://localhost:9090/v1/loras (port-forward the worker’s DYN_SYSTEM_PORT first on Kubernetes)
Check worker logs for discovery registration errors

Inference returns the base model response

Confirm the request model field matches the loaded lora_name
Verify the adapter is loaded on the worker handling the request
For disaggregated serving, load the adapter on both prefill and decode workers

Backend Support

Backend	Status	Notes
vLLM	Yes	Full support including KV-aware routing
SGLang	WIP	Support is in progress
TensorRT-LLM	No	Not yet supported

See the feature support matrix for full compatibility details.