LoRA Adapters | NVIDIA Dynamo Documentation

What Are LoRA Adapters

Low-Rank Adaptation (LoRA) serves specialized model variants without duplicating full base weights. Dynamo supports dynamic LoRA lifecycle management with vLLM and SGLang, with different validation levels for each backend. It provides:

Dynamic loading: Load and unload adapters without restarting workers
Multiple sources: file://, s3://, or hf:// URIs
Automatic caching: Downloaded adapters are cached under DYN_LORA_PATH
Discovery: Loaded adapters appear in /v1/models
KV-aware routing: Route requests to workers with the matching adapter and cached prefix blocks; validated with vLLM
Kubernetes native: Manage adapters declaratively through the DynamoModel CRD; documented for vLLM

Backend Support

Backend	Status	Support Scope
vLLM	Supported	Dynamic load/unload for aggregated and disaggregated workers; aggregated adapter-aware KV routing
SGLang	Experimental	Dynamic load/unload and aggregated inference; disaggregated serving and feature pairings are not end-to-end validated
TensorRT-LLM	Not supported	—

See the feature support matrix for the backend and interaction matrices.

Architecture

Rust core (lib/llm/src/lora/): Downloading, caching, and validation
Python manager (components/src/dynamo/common/lora/): Custom source support
Worker handlers (components/src/dynamo/vllm/handlers.py and components/src/dynamo/sglang/request_handlers/handler_base.py): Backend load/unload and inference integration

Serve a LoRA Adapter

The following Kubernetes and local workflows use vLLM. For the aggregated SGLang workflow, see Serve a LoRA Adapter with SGLang.

Keep DYN_SYSTEM_PORT on a trusted administrative network. The system API retrieves and loads model artifacts from configured URI sources; do not expose it to untrusted clients.

Kubernetes

Local

Prerequisites

A Kubernetes cluster with the Dynamo Platform installed and a vLLM runtime image
A LoRA adapter compatible with your base model
For s3:// sources: AWS credentials in a Kubernetes Secret

Full manifests and MinIO setup: Kubernetes LoRA deployment example.

Deploy a LoRA-enabled worker

Enable LoRA on the worker: set LoRA and system API environment variables, and pass vLLM LoRA flags in args:. DYN_SYSTEM_ENABLED and DYN_SYSTEM_PORT expose load/unload on the worker system port.

Adapted from the agg_lora.yaml example:

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
  name: vllm-agg-lora
spec:
  components:
  - name: Frontend
    type: frontend
    replicas: 1
    podTemplate:
      spec:
        containers:
        - name: main
          image: ${RUNTIME_IMAGE}
  - name: worker
    type: worker
    replicas: 1
    podTemplate:
      spec:
        containers:
        - name: main
          image: ${RUNTIME_IMAGE}
          workingDir: /workspace/examples/backends/vllm
          envFrom:
          - secretRef:
              name: hf-token-secret
          env:
          - name: DYN_LORA_ENABLED
            value: "true"
          - name: DYN_LORA_PATH
            value: /tmp/dynamo_loras
          - name: DYN_SYSTEM_ENABLED
            value: "true"
          - name: DYN_SYSTEM_PORT
            value: "9090"
          command:
          - python3
          - -m
          - dynamo.vllm
          args:
          - --model
          - Qwen/Qwen3-0.6B
          - --enable-lora
          - --max-lora-rank
          - "64"
          - --enforce-eager
          resources:
            limits:
              nvidia.com/gpu: "1"

Apply and wait for readiness:

kubectl apply -f vllm-agg-lora.yaml -n ${NAMESPACE}
kubectl wait --for=condition=Ready dynamographdeployment/vllm-agg-lora \
  -n ${NAMESPACE} --timeout=600s

Worker environment variables

Variable	Description	Default
`DYN_LORA_ENABLED`	Enable LoRA adapter support	`false`
`DYN_LORA_PATH`	Local cache directory for downloaded LoRAs	`~/.cache/dynamo_loras`
`DYN_SYSTEM_ENABLED`	Expose worker load/unload API	`false`
`DYN_SYSTEM_PORT`	System API port	—
`AWS_ACCESS_KEY_ID`	S3 access key from the environment credential provider	—
`AWS_SECRET_ACCESS_KEY`	S3 secret key from the environment credential provider	—
`AWS_PROFILE`	Shared AWS configuration and credentials profile	`default`
`AWS_SHARED_CREDENTIALS_FILE`	Shared credentials file path	`~/.aws/credentials`
`AWS_CONFIG_FILE`	Shared configuration file path	`~/.aws/config`
`AWS_ENDPOINT_URL_S3`	Service-specific custom S3 endpoint	—
`AWS_ENDPOINT_URL`	Generic custom AWS endpoint	—
`AWS_ENDPOINT`	Legacy custom S3 endpoint fallback	—
`AWS_REGION`	AWS region	`us-east-1`
`AWS_ALLOW_HTTP`	Allow HTTP (non-TLS) connections	`false`
`AWS_VIRTUAL_HOSTED_STYLE_REQUEST`	Use bucket-qualified virtual-hosted requests	`false`

For s3:// sources, Dynamo uses the standard AWS credential provider chain, including environment variables, shared configuration and credentials files, web identity, container credentials, and Amazon EC2 instance metadata. Endpoint precedence is AWS_ENDPOINT_URL_S3, the active profile’s S3 service endpoint, AWS_ENDPOINT_URL, the active profile’s generic endpoint_url, then the legacy AWS_ENDPOINT fallback.

vLLM worker arguments

Argument	Description
`--enable-lora`	Enable LoRA adapter support in vLLM
`--max-lora-rank`	Maximum LoRA rank (must be >= your adapter’s rank)
`--max-loras`	Maximum number of LoRAs loaded simultaneously

Set --max-lora-rank to at least your adapter’s rank. A lower value causes load failures.

S3 or MinIO credentials on the worker

Mount shared AWS files and set AWS_PROFILE, or store environment credentials in a Kubernetes Secret. Set non-secret endpoint and region values inline:

          env:
          - name: DYN_LORA_ENABLED
            value: "true"
          - name: AWS_ENDPOINT_URL
            value: http://minio:9000        # for MinIO; omit for AWS S3
          - name: AWS_REGION
            value: us-east-1
          - name: AWS_ALLOW_HTTP
            value: "true"                    # MinIO/non-TLS only
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: minio-secret
                key: AWS_ACCESS_KEY_ID
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: minio-secret
                key: AWS_SECRET_ACCESS_KEY

Load an adapter

Use a DynamoModel CRD for declarative, cluster-native loading. It discovers worker endpoints for baseModelName, creates a Service, and calls the load API on each pod.

apiVersion: nvidia.com/v1alpha1
kind: DynamoModel
metadata:
  name: customer-support-lora
  namespace: dynamo-system
spec:
  modelName: customer-support-adapter-v1
  baseModelName: Qwen/Qwen3-0.6B  # Must match the worker --model value
  modelType: lora
  source:
    uri: s3://my-models-bucket/loras/customer-support/v1

Verify readiness:

kubectl get dynamomodel customer-support-lora
# NAME                    TOTAL   READY   AGE
# customer-support-lora   2       2       30s

See the DynamoModel API reference for the CRD fields.

Load imperatively with the worker system API

Port-forward the worker system port and POST to /v1/loras:

kubectl port-forward svc/vllm-agg-lora-worker 9090:9090 -n ${NAMESPACE}

curl -X POST http://localhost:9090/v1/loras \
  -H "Content-Type: application/json" \
  -d '{
    "lora_name": "customer-support-lora",
    "source": {
      "uri": "s3://my-loras/customer-support-v1"
    }
  }'

List loaded adapters with GET /v1/loras. Unload with DELETE /v1/loras/{lora_name}.

Run inference

Port-forward the Frontend and set the request model field to the adapter name (lora_name or DynamoModel modelName):

kubectl port-forward svc/vllm-agg-lora-frontend 8000:8000 -n ${NAMESPACE}

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "customer-support-lora",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

The model field is case-sensitive and must match the loaded adapter name exactly. For vLLM disaggregated serving, load the adapter on both prefill and decode workers.

Serve a LoRA Adapter with SGLang

Experimental. The repository validates dynamic loading, discovery, and inference for aggregated SGLang workers. Unloading is implemented but not exercised by an end-to-end test. Prefill and decode lifecycle registration has unit coverage, but disaggregated SGLang LoRA and feature pairings such as KV-aware routing are not end-to-end validated. The Kubernetes workflow above and the adapter-aware routing demo below are vLLM-specific.

Prepare the adapter

Start MinIO and upload the example adapter:

cd examples/backends/sglang/launch/lora
./setup_minio.sh

See the SGLang LoRA example for the scripts and configurable environment variables.

Launch aggregated serving

Start the Dynamo frontend and a LoRA-enabled SGLang worker:

./agg_lora.sh

The script starts dynamo.sglang with --enable-lora, --max-lora-rank 64, and --lora-target-modules all. It also sets DYN_LORA_ENABLED, DYN_LORA_PATH, and the worker system port.

Load and query the adapter

Load the adapter through the worker system API, then address it by name in the OpenAI-compatible request:

curl -X POST http://localhost:8081/v1/loras \
  -H "Content-Type: application/json" \
  -d '{
    "lora_name": "codelion/Qwen3-0.6B-accuracy-recovery-lora",
    "source": {
      "uri": "s3://my-loras/codelion/Qwen3-0.6B-accuracy-recovery-lora"
    }
  }'

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "codelion/Qwen3-0.6B-accuracy-recovery-lora",
    "messages": [{"role": "user", "content": "What is deep learning?"}],
    "max_tokens": 100
  }'

List loaded adapters with GET /v1/loras. Unload with DELETE /v1/loras/{lora_name}.

KV Cache-Aware LoRA Routing

This section describes the validated vLLM path. KV-aware routing with SGLang LoRA remains experimental because the combined path is not end-to-end validated.

With DYN_LORA_ENABLED, only KV, random, and round-robin routing are LoRA-aware. Direct, power-of-two, least-loaded, and device-aware-weighted modes fail startup. Session affinity with LoRA is supported only in KV mode; random and round-robin plus affinity are rejected.

How LoRA-aware KV routing works

When KV-aware routing is enabled, the router accounts for LoRA adapter identity when computing block hashes:

Distinct hash spaces per adapter: Blocks cached under adapter A are never confused with adapter B or the base model, even when token sequences match. The adapter name is mixed into the LocalBlockHash computation.
Prefix sharing within the same adapter: Requests targeting the same LoRA adapter reuse KV prefix blocks like base-model requests.
No extra configuration: The LoRA name propagates through KV events (BlockStored) from the engine to the router. The router uses the lora_name field to route requests to workers with matching cached blocks.

This works across the publisher pipeline, the KV consolidator, and the routing query path.

For a local two-worker demo with KV-aware routing, run agg_lora_router.sh and load the adapter on both worker system ports.

Troubleshooting

LoRA fails to load

Check S3 connectivity:

aws s3 ls s3://my-loras/ --recursive

Check the cache directory:

ls -la ~/.cache/dynamo_loras/
# or the path set in DYN_LORA_PATH

Check worker logs:

kubectl logs deployment/my-worker | grep -i lora

Confirm --max-lora-rank is at least your adapter’s rank.

Model not found after loading

Verify the LoRA name matches exactly (case-sensitive)
List loaded adapters: curl http://localhost:9090/v1/loras (port-forward the worker’s DYN_SYSTEM_PORT first on Kubernetes)
Check worker logs for discovery registration errors

Inference returns the base model response

Confirm the request model field matches the loaded lora_name
Verify the adapter is loaded on the worker handling the request
For vLLM disaggregated serving, load the adapter on both prefill and decode workers