> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # LoRA Adapters ## What Are LoRA Adapters Low-Rank Adaptation (LoRA) serves specialized model variants without duplicating full base weights. Dynamo supports dynamic LoRA lifecycle management with vLLM and SGLang, with different validation levels for each backend. It provides: - **Dynamic loading**: Load and unload adapters without restarting workers - **Multiple sources**: `file://`, `s3://`, or `hf://` URIs - **Automatic caching**: Downloaded adapters are cached under `DYN_LORA_PATH` - **Discovery**: Loaded adapters appear in `/v1/models` - **KV-aware routing**: Route requests to workers with the matching adapter and cached prefix blocks; validated with vLLM - **Kubernetes native**: Manage adapters declaratively through the `DynamoModel` CRD; documented for vLLM ## Backend Support | Backend | Status | Support Scope | |---------|--------|-----------------| | vLLM | Supported | Dynamic load/unload for aggregated and disaggregated workers; aggregated adapter-aware KV routing | | SGLang | Experimental | Dynamic load/unload and aggregated inference; disaggregated serving and feature pairings are not end-to-end validated | | TensorRT-LLM | Not supported | — | See the [feature support matrix](../../reference/general/compatibility.mdx#feature-support) for the backend and interaction matrices. ```mermaid flowchart TD Frontend["Frontend"] --> Router["Router
(LoRA-aware)"] Router --> Workers["Workers
(LoRA-loaded)"] Workers --> ManagerNode["LoRA Manager"] subgraph ManagerGroup["LoRA Manager"] Downloader Cache end ManagerNode --> Local["file://
Local"] ManagerNode --> S3["s3://
S3/MinIO"] ManagerNode --> HF["hf://
(custom)"] ``` - **Rust core** (`lib/llm/src/lora/`): Downloading, caching, and validation - **Python manager** (`components/src/dynamo/common/lora/`): Custom source support - **Worker handlers** (`components/src/dynamo/vllm/handlers.py` and `components/src/dynamo/sglang/request_handlers/handler_base.py`): Backend load/unload and inference integration ## Serve a LoRA Adapter The following Kubernetes and local workflows use vLLM. For the aggregated SGLang workflow, see [Serve a LoRA Adapter with SGLang](#serve-a-lora-adapter-with-sglang). Keep `DYN_SYSTEM_PORT` on a trusted administrative network. The system API retrieves and loads model artifacts from configured URI sources; do not expose it to untrusted clients. - A Kubernetes cluster with the Dynamo Platform installed and a vLLM runtime image - A LoRA adapter compatible with your base model - For `s3://` sources: AWS credentials in a Kubernetes Secret Full manifests and MinIO setup: [Kubernetes LoRA deployment example](https://github.com/ai-dynamo/dynamo/tree/v1.5.0/examples/backends/vllm/deploy/lora). Enable LoRA on the **worker**: set LoRA and system API environment variables, and pass vLLM LoRA flags in `args:`. `DYN_SYSTEM_ENABLED` and `DYN_SYSTEM_PORT` expose load/unload on the worker system port. Adapted from the [`agg_lora.yaml` example](https://github.com/ai-dynamo/dynamo/blob/v1.5.0/examples/backends/vllm/deploy/lora/agg_lora.yaml): ```yaml apiVersion: nvidia.com/v1beta1 kind: DynamoGraphDeployment metadata: name: vllm-agg-lora spec: components: - name: Frontend type: frontend replicas: 1 podTemplate: spec: containers: - name: main image: ${RUNTIME_IMAGE} - name: VllmDecodeWorker type: decode replicas: 1 podTemplate: spec: containers: - name: main image: ${RUNTIME_IMAGE} workingDir: /workspace/examples/backends/vllm envFrom: - secretRef: name: hf-token-secret env: - name: DYN_LORA_ENABLED value: "true" - name: DYN_LORA_PATH value: /tmp/dynamo_loras - name: DYN_SYSTEM_ENABLED value: "true" - name: DYN_SYSTEM_PORT value: "9090" command: - python3 - -m - dynamo.vllm args: - --model - Qwen/Qwen3-0.6B - --enable-lora - --max-lora-rank - "64" - --enforce-eager resources: limits: nvidia.com/gpu: "1" ``` Apply and wait for readiness: ```bash kubectl apply -f vllm-agg-lora.yaml -n ${NAMESPACE} kubectl wait --for=condition=Ready dynamographdeployment/vllm-agg-lora \ -n ${NAMESPACE} --timeout=600s ``` **Worker environment variables** | Variable | Description | Default | |----------|-------------|---------| | `DYN_LORA_ENABLED` | Enable LoRA adapter support | `false` | | `DYN_LORA_PATH` | Local cache directory for downloaded LoRAs | `~/.cache/dynamo_loras` | | `DYN_SYSTEM_ENABLED` | Expose worker load/unload API | `false` | | `DYN_SYSTEM_PORT` | System API port | — | | `AWS_ACCESS_KEY_ID` | S3 access key from the environment credential provider | — | | `AWS_SECRET_ACCESS_KEY` | S3 secret key from the environment credential provider | — | | `AWS_PROFILE` | Shared AWS configuration and credentials profile | `default` | | `AWS_SHARED_CREDENTIALS_FILE` | Shared credentials file path | `~/.aws/credentials` | | `AWS_CONFIG_FILE` | Shared configuration file path | `~/.aws/config` | | `AWS_ENDPOINT_URL_S3` | Service-specific custom S3 endpoint | — | | `AWS_ENDPOINT_URL` | Generic custom AWS endpoint | — | | `AWS_ENDPOINT` | Legacy custom S3 endpoint fallback | — | | `AWS_REGION` | AWS region | `us-east-1` | | `AWS_ALLOW_HTTP` | Allow HTTP (non-TLS) connections | `false` | | `AWS_VIRTUAL_HOSTED_STYLE_REQUEST` | Use bucket-qualified virtual-hosted requests | `false` | For `s3://` sources, Dynamo uses the standard AWS credential provider chain, including environment variables, shared configuration and credentials files, web identity, container credentials, and Amazon EC2 instance metadata. Endpoint precedence is `AWS_ENDPOINT_URL_S3`, the active profile's S3 service endpoint, `AWS_ENDPOINT_URL`, the active profile's generic `endpoint_url`, then the legacy `AWS_ENDPOINT` fallback. **vLLM worker arguments** | Argument | Description | |----------|-------------| | `--enable-lora` | Enable LoRA adapter support in vLLM | | `--max-lora-rank` | Maximum LoRA rank (must be >= your adapter's rank) | | `--max-loras` | Maximum number of LoRAs loaded simultaneously | Set `--max-lora-rank` to at least your adapter's rank. A lower value causes load failures. Mount shared AWS files and set `AWS_PROFILE`, or store environment credentials in a Kubernetes Secret. Set non-secret endpoint and region values inline: ```yaml env: - name: DYN_LORA_ENABLED value: "true" - name: AWS_ENDPOINT_URL value: http://minio:9000 # for MinIO; omit for AWS S3 - name: AWS_REGION value: us-east-1 - name: AWS_ALLOW_HTTP value: "true" # MinIO/non-TLS only - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: minio-secret key: AWS_ACCESS_KEY_ID - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: minio-secret key: AWS_SECRET_ACCESS_KEY ``` Use a `DynamoModel` CRD for declarative, cluster-native loading. It discovers worker endpoints for `baseModelName`, creates a Service, and calls the load API on each pod. ```yaml apiVersion: nvidia.com/v1alpha1 kind: DynamoModel metadata: name: customer-support-lora namespace: dynamo-system spec: modelName: customer-support-adapter-v1 baseModelName: Qwen/Qwen3-0.6B # Must match the worker --model value modelType: lora source: uri: s3://my-models-bucket/loras/customer-support/v1 ``` Verify readiness: ```bash kubectl get dynamomodel customer-support-lora # NAME TOTAL READY AGE # customer-support-lora 2 2 30s ``` See the [DynamoModel API reference](/dynamo/reference/api/kubernetes/full-api-reference#dynamomodel) for the CRD fields. Port-forward the worker system port and POST to `/v1/loras`: ```bash kubectl port-forward svc/vllm-agg-lora-vllmdecodeworker 9090:9090 -n ${NAMESPACE} ``` ```bash curl -X POST http://localhost:9090/v1/loras \ -H "Content-Type: application/json" \ -d '{ "lora_name": "customer-support-lora", "source": { "uri": "s3://my-loras/customer-support-v1" } }' ``` List loaded adapters with `GET /v1/loras`. Unload with `DELETE /v1/loras/{lora_name}`. Port-forward the Frontend and set the request `model` field to the adapter name (`lora_name` or `DynamoModel` `modelName`): ```bash kubectl port-forward svc/vllm-agg-lora-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "customer-support-lora", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 100 }' ``` The `model` field is case-sensitive and must match the loaded adapter name exactly. For vLLM disaggregated serving, load the adapter on both prefill and decode workers. - Dynamo with the vLLM backend installed and a CUDA GPU - A LoRA adapter compatible with your base model For S3-compatible local storage, run the MinIO setup script first: ```bash cd examples/backends/vllm/launch/lora ./setup_minio.sh ``` See the [local LoRA example](https://github.com/ai-dynamo/dynamo/tree/v1.5.0/examples/backends/vllm/launch/lora) for the full workflow. Launch the frontend and a LoRA-enabled vLLM worker: ```bash cd examples/backends/vllm/launch/lora ./agg_lora.sh ``` The script sets `DYN_LORA_ENABLED`, `DYN_LORA_PATH`, MinIO credentials, and starts `dynamo.frontend` plus `dynamo.vllm` with `--enable-lora` and `--max-lora-rank 64`. Override the base model with the `MODEL` environment variable. To run the commands manually instead: ```bash export DYN_LORA_ENABLED=true export DYN_LORA_PATH=/tmp/dynamo_loras export DYN_SYSTEM_ENABLED=true export DYN_SYSTEM_PORT=8081 python -m dynamo.frontend & python -m dynamo.vllm --model Qwen/Qwen3-0.6B \ --enable-lora --max-lora-rank 64 --enforce-eager ``` Set `AWS_*` variables when loading from `s3://` URIs (the launch script configures MinIO defaults). POST to the worker system API on `DYN_SYSTEM_PORT` (default `8081` in `agg_lora.sh`): ```bash curl -X POST http://localhost:8081/v1/loras \ -H "Content-Type: application/json" \ -d '{ "lora_name": "my-lora", "source": { "uri": "s3://my-loras/my-lora" } }' ``` Use `file:///path/to/my-lora` for local adapters. List loaded adapters with `GET /v1/loras`. Unload with `DELETE /v1/loras/{lora_name}`. Send a chat completion request to the frontend (default port `8000`) with `model` set to the adapter name: ```bash curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "my-lora", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 100 }' ``` The `model` field is case-sensitive and must match `lora_name` exactly. For vLLM disaggregated serving, load the adapter on both prefill and decode workers. ## Serve a LoRA Adapter with SGLang **Experimental.** The repository validates dynamic loading, discovery, and inference for aggregated SGLang workers. Unloading is implemented but not exercised by an end-to-end test. Prefill and decode lifecycle registration has unit coverage, but disaggregated SGLang LoRA and feature pairings such as KV-aware routing are not end-to-end validated. The Kubernetes workflow above and the adapter-aware routing demo below are vLLM-specific. Start MinIO and upload the example adapter: ```bash cd examples/backends/sglang/launch/lora ./setup_minio.sh ``` See the [SGLang LoRA example](https://github.com/ai-dynamo/dynamo/tree/v1.5.0/examples/backends/sglang/launch/lora) for the scripts and configurable environment variables. Start the Dynamo frontend and a LoRA-enabled SGLang worker: ```bash ./agg_lora.sh ``` The script starts `dynamo.sglang` with `--enable-lora`, `--max-lora-rank 64`, and `--lora-target-modules all`. It also sets `DYN_LORA_ENABLED`, `DYN_LORA_PATH`, and the worker system port. Load the adapter through the worker system API, then address it by name in the OpenAI-compatible request: ```bash curl -X POST http://localhost:8081/v1/loras \ -H "Content-Type: application/json" \ -d '{ "lora_name": "codelion/Qwen3-0.6B-accuracy-recovery-lora", "source": { "uri": "s3://my-loras/codelion/Qwen3-0.6B-accuracy-recovery-lora" } }' ``` ```bash curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "codelion/Qwen3-0.6B-accuracy-recovery-lora", "messages": [{"role": "user", "content": "What is deep learning?"}], "max_tokens": 100 }' ``` List loaded adapters with `GET /v1/loras`. Unload with `DELETE /v1/loras/{lora_name}`. ## KV Cache-Aware LoRA Routing This section describes the validated vLLM path. KV-aware routing with SGLang LoRA remains experimental because the combined path is not end-to-end validated. When KV-aware routing is enabled, the router accounts for LoRA adapter identity when computing block hashes: - **Distinct hash spaces per adapter**: Blocks cached under adapter `A` are never confused with adapter `B` or the base model, even when token sequences match. The adapter name is mixed into the `LocalBlockHash` computation. - **Prefix sharing within the same adapter**: Requests targeting the same LoRA adapter reuse KV prefix blocks like base-model requests. - **No extra configuration**: The LoRA name propagates through KV events (`BlockStored`) from the engine to the router. The router uses the `lora_name` field to route requests to workers with matching cached blocks. This works across the publisher pipeline, the KV consolidator, and the routing query path. For a local two-worker demo with KV-aware routing, run [`agg_lora_router.sh`](https://github.com/ai-dynamo/dynamo/blob/v1.5.0/examples/backends/vllm/launch/lora/agg_lora_router.sh) and load the adapter on both worker system ports. ## Troubleshooting **Check S3 connectivity:** ```bash aws s3 ls s3://my-loras/ --recursive ``` **Check the cache directory:** ```bash ls -la ~/.cache/dynamo_loras/ # or the path set in DYN_LORA_PATH ``` **Check worker logs:** ```bash kubectl logs deployment/my-worker | grep -i lora ``` Confirm `--max-lora-rank` is at least your adapter's rank. - Verify the LoRA name matches exactly (case-sensitive) - List loaded adapters: `curl http://localhost:9090/v1/loras` (port-forward the worker's `DYN_SYSTEM_PORT` first on Kubernetes) - Check worker logs for discovery registration errors - Confirm the request `model` field matches the loaded `lora_name` - Verify the adapter is loaded on the worker handling the request - For vLLM disaggregated serving, load the adapter on both prefill and decode workers