KV Cache Offloading

Deploy LMCache MP

Run aggregated vLLM serving with KV cache offloaded to a per-node LMCache MP DaemonSet over cross-Pod CUDA IPC.

以 Markdown 格式查看

This guide deploys Dynamo aggregated vLLM serving on Kubernetes with KV cache offloaded to a per-node LMCache MP DaemonSet, sharing tensors with the worker via cross-Pod CUDA IPC. It uses the v1beta1/agg_lmcache.yaml manifest plus an LMCacheEngine CR managed by the LMCache operator.

Prerequisites

  • cert-manager — the LMCache operator’s webhook certificates are issued through it: 
    kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.1/cert-manager.yaml
1

Install the Dynamo platform

export RELEASE_VERSION=1.3.0
helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-$RELEASE_VERSION.tgz
helm install dynamo-platform dynamo-platform-$RELEASE_VERSION.tgz \
  --namespace dynamo-system --create-namespace \
  --wait

Verify the install:

kubectl get crd | grep nvidia.com
# expect (among others):
#   dynamographdeployments.nvidia.com
2

Install the LMCache operator

kubectl apply -f https://github.com/LMCache/LMCache/releases/download/operator-latest/install.yaml

Tested with the LMCache operator image lmcache/lmcache-operator:v0.5.2.

Verify the install:

kubectl get crd | grep lmcache
# expect (among others):
#   lmcacheengines.lmcache.lmcache.ai
3

Create the namespace

kubectl create namespace dynamo-lmcache
4

Create the HF token Secret

Both Frontend and worker reference hf-token-secret via a Secret ref. The Secret must exist or the pods fail to start with secret "hf-token-secret" not found.

# Replace the dummy token below with a real HF token.
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: hf-token-secret
  namespace: dynamo-lmcache
type: Opaque
stringData:
  HF_TOKEN: "hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
EOF
5

Deploy the LMCacheEngine

Replace my-tag below with the lmcache/vllm-openai image tag you want to run.

kubectl apply -f - <<'EOF'
apiVersion: lmcache.lmcache.ai/v1alpha1
kind: LMCacheEngine
metadata:
  name: lmcache-mp
  namespace: dynamo-lmcache
spec:
  image:
    repository: lmcache/vllm-openai
    tag: my-tag
    pullPolicy: IfNotPresent
  # L1 (CPU RAM) cache size — bump for production workloads.
  l1:
    sizeGB: 16
EOF

The server tag must match the LMCache version bundled inside the Dynamo worker image from the next step. Validated: server v0.4.6 paired with nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.3.0 (bundles LMCache 0.4.6). Check it with: docker run --rm --entrypoint python3 <worker-image> -c "import lmcache; print(lmcache.__version__)". Mismatched versions do not speak the same MP wire protocol.

Verify with:

kubectl -n dynamo-lmcache get lmcacheengine lmcache-mp
# expect: PHASE  Running
6

Deploy the Dynamo worker

Edit examples/backends/vllm/deploy/v1beta1/agg_lmcache.yaml: replace nvcr.io/nvidia/ai-dynamo/vllm-runtime:my-tag (on both Frontend and worker) with your Dynamo vllm-runtime image.

kubectl apply -n dynamo-lmcache -f examples/backends/vllm/deploy/v1beta1/agg_lmcache.yaml

Verify with:

kubectl -n dynamo-lmcache get pods -l nvidia.com/dynamo-component-type=worker
# expect: READY 1/1, STATUS Running
7

Verify

Send the same long prompt twice:

kubectl -n dynamo-lmcache port-forward svc/vllm-agg-lmcache-frontend 8000:8000 >/dev/null &
trap 'kill %1' EXIT
sleep 4
PROMPT=$(python3 -c "print('the quick brown fox jumps over the lazy dog '*60)")
REQ="{\"model\":\"Qwen/Qwen3-0.6B\",\"messages\":[{\"role\":\"user\",\"content\":\"$PROMPT\"}],\"max_tokens\":5}"
for label in cold warm; do
  echo "--- $label ---"
  curl -s http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" -d "$REQ" \
    | python3 -m json.tool | grep -E "prompt_tokens|cached_tokens"
done

Then check LMCache server metrics:

# The DaemonSet pod on the worker's node serves the traffic; check them all.
for LMC in $(kubectl -n dynamo-lmcache get pod -l app.kubernetes.io/instance=lmcache-mp -o name); do
  kubectl -n dynamo-lmcache exec "$LMC" -- curl -s localhost:8080/metrics | grep '^lmcache_mp_lookup'
done

Expected: warm response shows cached_tokens > 0, and lmcache_mp_lookup_hit_tokens_total > 0.

The lmcache_mp_ metrics are served on the control HTTP port 8080 (exposed by the lmcache-mp Service).

Cleanup

kubectl delete -n dynamo-lmcache -f examples/backends/vllm/deploy/v1beta1/agg_lmcache.yaml
kubectl -n dynamo-lmcache delete lmcacheengine lmcache-mp
kubectl delete namespace dynamo-lmcache
kubectl delete -f https://github.com/LMCache/LMCache/releases/download/operator-latest/install.yaml
helm uninstall dynamo-platform -n dynamo-system
kubectl delete namespace dynamo-system