KV Cache Offloading

Deploy LMCache MP

Run aggregated vLLM serving with KV cache offloaded to a per-node LMCache MP DaemonSet over cross-Pod CUDA IPC.

View as Markdown

This guide deploys Dynamo aggregated vLLM serving on Kubernetes with KV cache offloaded to a per-node LMCache MP DaemonSet, sharing tensors with the worker via cross-Pod CUDA IPC. It uses the v1beta1/agg_lmcache.yaml manifest plus an LMCacheEngine CR managed by the LMCache operator.

Prerequisites

  • cert-manager — the LMCache operator’s webhook certificates are issued through it: 
    $kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.1/cert-manager.yaml
1

Install the Dynamo platform

$export RELEASE_VERSION=1.3.0
$helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-$RELEASE_VERSION.tgz
$helm install dynamo-platform dynamo-platform-$RELEASE_VERSION.tgz \
>  --namespace dynamo-system --create-namespace \
>  --wait

Verify the install:

$kubectl get crd | grep nvidia.com
$# expect (among others):
$#   dynamographdeployments.nvidia.com
2

Install the LMCache operator

$kubectl apply -f https://github.com/LMCache/LMCache/releases/download/operator-latest/install.yaml

Tested with the LMCache operator image lmcache/lmcache-operator:v0.5.2.

Verify the install:

$kubectl get crd | grep lmcache
$# expect (among others):
$#   lmcacheengines.lmcache.lmcache.ai
3

Create the namespace

$kubectl create namespace dynamo-lmcache
4

Create the HF token Secret

Both Frontend and VllmDecodeWorker reference hf-token-secret via a Secret ref. The Secret must exist or the pods fail to start with secret "hf-token-secret" not found.

$# Replace the dummy token below with a real HF token.
$kubectl apply -f - <<'EOF'
$apiVersion: v1
$kind: Secret
$metadata:
$  name: hf-token-secret
$  namespace: dynamo-lmcache
$type: Opaque
$stringData:
$  HF_TOKEN: "hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$EOF
5

Deploy the LMCacheEngine

Replace my-tag below with the lmcache/vllm-openai image tag you want to run.

$kubectl apply -f - <<'EOF'
$apiVersion: lmcache.lmcache.ai/v1alpha1
$kind: LMCacheEngine
$metadata:
$  name: lmcache-mp
$  namespace: dynamo-lmcache
$spec:
$  image:
$    repository: lmcache/vllm-openai
$    tag: my-tag
$    pullPolicy: IfNotPresent
$  # L1 (CPU RAM) cache size — bump for production workloads.
$  l1:
$    sizeGB: 16
$EOF

The server tag must match the LMCache version bundled inside the Dynamo worker image from the next step. Validated: server v0.4.6 paired with nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.3.0 (bundles LMCache 0.4.6). Check it with: docker run --rm --entrypoint python3 <worker-image> -c "import lmcache; print(lmcache.__version__)". Mismatched versions do not speak the same MP wire protocol.

Verify with:

$kubectl -n dynamo-lmcache get lmcacheengine lmcache-mp
$# expect: PHASE  Running
6

Deploy the Dynamo worker

Edit examples/backends/vllm/deploy/v1beta1/agg_lmcache.yaml: replace nvcr.io/nvidia/ai-dynamo/vllm-runtime:my-tag (on both Frontend and VllmDecodeWorker) with your Dynamo vllm-runtime image.

$kubectl apply -n dynamo-lmcache -f examples/backends/vllm/deploy/v1beta1/agg_lmcache.yaml

Verify with:

$kubectl -n dynamo-lmcache get pods -l nvidia.com/dynamo-component-type=worker
$# expect: READY 1/1, STATUS Running
7

Verify

Send the same long prompt twice:

$kubectl -n dynamo-lmcache port-forward svc/vllm-agg-lmcache-frontend 8000:8000 >/dev/null &
$trap 'kill %1' EXIT
$sleep 4
$
$PROMPT=$(python3 -c "print('the quick brown fox jumps over the lazy dog '*60)")
$REQ="{\"model\":\"Qwen/Qwen3-0.6B\",\"messages\":[{\"role\":\"user\",\"content\":\"$PROMPT\"}],\"max_tokens\":5}"
$
$for label in cold warm; do
$  echo "--- $label ---"
$  curl -s http://localhost:8000/v1/chat/completions \
>    -H "Content-Type: application/json" -d "$REQ" \
>    | python3 -m json.tool | grep -E "prompt_tokens|cached_tokens"
$done

Then check LMCache server metrics:

$# The DaemonSet pod on the worker's node serves the traffic; check them all.
>for LMC in $(kubectl -n dynamo-lmcache get pod -l app.kubernetes.io/instance=lmcache-mp -o name); do
>  kubectl -n dynamo-lmcache exec "$LMC" -- curl -s localhost:8080/metrics | grep '^lmcache_mp_lookup'
>done

Expected: warm response shows cached_tokens > 0, and lmcache_mp_lookup_hit_tokens_total > 0.

The lmcache_mp_ metrics are served on the control HTTP port 8080 (exposed by the lmcache-mp Service).

Cleanup

$kubectl delete -n dynamo-lmcache -f examples/backends/vllm/deploy/v1beta1/agg_lmcache.yaml
$kubectl -n dynamo-lmcache delete lmcacheengine lmcache-mp
$kubectl delete namespace dynamo-lmcache
$kubectl delete -f https://github.com/LMCache/LMCache/releases/download/operator-latest/install.yaml
$helm uninstall dynamo-platform -n dynamo-system
$kubectl delete namespace dynamo-system