KV Cache Transfer

For general TensorRT-LLM features and configuration, see the Reference Guide.

In disaggregated serving architectures, KV cache must be transferred between prefill and decode workers. TensorRT-LLM supports three methods for this transfer:

• NIXL with UCX (default)
• NIXL with Libfabric
• using UCX directly

Using NIXL for KV Cache Transfer

Start the disaggregated service: See Disaggregated Serving to learn how to start the deployment.

Default Method: NIXL with UCX

By default, TensorRT-LLM uses NIXL (NVIDIA Inference Xfer Library) with UCX (Unified Communication X) as backend for KV cache transfer between prefill and decode workers. NIXL is NVIDIA’s high-performance communication library designed for efficient data transfer in distributed GPU environments.

Specify Backends for NIXL

TensorRT-LLM supports two NIXL communication backends: UCX and LIBFABRIC. By default, UCX is used if no backend is explicitly specified. Dynamo currently supports both backends. For AWS EFA deployments, UCX with SRD transport is the tested and recommended backend (see AWS EFA below).

Alternative Method: UCX

TensorRT-LLM can also leverage UCX (Unified Communication X) directly for KV cache transfer between prefill and decode workers. To enable UCX as the KV cache transfer backend, set cache_transceiver_config.backend: UCX in your engine configuration YAML file.

cache_transceiver_config.backend accepts the following values:

The precedence above matches TensorRT-LLM 1.3.0rc22; check CacheTransceiverConfig._resolve_default_backend in tensorrt_llm/llmapi/llm_args.py.

AWS EFA

On AWS, UCX uses the SRD (Scalable Reliable Datagram) transport over EFA devices. NIXL discovers EFA rdmap* devices automatically through UCX — no NIXL-level configuration changes are needed.

Image options:

• Pre-built EFA image: A dedicated EFA image with the EFA SDK baked in is available on NGC, for both AMD64 and ARM64:

nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-efa

On 1.2.1 the same image is tagged 1.2.1-efa-amd64. Despite the suffix that tag is a multi-arch manifest covering AMD64 and ARM64; the name was corrected to -efa in 1.3.0. Pull the tag exactly as written — 1.2.1-efa was never published.

See Release Artifacts for all available EFA images.

• Host-mount approach (ARM64 / GB200): Instead of the pre-built image, you can run the standard tensorrtllm-runtime image and mount the EFA SDK from the host node, which keeps the SDK in step with the host driver. This is what we tested on GB200 NVL72:

1
volumeMounts:
2
  - name: efa-sdk
3
    mountPath: /opt/amazon/efa
4
volumes:
5
  - name: efa-sdk
6
    hostPath:
7
      path: /opt/amazon/efa

EFA resource requests:

1
resources:
2
  requests:
3
    vpc.amazonaws.com/efa: "4"
4
  limits:
5
    vpc.amazonaws.com/efa: "4"

Required environment variables for EFA workers (set on both prefill and decode):

1
env:
2
  - name: FI_PROVIDER
3
    value: "efa"
4
  - name: FI_EFA_USE_DEVICE_RDMA
5
    value: "1"
6
  - name: FI_EFA_ENABLE_SHM_TRANSFER
7
    value: "0"
8
  - name: LD_LIBRARY_PATH
9
    value: "/opt/amazon/efa/lib:/usr/local/lib:/usr/lib"

Security context: AWS EFA currently requires privileged mode:

1
securityContext:
2
  privileged: true

NIXL Plugin ABI Mismatch on Decode Multinode

When running multinode decode, the decode leader launches workers via mpirun -> mgmn_worker_node, which loads TRT-LLM’s bundled NIXL rather than the system nixl_cu13. The container’s default NIXL_PLUGIN_DIR points to system plugins that are ABI-incompatible with TRT-LLM’s bundled NIXL. Override this on the decode service only:

1
env:
2
  - name: NIXL_PLUGIN_DIR
3
    value: "/opt/dynamo/venv/lib/python3.12/site-packages/tensorrt_llm/libs/nixl/plugins"

Do not set this on prefill workers — they use nixl_cu13 which is compatible with the system plugins.

ComputeDomain for GB200 NVL72

On GB200 NVL72 racks, NCCL requires a ComputeDomain CR for proper cuMem/NVLS initialization. Without it, workers fail with NCCL error 'unhandled system error' during model loading.

1
apiVersion: resource.nvidia.com/v1beta1
2
kind: ComputeDomain
3
metadata:
4
  name: my-compute-domain
5
spec:
6
  numNodes: 3    # total nodes across prefill + decode
7
  channel:
8
    resourceClaimTemplate:
9
      name: my-compute-domain-channel

Both prefill and decode services must include ResourceClaims:

1
resources:
2
  claims:
3
    - name: compute-domain-channel
4
extraPodSpec:
5
  resourceClaims:
6
    - name: compute-domain-channel
7
      resourceClaimTemplateName: my-compute-domain-channel

Required NCCL environment variables for GB200:

1
env:
2
  - name: NCCL_MNNVL_ENABLE
3
    value: "1"
4
  - name: NCCL_CUMEM_ENABLE
5
    value: "1"
6
  - name: NCCL_NVLS_ENABLE
7
    value: "1"
8
  - name: NVIDIA_GDRCOPY
9
    value: "1"

Verifying EFA is Active

After deployment, confirm NIXL is using SRD over EFA in the worker logs:

$
kubectl logs <prefill-pod> | grep -iE "NixlTransfer|srd|rdmap"

Expected output:

NixlTransferAgent using NIXL backend: UCX
ucp_context_2 self cfg#1 rma_am(srd/rdmap40s0:1) am(srd/rdmap40s0:1 srd/rdmap62s0:1 ...)
NixlTransferAgent mAddress: 100.x.x.x:32939

• srd/rdmap* confirms SRD transport over EFA devices
• Multiple rdmap entries correspond to one EFA device per GPU