| **Serving Claude Code with Dynamo** | **Serving Codex with Dynamo** |
### Component Responsibilities
| Component | Role | Location |
|-----------|------|----------|
| **NIXL** | High-level KV cache transfer API | Dynamo runtime library |
| **UCX or libfabric** | Low-level communication framework | System library |
| **Transports** | Physical data movement | Hardware/kernel drivers |
---
## Why NVLink Cannot Be Used Between Pods
### The Fundamental Constraint
NVLink is a **direct GPU-to-GPU interconnect** that operates at the hardware level. It requires:
1. **Same process** - Both GPUs must be visible to a single process so `cudaDeviceEnablePeerAccess()` can be called
2. **Direct memory access** - Process must have permission to access both GPU memory regions
3. **Peer-to-peer mapping** - CUDA runtime must establish memory mappings between GPUs
**Kubernetes pods violate all three requirements:**
### Technical Explanation
1. **Process Isolation**: Kubernetes pods run in separate Linux namespaces. Even on the same node, Pod A cannot directly access Pod B's memory space.
2. **GPU Partitioning**: The Kubernetes device plugin assigns specific GPUs to each pod via `CUDA_VISIBLE_DEVICES`. Pod A's GPU 0 and Pod B's GPU 0 are physically different devices.
3. **Process/Namespace Isolation**: Each pod runs in a separate process namespace. NVLink peer-to-peer transfers require both GPUs to be within the same process so `cudaDeviceEnablePeerAccess()` can be called.
4. **Memory Registration**: NVLink transfers use `cudaMemcpy` with peer access enabled. This requires calling `cudaDeviceEnablePeerAccess()` - impossible across process boundaries.
### Where NVLink DOES Work
NVLink works **within a pod** for parallelism strategies (TP, EP) where all GPUs are in the same process:
```yaml
# Decode worker with TP=4 uses NVLink between its 4 GPUs
VLLMDecodeWorker:
resources:
limits:
gpu: "4" # All 4 GPUs visible to single process
args:
- --tensor-parallel-size
- "4" # NVLink used for TP/EP communication within pod
```
---
## Supported Communication Options
### Transport Comparison
| Transport | Bandwidth | Latency | Same-Node | Cross-Node | GPU Direct |
|-----------|-----------|---------|-----------|------------|------------|
| **NVLink** | 450-900 GB/s | ~µs | ✅ (intra-pod only) | ❌ | ✅ |
| **InfiniBand RDMA** | 20-50 GB/s | ~1 µs | ✅ | ✅ | ✅ (with GPUDirect) |
| **RoCE RDMA** | 10-25 GB/s | ~2 µs | ✅ | ✅ | ✅ (with GPUDirect) |
| **TCP** | 1-3 GB/s | ~50 µs | ✅ | ✅ | ❌ (host staging) |
### Same-Node Communication
When prefill and decode workers are on the **same physical node**:
**Options (best to worst):**
1. InfiniBand RDMA with GPUDirect → GPU-to-GPU, bypasses CPU
2. RoCE RDMA with GPUDirect → GPU-to-GPU, bypasses CPU
3. Host-staged RDMA → GPU→CPU→RDMA→CPU→GPU
4. TCP (fallback) → GPU→CPU→TCP→CPU→GPU
**Best Practice**: Use RDMA even for same-node communication. The overhead is minimal and it provides consistent behavior whether pods land on the same or different nodes.
### Cross-Node Communication
When prefill and decode workers are on **different nodes**:
**Requirements for optimal cross-node performance:**
- RDMA network fabric (InfiniBand, RoCE, or AWS EFA)
- GPUDirect RDMA enabled (GPU memory registered with NIC)
- Proper UCX or libfabric configuration
---
## UCX Configuration Reference
### Environment Variables
UCX behavior is controlled through environment variables. Set these on both prefill and decode worker pods.
#### Core Transport Selection
```yaml
env:
- name: UCX_TLS
value: "rc_x,rc,dc_x,dc,cuda_copy,cuda_ipc"
```
| Transport | Description | When to Use |
|-----------|-------------|-------------|
| `rc_x` | Reliable Connection (accelerated) | Primary RDMA transport |
| `rc` | Reliable Connection (standard) | Fallback RDMA |
| `dc_x` | Dynamically Connected (accelerated) | Scalable RDMA (many endpoints) |
| `dc` | Dynamically Connected (standard) | Fallback scalable RDMA |
| `cuda_copy` | GPU↔Host memory staging | Required for GPU buffers |
| `cuda_ipc` | CUDA IPC (same-node, same-pod) | Intra-pod GPU transfers |
| `tcp` | TCP sockets | Fallback when RDMA unavailable |
| `srd` | Scalable Reliable Datagram (AWS EFA) | AWS-specific (provided by EFA, not core UCX) |
**Excluding transports**: Use `^` prefix to exclude (e.g., `UCX_TLS=^mm` excludes memory mapping).
**Note**: When specifying `UCX_TLS` explicitly with GPU memory, you must include `cuda_copy` or `cuda_ipc` for UCX to recognize GPU buffers.
#### Rendezvous Protocol Settings
```yaml
env:
- name: UCX_RNDV_SCHEME
value: "get_zcopy"
- name: UCX_RNDV_THRESH
value: "0"
```
| Variable | Value | Description |
|----------|-------|-------------|
| `UCX_RNDV_SCHEME` | `get_zcopy` | Zero-copy RDMA GET (receiver pulls data) |
| `UCX_RNDV_SCHEME` | `put_zcopy` | Zero-copy RDMA PUT (sender pushes data) |
| `UCX_RNDV_SCHEME` | `auto` | Let UCX choose based on message size |
| `UCX_RNDV_THRESH` | `0` | Use rendezvous for all message sizes |
| `UCX_RNDV_THRESH` | `8192` | Use rendezvous for messages ≥8KB |
| `UCX_RNDV_THRESH` | `auto` | Let UCX calculate optimal threshold |
**Recommendation**: Use `get_zcopy` with threshold `0` for KV cache transfers (always large).
> **⚠️ AWS EFA Exception**: Do NOT use `get_zcopy` on AWS with Ubuntu 24.04 + Kernel ≥6.8. See [AWS EFA Configuration](#aws-efa-configuration) for required settings.
#### Memory Registration
```yaml
env:
- name: UCX_IB_REG_METHODS
value: "odp,rcache"
```
| Method | Description |
|--------|-------------|
| `odp` | On-Demand Paging (dynamic registration) |
| `rcache` | Registration cache (reuse registrations) |
| `direct` | Direct registration (each transfer) |
#### Debugging and Diagnostics
```yaml
env:
- name: UCX_LOG_LEVEL
value: "info" # Options: fatal, error, warn, info, debug, trace, data, func
- name: UCX_LOG_FILE
value: "/tmp/ucx.log" # Optional: log to file instead of stdout
```
**Note**: UCX statistics (`UCX_STATS_DEST`, `UCX_STATS_TRIGGER`) require UCX compiled with `--enable-stats` flag, which is not enabled in default builds.
### Complete Production Configuration
```yaml
env:
# Transport selection - RDMA with GPU support
- name: UCX_TLS
value: "rc_x,rc,dc_x,dc,cuda_copy,cuda_ipc"
# Rendezvous for large transfers
- name: UCX_RNDV_SCHEME
value: "get_zcopy"
- name: UCX_RNDV_THRESH
value: "0"
# Memory registration optimization
- name: UCX_IB_REG_METHODS
value: "odp,rcache"
# RDMA settings
- name: UCX_IB_GID_INDEX
value: "3" # RoCE v2 GID index (cluster-specific)
```
### InfiniBand Configuration
For clusters with InfiniBand RDMA (e.g., ConnectX NICs), use UCX with the `rc` (Reliable Connection) transport. This is the standard path for on-premises and bare-metal Kubernetes clusters.
**RDMA Resources:**
Request one `rdma/ib` device per GPU. The RDMA device plugin injects `/dev/infiniband/*` devices automatically:
```yaml
resources:
limits:
gpu: "4"
custom:
rdma/ib: "4"
```
No pod annotations are needed. InfiniBand devices are injected by the device plugin.
**Security Context:**
Add `IPC_LOCK` and `SYS_RESOURCE` capabilities. `IPC_LOCK` allows RDMA memory pinning, `SYS_RESOURCE` allows memlock limit escalation:
```yaml
securityContext:
runAsUser: 0
capabilities:
add:
- IPC_LOCK
- SYS_RESOURCE
```
**Environment Variables (worker containers):**
```yaml
env:
# --- UCX (RDMA transport) ---
- name: UCX_TLS
value: "rc_x,rc,cuda_copy,cuda_ipc"
- name: UCX_NET_DEVICES
value: "