Deploy the DC KV Relay

Experimental. Deploy NVIDIA Dynamo’s DC KV Relay alongside existing inference workers. The Relay uses the shared Dynamo runtime and universal publisher; it does not serve inference requests or choose a destination data center. For the producer model, see DC KV Relay Concepts.

Prerequisites

• A Dynamo operator installation with the DynamoWorkerMetadata CRD.
• Ready inference workers using Kubernetes discovery in one Kubernetes namespace. They must advertise model cards, KV event sources, and a recoverable KV-state endpoint. Enabling a listener on Relay does not enable worker KV events.
• A container image built from a revision that includes dynamo.kv_dc_relay, its Rust bindings, and the WAN protocol. Older released images may not contain this module; use the repository’s container build instructions from the same revision.
• The workers’ event-plane settings: direct ZeroMQ (ZMQ) over TCP, or NATS. Only the NATS variant requires a NATS server and its connection credentials.
• Network access to the Kubernetes API and advertised worker event/recovery endpoints. TCP recovery also needs a return path from workers to Relay’s advertised response-stream address. The NATS variant additionally requires access to the workers’ NATS server.
• kubectl, and grpcurl on the machine used for verification.
• Kubernetes support for native gRPC startup and readiness probes.

See Using the Dynamo Frontend for worker KV-event configuration and Runtime Configuration for shared runtime settings.

Discovery Scope

The example assumes an existing Kubernetes namespace named dynamo and watches every Dynamo namespace visible within it. Change the namespace consistently in the commands and RoleBinding if your workers run elsewhere.

DYN_NAMESPACE names Relay’s own runtime endpoints and does not select the watched workers. To narrow the visible logical scope, replace --watch-all with --namespaces <dynamo-namespace>. Use the namespace in the workers’ advertised endpoint identities, not an assumed Kubernetes name.

Before deployment, confirm discovery resources exist:

$
kubectl get crd dynamoworkermetadatas.nvidia.com
$
kubectl -n dynamo get dynamoworkermetadatas
$
kubectl -n dynamo get endpointslices \
>
  -l nvidia.com/dynamo-discovery-backend=kubernetes,nvidia.com/dynamo-discovery-enabled=true

This example uses pod-mode discovery, which joins ready EndpointSlices with worker metadata. Worker Services must carry the discovery labels so their EndpointSlices are watched. Container-mode discovery instead watches labeled Pods and needs Pod get/list/watch permissions; do not mix discovery modes without checking the workers’ registration mode.

Deploy the Relay

Save the following manifest as kv-dc-relay.yaml. Replace REPLACE_WITH_RELAY_IMAGE with your image and set --dc-id to your stable logical data-center name. Add imagePullSecrets if needed. The manifest uses TCP requests and NATS events. For a deployment without NATS, apply the TCP-only settings below before deploying. Otherwise, replace the sample NATS address with the workers’ address and supply any credentials through Secrets.

The manifest runs one replica with Recreate updates. Restarting Relay changes its incarnation and requires consumers to reconnect; this is not an HA deployment. CPU and memory values are starting allocations, not sizing guarantees: pool count and expected unique blocks affect memory.

1
apiVersion: v1
2
kind: ServiceAccount
3
metadata:
4
  name: kv-dc-relay
5
---
6
apiVersion: rbac.authorization.k8s.io/v1
7
kind: Role
8
metadata:
9
  name: kv-dc-relay
10
rules:
11
  - apiGroups: ["nvidia.com"]
12
    resources: ["dynamoworkermetadatas"]
13
    verbs: ["get", "list", "watch", "create", "patch", "delete"]
14
  - apiGroups: ["discovery.k8s.io"]
15
    resources: ["endpointslices"]
16
    verbs: ["get", "list", "watch"]
17
---
18
apiVersion: rbac.authorization.k8s.io/v1
19
kind: RoleBinding
20
metadata:
21
  name: kv-dc-relay
22
subjects:
23
  - kind: ServiceAccount
24
    name: kv-dc-relay
25
    namespace: dynamo
26
roleRef:
27
  apiGroup: rbac.authorization.k8s.io
28
  kind: Role
29
  name: kv-dc-relay
30
---
31
apiVersion: apps/v1
32
kind: Deployment
33
metadata:
34
  name: kv-dc-relay
35
spec:
36
  replicas: 1
37
  strategy:
38
    type: Recreate
39
  selector:
40
    matchLabels:
41
      app.kubernetes.io/name: kv-dc-relay
42
  template:
43
    metadata:
44
      labels:
45
        app.kubernetes.io/name: kv-dc-relay
46
        nvidia.com/dynamo-discovery-backend: kubernetes
47
        nvidia.com/dynamo-discovery-enabled: "true"
48
    spec:
49
      serviceAccountName: kv-dc-relay
50
      terminationGracePeriodSeconds: 60
51
      containers:
52
        - name: relay
53
          image: REPLACE_WITH_RELAY_IMAGE
54
          command: ["python3", "-m", "dynamo.kv_dc_relay"]
55
          args: ["--dc-id", "dc-a", "--watch-all", "--bind", "0.0.0.0:5561"]
56
          env:
57
            - name: DYN_NAMESPACE
58
              value: relay
59
            - name: DYN_DISCOVERY_BACKEND
60
              value: kubernetes
61
            - name: DYN_KUBE_DISCOVERY_MODE
62
              value: pod
63
            - name: DYN_REQUEST_PLANE
64
              value: tcp
65
            - name: DYN_EVENT_PLANE
66
              value: nats
67
            - name: NATS_SERVER
68
              value: nats://nats.dynamo-system.svc.cluster.local:4222
69
            - name: DYN_SYSTEM_PORT
70
              value: "8081"
71
            - name: POD_NAME
72
              valueFrom:
73
                fieldRef:
74
                  fieldPath: metadata.name
75
            - name: POD_NAMESPACE
76
              valueFrom:
77
                fieldRef:
78
                  fieldPath: metadata.namespace
79
            - name: POD_UID
80
              valueFrom:
81
                fieldRef:
82
                  fieldPath: metadata.uid
83
          ports:
84
            - name: grpc
85
              containerPort: 5561
86
            - name: system
87
              containerPort: 8081
88
          startupProbe:
89
            grpc:
90
              port: 5561
91
              service: dynamo.kvrelay.v1.KvEventRelay
92
            periodSeconds: 5
93
            failureThreshold: 60
94
          readinessProbe:
95
            grpc:
96
              port: 5561
97
              service: dynamo.kvrelay.v1.KvEventRelay
98
            periodSeconds: 10
99
          resources:
100
            requests:
101
              cpu: "1"
102
              memory: 1Gi
103
            limits:
104
              memory: 4Gi
105
---
106
apiVersion: v1
107
kind: Service
108
metadata:
109
  name: kv-dc-relay
110
  labels:
111
    nvidia.com/dynamo-discovery-backend: kubernetes
112
    nvidia.com/dynamo-discovery-enabled: "true"
113
spec:
114
  type: ClusterIP
115
  selector:
116
    app.kubernetes.io/name: kv-dc-relay
117
  ports:
118
    - name: grpc
119
      port: 5561
120
      targetPort: grpc

Relay needs write access to worker metadata because its runtime registers its own endpoints. The Role is namespaced; it does not grant cross-namespace discovery. This example assumes a trusted cluster network. The Service exposes plaintext gRPC without authentication; ClusterIP does not itself restrict which pods can connect. Use local port-forwarding to inspect runtime diagnostic ports, which are not included in the Service.

TCP-Only Local Planes (No NATS)

Use Kubernetes discovery, TCP requests, and direct ZMQ events for a deployment without a messaging server. DYN_EVENT_PLANE=zmq selects ZeroMQ over TCP; tcp is not an event-plane value. This is independent of the WAN Protobuf/gRPC listener on 5561.

In the Relay container’s env list, replace the request/event-plane entries with the following and remove NATS_SERVER. Keep the other environment entries from the manifest:

1
- name: DYN_REQUEST_PLANE
2
  value: tcp
3
- name: DYN_EVENT_PLANE
4
  value: zmq
5
- name: DYN_TCP_RESPONSE_STREAM_HOST
6
  valueFrom:
7
    fieldRef:
8
      fieldPath: status.podIP
9
- name: DYN_TCP_RESPONSE_STREAM_PORT
10
  value: "5562"

The fixed response-stream port is optional; without it, the runtime allocates a free port. Port 5562 is used for worker responses to Relay’s runtime requests, not for WAN subscriptions. It does not need to be added to the WAN Service: workers connect to the advertised pod address.

Workers and other local event consumers must also use DYN_EVENT_PLANE=zmq. For an entirely TCP-based request path, configure workers with DYN_REQUEST_PLANE=tcp. Changing Relay alone does not migrate worker publishers. Keep worker KV-event publication enabled, and do not register a ZMQ broker for these scopes when using direct mode.

Relay discovers direct ZMQ publishers through Kubernetes metadata and connects to their advertised TCP addresses. Allow those pod-to-pod connections, worker recovery requests, and the return path to Relay’s response port. Publisher ports may be dynamically allocated; opening only 5561 is insufficient. On multi-interface workers, set DYN_EVENT_PLANE_HOST to a reachable pod IP if automatic selection advertises the wrong address. It changes the advertised address, not the listener’s bind address; use routable IPv4 addresses for direct ZMQ.

Discovery RBAC, the WAN Service, optional mTLS sidecar, and the gRPC checks below are unchanged.

Apply the Manifest

$
kubectl -n dynamo apply -f kv-dc-relay.yaml
$
kubectl -n dynamo rollout status deployment/kv-dc-relay --timeout=300s
$
kubectl -n dynamo logs deployment/kv-dc-relay --tail=100

Verify Discovery and Published Metadata

Forward the listener to your machine:

$
kubectl -n dynamo port-forward service/kv-dc-relay 5561:5561

In another terminal, query the protocol identity. The decimal marker below is KVR1 (0x4B565231); it is required by every Relay request.

$
grpcurl -plaintext -d '{"contractMarker":1263948337}' \
>
  localhost:5561 dynamo.kvrelay.v1.KvEventRelay/GetRelayInfo

Expect this shape; identity values vary per deployment and restart:

1
{
2
  "protocolVersion": 1,
3
  "relay": {"drtInstanceId": "123", "relayIncarnation": "456"},
4
  "contractMarker": 1263948337
5
}

Then inspect catalog and readiness. Each command opens a stream; stop it with Ctrl-C after the first update.

$
grpcurl -plaintext -max-msg-sz 8388608 \
>
  -d '{"contractMarker":1263948337,"subscriberId":"deployment-check-catalog"}' \
>
  localhost:5561 dynamo.kvrelay.v1.KvEventRelay/WatchKvPoolCatalog

$
grpcurl -plaintext -max-msg-sz 8388608 \
>
  -d '{"contractMarker":1263948337,"subscriberId":"deployment-check-readiness"}' \
>
  localhost:5561 dynamo.kvrelay.v1.KvEventRelay/SubscribeServingReadiness

Check these fields in the responses:

For a ready disaggregated model, expect separate Prefill and Decode pools but one readiness entry with both roles. LoRA readiness appears under the base entry’s adapters, not as another top-level entry. Catalog and readiness revisions are independent.

An empty catalog does not verify discovery; a passing pod probe does not prove model readiness. These checks expose metadata, not CKF contents. CKF validation requires subscribing to an advertised producer and validating its complete CBI1 snapshot and subsequent deltas.

Expose the WAN Listener

The example Service is cluster-internal. A trusted in-cluster consumer can use kv-dc-relay.dynamo.svc.cluster.local:5561. Do not turn this plaintext Service into an unrestricted LoadBalancer or expose the pod port to another data center directly.

For access across a trust boundary, terminate TLS in an external proxy and route only its protected listener through your network ingress. Keep its upstream connection HTTP/2 and allow long-lived server streams. See the gRPC contract for message sizes, reconnect behavior, and error reasons.

Optional mTLS Sidecar

Mutual TLS (mTLS) is optional and implemented outside Relay. Relay has no built-in TLS configuration, certificate loading, or authentication. For a protected deployment:

1. Change Relay’s bind address to 127.0.0.1:5561.
2. Add a gRPC-capable sidecar that accepts authenticated TLS connections on a separate pod port and forwards HTTP/2 to 127.0.0.1:5561, without retries or buffering.
3. Mount the sidecar’s certificate, key, and trust bundle from Secrets; configure client authorization, certificate rotation, and expiry monitoring in the sidecar.
4. Point the Service at the proxy port only; keep Relay’s port 5561 bound to loopback.
5. Replace the pod-IP gRPC probes from the example: they cannot reach a loopback-only listener. Use probes suitable for your proxy and a local Relay check; native Kubernetes gRPC probes do not authenticate through mTLS.

The sidecar’s image and configuration depend on your organization’s proxy and PKI.

For local inspection after this change, forward directly to Relay’s loopback listener:

$
kubectl -n dynamo port-forward deployment/kv-dc-relay 5561:5561

The plaintext grpcurl commands above still apply. This checks Relay through the Kubernetes tunnel, not the externally exposed mTLS path.

Troubleshooting

Clean Up

Remove only the resources created by this guide; retain the existing workers and namespace:

$
kubectl -n dynamo delete -f kv-dc-relay.yaml

For all CLI and tuning options, see Multi-Datacenter KV Relay Configuration.