> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # TLS Dynamo supports opt-in TLS encryption on the network transports between its components: the TCP request and response streams between frontends and workers, and connections to NATS. On the TCP transport, both the **request plane** (frontend → worker inference requests) and the **response stream** (worker → frontend inference output) are encrypted using [rustls](https://github.com/rustls/rustls) with the `ring` cryptographic provider. When no TLS configuration is provided, these transports operate in plaintext exactly as before. The `DYN_TCP_TLS_*` environment variables encrypt the frontend↔worker TCP request and response streams; NATS traffic is encrypted separately via NATS TLS (see the NATS TLS section below). The KV event plane is a separate transport: when it runs over ZMQ it is **not** encrypted, and when it runs over NATS it is encrypted only if NATS TLS is configured. ## Environment variables All TLS configuration is driven by environment variables. The Rust runtime reads these directly at first connection (lazy initialization). Both frontends and workers act as TCP server and client depending on the stream direction (response streams: worker dials frontend; request streams: frontend dials worker). All TLS env vars should be set on every pod. ### Server role (accepting connections) | Variable | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------ | | `DYN_TCP_TLS_CERT_PATH` | Path to the PEM certificate file. When set together with `DYN_TCP_TLS_KEY_PATH`, TLS is enabled on the TCP server. | | `DYN_TCP_TLS_KEY_PATH` | Path to the PEM private key for the server certificate. | ### Client role (dialing connections) | Variable | Description | | ------------------------------------ | -------------------------------------------------------------------------------------------------------- | | `DYN_TCP_TLS_CA_CERT_PATH` | Path to the PEM CA certificate used to verify the peer's server certificate. | | `DYN_TCP_TLS_INSECURE` | Set to `1` or `true` to skip certificate verification. For local development only. | | `DYN_TCP_TLS_SERVER_NAME` | Override the TLS SNI hostname. Useful when connecting by IP to a server whose certificate has a DNS SAN. | | `DYN_TCP_TLS_HANDSHAKE_TIMEOUT_SECS` | TLS handshake timeout in seconds (default: 3). | ## CLI flags The same configuration is available via command-line flags on all backends (vllm, sglang, trtllm, tokenspeed) through `DynamoRuntimeArgGroup`: ``` --tcp-tls-cert-path PATH Server certificate (PEM) --tcp-tls-key-path PATH Server private key (PEM) --tcp-tls-ca-cert-path PATH CA certificate for server verification (PEM) --tcp-tls-insecure Disable certificate verification --tcp-tls-server-name NAME Override TLS SNI hostname --tcp-tls-handshake-timeout N Handshake timeout in seconds (default: 3) ``` The frontend (`dynamo.frontend`) also accepts `--tcp-tls-cert-path`, `--tcp-tls-key-path`, and `--tcp-tls-ca-cert-path`. ## Quick start Generate a self-signed certificate for local testing: ```bash # Generate CA openssl req -x509 -newkey rsa:2048 -keyout ca-key.pem -out ca-cert.pem \ -days 365 -nodes -subj "/CN=DynamoCA" # Generate server cert with SAN openssl req -newkey rsa:2048 -keyout server-key.pem -out server-csr.pem \ -nodes -subj "/CN=localhost" \ -addext "subjectAltName=DNS:localhost,IP:127.0.0.1" openssl x509 -req -in server-csr.pem -CA ca-cert.pem -CAkey ca-key.pem \ -CAcreateserial -out server-cert.pem -days 365 -copy_extensions copyall ``` Both frontend and worker need the same flags (both act as server and client): ```bash python -m dynamo.vllm \ --tcp-tls-cert-path server-cert.pem \ --tcp-tls-key-path server-key.pem \ --tcp-tls-ca-cert-path ca-cert.pem \ --tcp-tls-server-name localhost \ ... python -m dynamo.frontend \ --tcp-tls-cert-path server-cert.pem \ --tcp-tls-key-path server-key.pem \ --tcp-tls-ca-cert-path ca-cert.pem \ --tcp-tls-server-name localhost \ ... ``` ## Kubernetes deployment In Kubernetes, TLS certificates are typically delivered by a certificate management system and mounted into pods. Set the environment variables on each component's pod template in the `DynamoGraphDeployment` spec: ```yaml spec: components: - name: Frontend podTemplate: spec: containers: - name: main env: - name: DYN_TCP_TLS_CERT_PATH value: /etc/certs/server/cert.pem - name: DYN_TCP_TLS_KEY_PATH value: /etc/certs/server/key.pem - name: DYN_TCP_TLS_CA_CERT_PATH value: /etc/certs/ca/ca.pem - name: VllmWorker podTemplate: spec: containers: - name: main env: - name: DYN_TCP_TLS_CERT_PATH value: /etc/certs/server/cert.pem - name: DYN_TCP_TLS_KEY_PATH value: /etc/certs/server/key.pem - name: DYN_TCP_TLS_CA_CERT_PATH value: /etc/certs/ca/ca.pem ``` Both components need the same TLS env vars because each acts as both TCP server and client depending on the stream direction. For platform-wide TLS that the operator injects into every deployment automatically — instead of setting env vars on each component — see [Operator TLS](/dynamo/kubernetes/installation/operator-tls). ## NATS TLS NATS carries JetStream indexer recovery/replay, the audit sink, and — when the request plane is set to NATS (`--request-plane nats`) — inference request distribution. TLS is configured separately from TCP: | Variable | Description | | ----------------------- | ----------------------------------------------------------------------------------- | | `NATS_TLS_CA_CERT_PATH` | CA certificate to verify the NATS server. When set, a custom TLS config is applied. | | `NATS_TLS_INSECURE` | Skip NATS server certificate verification (dev only). | When only the `tls://` URL scheme is used without explicit TLS env vars, async-nats handles TLS natively with system roots. The `NATS_SERVER` URL accepts both `nats://` and `tls://` schemes, but setting any explicit NATS TLS variable (`NATS_TLS_*`) requires the `tls://` scheme — startup fails with a clear error otherwise. CLI flags (all backends via `DynamoRuntimeArgGroup`): ```text --nats-tls-ca-cert-path PATH CA certificate for NATS server verification (PEM) --nats-tls-insecure Disable NATS certificate verification ``` The frontend (`dynamo.frontend`) also accepts `--nats-tls-ca-cert-path` and `--nats-tls-insecure`. ### Enabling TLS on the NATS server The env vars above configure the **client** side (Dynamo verifying the NATS server). For TLS to work, the NATS server itself must also be configured to listen on TLS — otherwise a client that sets `NATS_TLS_CA_CERT_PATH` (or the operator-level `natsTLSCAPath`) will attempt a TLS handshake against a plaintext port and fail. When deploying Dynamo's platform chart, the bundled NATS subchart exposes this via `nats.config.nats.tls`. The certificates are typically delivered by a certificate management system such as cert-manager. **One-way TLS** (server presents a cert, clients verify it): ```yaml # NATS server side — platform chart values nats: config: nats: tls: enabled: true secretName: nats-server-tls # Secret with tls.crt / tls.key (e.g. cert-manager) ``` Or, if the cert files are mounted into the pod by an external system, use the `merge` block to point at the on-disk paths directly: ```yaml nats: config: nats: tls: enabled: true merge: cert_file: /etc/certs/server/cert.pem key_file: /etc/certs/server/key.pem timeout: 2 ``` Then point the Dynamo clients at the TLS endpoint and give them the CA. These are operator-subchart values, so nest them under `dynamo-operator:` when using the platform chart: ```yaml dynamo-operator: natsAddr: "tls://dynamo-platform-nats.dynamo-system.svc.cluster.local:4222" natsTLSCAPath: /etc/certs/ca/ca.pem # → NATS_TLS_CA_CERT_PATH ``` **mTLS** (server also verifies client certs) — add `ca_file` and `verify` to the server's `merge` block, and present a client identity from the Dynamo side: ```yaml # NATS server side nats: config: nats: tls: enabled: true merge: cert_file: /etc/certs/server/cert.pem key_file: /etc/certs/server/key.pem ca_file: /etc/certs/client-ca/ca.pem # verify client certs verify: true # require client certs ``` ```yaml # Dynamo client side (operator-level, nested under dynamo-operator:) dynamo-operator: natsTLSCAPath: /etc/certs/ca/ca.pem natsTLSClientCertPath: /etc/certs/client/cert.pem natsTLSClientKeyPath: /etc/certs/client/key.pem ``` See the [NATS TLS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) for the full server-side TLS schema. ## Mutual TLS (mTLS) By default only the server is authenticated (the client verifies the server's certificate). Mutual TLS additionally makes the **client present a certificate** that the **server verifies**, so both ends of a connection prove their identity. mTLS is opt-in and layered on top of the TLS configuration above. For the **TCP** transports (request plane + response stream) Dynamo owns both ends, so configure two things and — since every component is both TCP client and server — set both on every pod: * **On the client:** a client certificate/key to present. * **On the server:** a CA certificate (`DYN_TCP_TLS_CLIENT_CA_CERT_PATH`) to verify the presented client certificate. When set, the server **requires** a trusted client certificate and rejects the handshake otherwise. For **NATS** the broker is external, so Dynamo only presents a client identity — there is no Dynamo-side "server" half. Whether client certificates are actually required is enforced by the **NATS server** configuration (e.g. `tls { ca_file: ...; verify: true }`), not by Dynamo. ### TCP request/response streams | Variable | Role | Description | | --------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------- | | `DYN_TCP_TLS_CLIENT_CERT_PATH` | Client | PEM client certificate presented to the server. Set with `DYN_TCP_TLS_CLIENT_KEY_PATH`. | | `DYN_TCP_TLS_CLIENT_KEY_PATH` | Client | PEM private key for the client certificate. | | `DYN_TCP_TLS_CLIENT_CA_CERT_PATH` | Server | PEM CA used to verify client certificates. When set, clients **must** present a certificate signed by this CA. | ### NATS | Variable | Role | Description | | --------------------------- | ------ | ----------------------------------------------------------------------------------------- | | `NATS_TLS_CLIENT_CERT_PATH` | Client | PEM client certificate presented to the NATS server. Set with `NATS_TLS_CLIENT_KEY_PATH`. | | `NATS_TLS_CLIENT_KEY_PATH` | Client | PEM private key for the NATS client certificate. | > **Note:** Dynamo cannot detect whether the NATS broker actually requests a > client certificate. If you set `NATS_TLS_CLIENT_CERT_PATH`/`_KEY_PATH` but the > broker is not configured to verify client certs, the connection silently > falls back to ordinary one-way TLS. Enable verification on the NATS server > (`verify: true`) to make NATS mTLS effective. Client certificate and key must be set **together**, and a client identity requires a server CA (`*_CA_CERT_PATH`) — or insecure mode for development — so the server can still be verified. The TCP **servers** and the **NATS** client validate this at startup and fail closed. The TCP **client** connectors, however, initialize lazily on the first outbound connection, so an incomplete TCP client config surfaces as a per-connection handshake failure rather than a startup crash. The client certificate/key hot-reload from disk on rotation, exactly like the server certificate (see Design notes). The same options are available as CLI flags on all backends (via `DynamoRuntimeArgGroup`) and the frontend (`dynamo.frontend`): ```text --tcp-tls-client-cert-path PATH Client certificate presented for TCP mTLS (PEM) --tcp-tls-client-key-path PATH Client private key for TCP mTLS (PEM) --tcp-tls-client-ca-cert-path PATH CA to verify client certs; enforces TCP mTLS (PEM) --nats-tls-client-cert-path PATH Client certificate presented for NATS mTLS (PEM) --nats-tls-client-key-path PATH Client private key for NATS mTLS (PEM) ``` ## Encrypted paths When TLS is configured, the following transports are encrypted (the ZMQ event plane is not covered): | Path | Direction | Data | Transport | | --------------- | ---------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------- | | Request plane | Frontend → Worker | User prompts, request metadata | `egress/tcp_client` → `ingress/shared_tcp_endpoint` | | Response stream | Worker → Frontend | Inference output tokens | `tcp/client` → `tcp/server` | | Request stream | Frontend → Worker | Streaming input (bidirectional) | `tcp/client` → `tcp/server` | | NATS | Frontend/Worker ↔ NATS | Inference requests (when `--request-plane nats`), JetStream recovery, audit logs | `transports/nats` | ## Design notes * Server certificates hot-reload: the request-plane and response-stream servers serve their leaf cert/key through a resolver that re-reads the files from disk when their contents change (detected by a content hash, so rotations done by an atomic symlink swap are handled too), so certificate rotation takes effect **without a process restart**. The check is rate-limited (at most once every 30s, sooner after a failed reload) and never blocks a handshake; a failed reload keeps serving the last valid certificate. Client trust anchors (the CA) are still loaded once, so rotating the CA itself requires a restart. * mTLS **client identity** certificates hot-reload through the same resolver, so a rotated client cert/key is picked up without a restart. The CA used by the server to verify client certificates is loaded once (rotating it needs a restart). * Client TLS connectors are built once and cached via `OnceCell` on the first outbound connection. * The TLS handshake is spawned per-connection on both the request plane and response stream servers so the accept loop is never blocked. * Invalid TLS configuration on the request plane (e.g. bad cert path) prevents server startup rather than silently falling back to plaintext. * When server and client TLS configurations are mismatched (e.g., server has TLS but client does not), a warning is logged at startup. * An empty CA certificate file is detected at load time and rejected with a clear error message.