LLM Function Enablement

Enable the LLM addon before creating or invoking functions with functionType: "LLM" through the LLM invocation route. The addon deploys the LLM API Gateway and LLM request router, creates the external LLM invocation route, and configures worker pods to use the pylon sidecar for model-aware routing.

For LLM function payload shape and invocation examples, see Function Creation and LLM Gateway. For request-router deployment, trusted headers, and rollout validation, see LLM Request Router Load Balancing.

When to Enable

Enable the LLM addon when NVCF should route OpenAI-compatible requests by function and model through llm.invocation.<domain>. The gateway extracts the function ID from the OpenAI model field, applies LLM-specific validation and rate limits, and sends the request through the LLM request router.

Standard HTTP, gRPC, and LLS functions do not require this addon, even when a container exposes paths such as /v1/chat/completions, /v1/responses, or /v1/embeddings.

When enabled, the stack creates:

• llm-api-gateway in the nvcf namespace.
• llm-request-router in the nvcf namespace.
• The llm.invocation.<domain> HTTPRoute when Gateway API ingress is enabled.
• LLM worker pods with a pylon sidecar that forwards requests to the function container on the configured inferencePort.

Production TLS Configuration

Production deployments secure two independent worker-to-router paths:

The current production gRPC design uses a public ACM certificate on the NLB TLS listener. The NLB certificate covers the public gRPC dial hostname. An unmanaged Pylon process, or one using NVCA system trust mode, can leave both trust paths unset so Tonic uses its enabled system and public roots. In reverse mode, Pylon reuses --tls-cert-path as gRPC trust when the gRPC-specific override is unset. NVCA bundle mode points both options at its merged CA file.

--grpc-tls-ca-cert-path or STARGATE_GRPC_TLS_CA_CERT_PATH is the optional gRPC-specific override. A configured custom bundle augments the enabled system and public roots. It does not replace them. In direct mode, --tls-cert-path is the Pylon QUIC server identity and is never treated as a gRPC CA bundle.

A private-CA certificate on the NLB listener is a supported alternative. Give Pylon the CA bundle with --grpc-tls-ca-cert-path <path> or STARGATE_GRPC_TLS_CA_CERT_PATH=<path>. The input is a PEM bundle containing the CA certificates required by the selected certificate chain, including a private root when applicable. It does not disable certificate or hostname verification. Pylon reads the file once during startup. An unreadable path fails startup with the configured path in the error. Invalid PEM, untrusted chains, and hostname mismatches prevent the gRPC watch and registration connections. Replace or rotate the bundle with a rolling restart of the worker pods. The pylonGrpcDialAddress override must be an explicit https:// URI when a custom CA is configured. The separate global.workerEndpoints.llmRequestRouterAddress input remains a scheme-less host:port initial address.

For gRPC, TLS SNI and hostname verification always use the external HTTPS dial hostname. After discovery, Pylon separately sends the concrete request-router pod hostname as the HTTP/2 :authority so stargate-k8s-router can select the pod. The authority override does not become the TLS SNI. A gRPC NLB leaf certificate therefore needs the public NLB DNS SAN, not Kubernetes-internal pod SANs.

The QUIC identity remains separate. The request router presents a certificate issued by cert-manager, or one you issue and supply in a pre-created Secret. For a single-cluster deployment, workers dial llm-request-router.nvcf.svc.cluster.local:50071. For remote workers, the backend router preserves the advertised request-router pod hostname as the QUIC SNI while it sends traffic through an external UDP endpoint. The QUIC certificate must cover that advertised hostname. The default wildcard SAN is *.llm-request-router-headless.nvcf.svc.cluster.local. It does not need the external UDP load-balancer hostname. Do not reuse the gRPC NLB leaf certificate as the QUIC leaf certificate.

Managed OpenBao issuer

The default managed configuration installs ClusterIssuer/nvcf-openbao-pki through the helm-nvcf-pki chart. It uses the stack’s OpenBao service as the signing backend.

Add the following values to the Helmfile environment:

1
certManager:
2
  enabled: true
3
4
openbao:
5
  enabled: true
6
7
addons:
8
  llm:
9
    enabled: true
10
    pki:
11
      enabled: true
12
      allowedDomains: nvcf.svc.cluster.local
13
      dnsNames:
14
        - llm-request-router.nvcf.svc.cluster.local
15
        - "*.llm-request-router-headless.nvcf.svc.cluster.local"
16
    gateway:
17
      replicaCount: 2
18
    requestRouter:
19
      replicaCount: 2

The stable Service SAN covers the in-cluster Service name. The wildcard SAN covers the pod-specific headless names advertised through the backend-router path.

To use a custom managed ClusterIssuer, set both the issuer identity and the management override:

1
addons:
2
  llm:
3
    pki:
4
      issuerKind: ClusterIssuer
5
      issuerName: custom-openbao-pki
6
      clusterIssuer:
7
        enabled: true

Issuer management supports only issuerKind: ClusterIssuer. OpenBao must be enabled when the stack manages the issuer.

External issuer

Set clusterIssuer.enabled: false when the issuer is managed outside this stack. For an external ClusterIssuer, use:

1
addons:
2
  llm:
3
    enabled: true
4
    pki:
5
      enabled: true
6
      issuerKind: ClusterIssuer
7
      issuerName: external-llm-pki
8
      clusterIssuer:
9
        enabled: false
10
      dnsNames:
11
        - llm-request-router.nvcf.svc.cluster.local
12
        - "*.llm-request-router-headless.nvcf.svc.cluster.local"

For a namespaced issuer, create the Issuer in the nvcf namespace and use:

1
addons:
2
  llm:
3
    enabled: true
4
    pki:
5
      enabled: true
6
      issuerKind: Issuer
7
      issuerName: external-llm-pki
8
      clusterIssuer:
9
        enabled: false
10
      dnsNames:
11
        - llm-request-router.nvcf.svc.cluster.local
12
        - "*.llm-request-router-headless.nvcf.svc.cluster.local"

The external issuer must allow the requested DNS names and issue a server certificate from the root CA distributed to the compute planes. You can set openbao.enabled: false when no other stack component requires OpenBao.

addons.llm.pki.allowedDomains constrains the managed OpenBao signing role only. The stack ignores it for an external issuer. Apply the equivalent constraint in the external issuer’s own configuration.

When addons.llm.enabled is true, the stack defaults global.workerEndpoints.llmRequestRouterAddress to llm-request-router.nvcf.svc.cluster.local:50071. Colocated workers require no additional configuration. For a split control-plane and compute-plane deployment, override this value with a host and port that worker pods can reach.

The stack maps the configured or default address to api.remoteConfig.configData.nvcf.llm-request-router.worker-address. The NVCF API then includes the address in LLM worker configuration. Do not configure the worker address under api.env. When the LLM addon is disabled, the stack does not pass a staged endpoint to the API chart.

Remote compute clusters and regions

One self-managed control plane can serve LLM workers in separate GPU clusters or regions when every worker cluster has routable DNS and network paths to both backend-router endpoints.

Configure distinct HTTPS and UDP endpoints when the infrastructure uses separate Gateways or load balancers:

1
global:
2
  workerEndpoints:
3
    llmRequestRouterAddress: https://llm-grpc.example.com:50071
4
5
addons:
6
  llm:
7
    enabled: true
8
    pki:
9
      # Include the external gRPC suffix when using the stack-managed issuer.
10
      allowedDomains: cluster.local,example.com
11
    requestRouter:
12
      grpcTls:
13
        enabled: true
14
        mode: certManager
15
        secretName: llm-grpc-tls
16
        dnsNames:
17
          - llm-grpc.example.com
18
        issuerRef:
19
          kind: ClusterIssuer
20
          name: nvcf-openbao-pki
21
      backendRouter:
22
        pylonGrpcDialAddress: https://llm-grpc.example.com:50071
23
        pylonReverseTunnelDialAddress: llm-quic.example.com:50072
24
25
ingress:
26
  gatewayApi:
27
    routes:
28
      llmWorker:
29
        enabled: true
30
        backend:
31
          namespace: nvcf
32
    gateways:
33
      llmGrpc:
34
        name: llm-grpc-gateway
35
        namespace: envoy-gateway
36
        listenerName: llm-grpc
37
      llmQuic:
38
        name: llm-quic-gateway
39
        namespace: envoy-gateway
40
        listenerName: llm-quic

Set both pylonGrpcDialAddress and pylonReverseTunnelDialAddress, or omit both to use the in-cluster backend-router Service. Helmfile rendering rejects a partial override. The gRPC worker address normally uses the same HTTPS endpoint as pylonGrpcDialAddress. A secure external route requires the same explicit https:// URI in both values and grpcTls.enabled: true. Development-only plaintext requires http:// in both values and grpcTls.allowInsecureHttp: true; a scheme-less external route is rejected. Port 443 alone does not make a connection secure.

To recursively discover request routers in another region, set explicit remote Watch dial URIs on the self-managed operator surface:

1
addons:
2
  llm:
3
    requestRouter:
4
      discovery:
5
        remoteWatchUrls:
6
          - https://region-b-watch.example.com:50071

Each URI must use https://. Development-only plaintext endpoints require an explicit http:// URI and addons.llm.requestRouter.discovery.allowInsecureRemoteWatchHttp: true. Scheme-less and unsupported endpoints are rejected rather than defaulting to plaintext. For an HTTPS URI, the dial hostname selects TLS SNI. Identities advertised by the remote Watch response remain the HTTP/2 authorities used for registration.

With grpcTls.mode: certManager, the gateway-routes chart creates a dedicated Certificate in the gRPC Gateway namespace. With mode: existingSecret, the operator must create the named TLS Secret in that namespace. In both modes the Gateway HTTPS listener must reference the same Secret. This identity is separate from the QUIC certificate on port 50072. The operator must ensure an existing certificate covers the external dial hostname; dnsNames is required only when the chart requests the certificate.

For an HTTPS dial URI, the gRPC dial hostname is the TLS SNI and must be a SAN on the Gateway listener certificate. It does not replace the advertised request-router pod hostname used as the HTTP/2 authority or as the QUIC SNI. The dedicated HTTPS listener and LLM GRPCRoute therefore do not set hostname or hostnames. The listener serves the dedicated certificate for normal public-SNI verification, while the backend router receives the selected Stargate identity in :authority. Setting the listener hostname to the public dial name would also constrain :authority and reject these streams. Envoy Gateway’s route-scoped traffic policy disables the request and maximum stream durations for long-lived Watch and Register streams. Keep the default wildcard SAN on the separate QUIC leaf, or issue a certificate that covers a customized advertised hostname. Every compute cluster must trust the required issuing CAs as described in Compute-plane trust. Server certificate and key updates follow Certificate Renewal. A CA or trust-bundle change requires a worker rollout.

The stack does not provision cross-region networking, firewall rules, load balancers, DNS, or trust distribution. Supply those dependencies and register the trust bundle in every compute cluster. A successful Helm render and accepted Gateway routes do not prove remote TCP or UDP reachability. Test both paths and an LLM invocation from each region before production use.

The request router uses power-of-two when no load-balancer configuration is set, and accepts any supported routingMethod from a function. When a load-balancer configuration is set, a function can only select an algorithm that the configuration enables.

Pre-created request-router Secret

Set mode: existingSecret when you already issue the request-router server certificate yourself and want the stack to mount it without managing issuance:

1
addons:
2
  llm:
3
    enabled: true
4
    pki:
5
      enabled: true
6
      mode: existingSecret
7
      secretName: stargate-quic-tls

The stack renders no Certificate, installs no ClusterIssuer, and adds no cert-manager or OpenBao dependency for the request router. You can set certManager.enabled: false and openbao.enabled: false when no other stack component needs them.

Create the Secret in the nvcf namespace before installing the stack. It must carry the tls.crt and tls.key entries, as a kubernetes.io/tls Secret does:

$
kubectl create secret tls stargate-quic-tls \
>
  --namespace nvcf \
>
  --cert path/to/tls.crt \
>
  --key path/to/tls.key

clusterIssuer.enabled, dnsNames, and allowedDomains only steer stack-managed issuance. Rendering fails if any of them is set in this mode, so a configuration that expects the stack to issue a certificate cannot be mistaken for one that expects you to.

The certificate must carry a SAN covering the router’s advertised hostname. The stack enables backend routing and advertises per-pod headless names, so include *.llm-request-router-headless.nvcf.svc.cluster.local along with the stable llm-request-router.nvcf.svc.cluster.local name. An external TCP or UDP dial hostname does not need to be a SAN unless the advertised identity is also customized to use that hostname. The stack cannot read your Secret at render time, so it validates neither the SANs nor the expiry. A certificate that does not cover the advertised hostname fails at worker connection time, not at install time.

You own issuance, renewal, rotation, and recovery in this mode:

• Renewal and rotation: hot reload requires helm-nvcf-llm-request-router 1.10.0 or later and Stargate 0.11.1 or later. With those versions, update the Secret and follow the transport TLS rotation runbook. Older chart releases and Stargate image overrides older than 0.11.1 require a request-router restart after the Secret update.
• Expiry: track it yourself. Nothing in the stack renews the certificate or alerts on an approaching expiry.
• Recovery: a running router keeps its last-known-good identity after rejecting a malformed replacement. A new pod cannot start from a missing or malformed Secret. Restore a valid Secret and follow the rotation runbook to verify the active identity.

Compute-plane trust works exactly as described in Compute-plane trust. Author the transportTls block in the control-plane profile by hand with the public root CA that signed your certificate, as you would for an external issuer. The profile exporter sources a bundle only from the managed OpenBao hierarchy, so it leaves the block empty here. The bundle must contain CERTIFICATE blocks only. Profile validation rejects a private key, so the request-router private key never reaches the compute plane.

External cert-manager

Set certManager.enabled: false when cert-manager is installed and managed outside this stack:

1
certManager:
2
  enabled: false

Install the cert-manager CRDs and controller before applying the NVCF stack. When the stack manages the OpenBao issuer, the external installation must also provide ServiceAccount/cert-manager in the cert-manager namespace because the managed issuer uses Kubernetes authentication with that identity.

Compute-plane trust

The control-plane profile carries the public transport CA:

1
transportTls:
2
  trustMode: bundle
3
  trustBundleFingerprint: sha256:<64-lowercase-hex-digits>
4
  trustBundlePem: |
5
    -----BEGIN CERTIFICATE-----
6
    <public-root-ca-pem-body>
7
    -----END CERTIFICATE-----

For the managed OpenBao issuer, export a refreshed profile after the managed PKI hierarchy is available:

$
nvcf-cli self-hosted \
>
  --control-plane-stack deploy/stacks/self-managed \
>
  --env <environment-name> \
>
  --control-plane-context <control-plane-context> \
>
  --compute-plane-context <compute-plane-context> \
>
  control-plane profile export \
>
  --cluster-name <control-plane-cluster-name> \
>
  --nca-id <nca-id> \
>
  --region <region>

The exporter cannot infer the worker-facing request-router endpoint. Add the reachable host:port before validating or registering the profile:

1
controlPlane:
2
  addons:
3
    llm:
4
      requestRouterAddress: llm-router.example.com:443

Use the gRPC endpoint that compute-plane workers can resolve and reach. This is a dial endpoint. It does not replace the advertised request-router hostname that the certificate covers.

Registration renders this field as agent.llm.requestRouterAddress in the compute-plane values. That is operator configuration, not a runtime fallback for workers. At this release the worker sidecar takes its --stargate-address from the LLM_REQUEST_ROUTER_ADDRESS variable in its launch environment, with STARGATE_ADDRESS accepted as a legacy alias. The NVCF API injects LLM_REQUEST_ROUTER_ADDRESS on the normal launch path, derived from global.workerEndpoints.llmRequestRouterAddress. If neither variable reaches the workload, translation rejects the launch instead of falling back to the registered address.

Keep the default stable and wildcard entries in addons.llm.pki.dnsNames. Add another SAN only when the advertised request-router identity changes, not when only a load-balancer dial endpoint changes. For the managed issuer, any new advertised DNS suffix must also be covered by allowedDomains. The OpenBao role allows subdomains and wildcards but not bare domains, so configure the parent domain rather than the exact leaf name.

The managed export reads the public root CA certificate from services/all/pki/root in the stack’s OpenBao service and calculates the canonical NVCF trust-bundle fingerprint. It does not discover the CA for an external issuer. For an external issuer, add the issuer owner’s public CA bundle and canonical NVCF fingerprint to transportTls in the existing profile.

Calculate the canonical fingerprint for a public CA bundle with:

$
set -euo pipefail
$
$
bundle_file=path/to/public-ca-bundle.pem
$
fingerprint_tmp="$(mktemp -d)"
$
trap 'rm -rf "$fingerprint_tmp"' EXIT
$
$
awk -v output_dir="$fingerprint_tmp" '
>
  /-----BEGIN CERTIFICATE-----/ { certificate++; in_certificate=1 }
>
  in_certificate {
>
    print > (output_dir "/certificate-" certificate ".pem")
>
  }
>
  /-----END CERTIFICATE-----/ { in_certificate=0 }
>
' "$bundle_file"
$
$
for certificate_file in "$fingerprint_tmp"/certificate-*.pem; do
$
  openssl x509 -in "$certificate_file" -outform DER \
>
    | openssl dgst -sha256 -r \
>
    | awk '{print $1}'
$
done | sort -u >"$fingerprint_tmp/certificate-hashes"
$
$
{
>
  printf 'nvcf-trust-bundle-v1\n'
>
  cat "$fingerprint_tmp/certificate-hashes"
>
} | openssl dgst -sha256 -r \
>
  | awk '{print "sha256:" $1}'

The procedure hashes each certificate’s DER bytes, removes duplicates, sorts the certificate hashes, and hashes the versioned canonical text. Use the result for trustBundleFingerprint.

Validate the profile after updating it:

$
nvcf-cli self-hosted control-plane profile validate \
>
  --file deploy/stacks/self-managed/out/control-plane-profile.yaml \
>
  --require compute-reachable

Compute-plane registration converts transportTls into this exact generated NVCA fragment:

1
agentConfig:
2
  mergeConfig: |
3
    workload:
4
      transportTLS:
5
        trustMode: bundle
6
        trustBundleFingerprint: sha256:<64-lowercase-hex-digits>
7
        trustBundlePem: |
8
          -----BEGIN CERTIFICATE-----
9
          <public-root-ca-pem-body>
10
          -----END CERTIFICATE-----

Use only public CA certificates in trustBundlePem. Do not add a private key, leaf certificate, or OpenBao token.

In bundle mode, NVCA creates one merged system and private CA file in the llm-worker container and explicitly points both Pylon trust inputs at it:

STARGATE_TLS_CERT_PATH=/etc/ssl/certs/ca-certificates.crt
STARGATE_GRPC_TLS_CA_CERT_PATH=/etc/ssl/certs/ca-certificates.crt

Sharing this CA bundle does not couple the protocols or reuse a leaf certificate. STARGATE_TLS_CERT_PATH retains its QUIC meaning. STARGATE_GRPC_TLS_CA_CERT_PATH applies only to Stargate gRPC HTTPS. The bundle can contain roots for both independent leaf-certificate hierarchies. In system trust mode, NVCA injects neither variable and Pylon keeps Tonic’s default HTTPS root behavior.

For an unmanaged Pylon process, set the equivalent --grpc-tls-ca-cert-path <path> option directly. The file is read once at startup and is not reloaded.

When replacing a local plaintext configuration, also set this in the compute-plane Helmfile environment:

1
agentConfig:
2
  mergeConfig: |
3
    workload:
4
      stargateQUICInsecure: false

The compute-plane Helmfile merges the environment fragment over the generated registration fragment, so the TLS trust configuration is retained and an old plaintext override is disabled.

If you mirror images to a registry that does not use the stack’s default global.image.registry and global.image.repository, override the pylon sidecar image passed to generated LLM workers:

1
api:
2
  remoteConfig:
3
    configData:
4
      nvcf:
5
        sidecars:
6
          llm-router-client-image: <registry>/<repository>/pylon:0.14.1

api.env.NVCF_SIDECARS_LLM_ROUTER_CLIENT_IMAGE is deprecated. The stack translates it into remote config for one compatibility window and omits it from the API environment. Do not set both paths to different values.

The LLM API Gateway and request router images are resolved from the same stack artifact registry settings as the other control-plane services.

The self-managed stack passes API environment and remote configuration through separate chart values:

Configured api.env entries override the fixed stack environment. The remote ConfigMap carries the request-router worker address and Pylon image used when the API creates a new LLM function version.

Apply and Verify

Apply the updated control-plane environment:

$
cd path/to/nvcf-self-managed-stack
$
make apply HELMFILE_ENV=<environment-name>

For the managed issuer, wait for the ClusterIssuer:

$
kubectl wait --for=condition=Ready \
>
  clusterissuer/nvcf-openbao-pki \
>
  --timeout=2m
$
kubectl get clusterissuer nvcf-openbao-pki

For an external issuer, wait for the configured resource instead:

$
kubectl wait --for=condition=Ready \
>
  clusterissuer/<issuer-name> \
>
  --timeout=2m
$
# For a namespaced Issuer:
$
kubectl -n nvcf wait --for=condition=Ready \
>
  issuer/<issuer-name> \
>
  --timeout=2m

Set REQUEST_ROUTER_TLS_NAME to the value of addons.llm.pki.secretName. Its default is stargate-quic-tls. In certManager mode, verify the request-router Certificate and its issuer reference. The existingSecret mode does not render a Certificate, so skip the first two commands in that mode. The final command applies to both modes and reads only the public certificate:

$
export REQUEST_ROUTER_TLS_NAME="<configured-request-router-tls-name>"
$
kubectl -n nvcf wait --for=condition=Ready \
>
  "certificate/$REQUEST_ROUTER_TLS_NAME" \
>
  --timeout=2m
$
kubectl -n nvcf get certificate "$REQUEST_ROUTER_TLS_NAME" \
>
  -o jsonpath='{.spec.issuerRef.kind}{"/"}{.spec.issuerRef.name}{"\n"}'
$
kubectl -n nvcf get secret "$REQUEST_ROUTER_TLS_NAME" \
>
  -o jsonpath='{.data.tls\.crt}' \
>
  | base64 --decode \
>
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

Regenerate and apply the registration values for each compute cluster:

$
nvcf-cli self-hosted \
>
  --control-plane-stack deploy/stacks/self-managed \
>
  --compute-plane-stack deploy/stacks/nvcf-compute-plane \
>
  --env <environment-name> \
>
  --control-plane-context <control-plane-context> \
>
  --compute-plane-context <compute-plane-context> \
>
  compute-plane register \
>
  --control-plane-profile deploy/stacks/self-managed/out/control-plane-profile.yaml \
>
  --cluster-name <compute-plane-cluster-name> \
>
  --kube-context <compute-plane-context> \
>
  --region <region> \
>
  --output deploy/stacks/nvcf-compute-plane/out/<compute-plane-cluster-name>-register-values.yaml
$
$
nvcf-cli self-hosted \
>
  --compute-plane-stack deploy/stacks/nvcf-compute-plane \
>
  --env <environment-name> \
>
  compute-plane install \
>
  --values deploy/stacks/nvcf-compute-plane/out/<compute-plane-cluster-name>-register-values.yaml \
>
  --kube-context <compute-plane-context> \
>
  --cluster-name <compute-plane-cluster-name>

Existing LLM function versions retain the worker-sidecar image metadata captured when the version is created. Replacing pods or redeploying the same version does not apply a new Pylon image. After updating the control plane, create and deploy a new function version.

After deploying an LLM function, verify the workload trust bundle. Compare only .data.fingerprint with transportTls.trustBundleFingerprint in the profile. openssl x509 -fingerprint does not calculate the canonical NVCF bundle fingerprint.

$
kubectl -n nvcf-backend get configmap nvcf-transport-trust-bundle \
>
  -o jsonpath='{.data.fingerprint}{"\n"}'
$
kubectl -n nvcf-backend get configmap nvcf-transport-trust-bundle \
>
  -o jsonpath='{.data.nvcf-ca-bundle\.pem}' \
>
  | openssl x509 -noout -subject -issuer

Verify the worker sidecar:

$
kubectl get pods -n nvcf-backend -L FUNCTION_ID
$
kubectl -n nvcf-backend get pod <function-pod> \
>
  -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}'
$
kubectl -n nvcf-backend get pod <function-pod> \
>
  -o jsonpath='{range .spec.containers[?(@.name=="llm-worker")].args[*]}{.}{"\n"}{end}'
$
kubectl -n nvcf-backend get pod <function-pod> \
>
  -o jsonpath='{range .spec.containers[?(@.name=="llm-worker")].env[*]}{.name}{"="}{.value}{"\n"}{end}' \
>
  | grep -E '^STARGATE_(GRPC_TLS_CA_CERT_PATH|TLS_CERT_PATH)='

The worker args must contain --stargate-address=llm-request-router.nvcf.svc.cluster.local:50071, or the configured routable DNS name, and must not contain --quic-insecure. The external address is the initial gRPC dial endpoint. The reverse tunnel verifies the advertised request-router pod hostname instead. The environment must contain:

STARGATE_TLS_CERT_PATH=/etc/ssl/certs/ca-certificates.crt
STARGATE_GRPC_TLS_CA_CERT_PATH=/etc/ssl/certs/ca-certificates.crt

Also verify the control-plane components:

$
kubectl get deployment -n nvcf llm-api-gateway
$
kubectl get deployment -n nvcf llm-request-router-backend-router
$
kubectl get deployment,statefulset -n nvcf llm-request-router
$
kubectl get service -n nvcf llm-request-router-backend-router
$
kubectl get pods -n nvcf | grep -E 'llm-api-gateway|llm-request-router'
$
kubectl get httproute -A | grep llm

For remote workers, also verify the LLM GRPCRoute, UDPRoute, dedicated gRPC Certificate, stream timeout policy, and ReferenceGrant as described in Gateway Routing and DNS.

Certificate Renewal

cert-manager renews the request-router certificate and updates the Secret named by addons.llm.pki.secretName. The default is Secret/stargate-quic-tls. With mode: existingSecret there is no renewal loop and you update the configured Secret yourself.

Hot reload requires helm-nvcf-llm-request-router 1.10.0 or later and Stargate 0.11.1 or later. The request router polls the mounted server certificate and private key every 30 seconds. A valid replacement becomes active for new handshakes without restarting the request router. Established connections remain open. If a replacement is invalid, the router rejects it and keeps the last-known-good identity. Older chart releases and Stargate image overrides older than 0.11.1 require a request-router restart after the Secret update.

This reload contract covers the server certificate and private key only. Pylon loads both its gRPC CA bundle and QUIC trust bundle at startup. A trust-bundle change, including a root CA rotation, requires a rolling restart of the LLM worker pods. Renewing an intermediate under an already-trusted root does not change the trust bundle and does not require a worker-pod restart. Follow the transport TLS rotation runbook for the atomic Secret update, reload checks, trust-bundle rollout order, and recovery procedure.

Upgrade and Rollback

Use this order for an upgrade from plaintext transport:

1. Make the helm-nvcf-pki chart available from the configured chart source.

2. Remove the old plaintext setting or set workload.stargateQUICInsecure: false.

3. Apply the dependency Helmfile stage to install cert-manager, OpenBao, and the managed issuer, or prepare the external cert-manager and issuer:

   $
   HELMFILE_ENV=<environment-name> helmfile \
   >
     --file deploy/stacks/self-managed/helmfile.d/01-dependencies.yaml.gotmpl \
   >
     --environment default \
   >
     apply

4. Schedule a maintenance window and undeploy the existing LLM functions. The router does not support a mixed plaintext and TLS transition.

5. Apply the remaining control-plane stack. The managed router hook prepares the OpenBao signing path before cert-manager reconciles the request-router Certificate.

6. Wait for the issuer and the Certificate named by addons.llm.pki.secretName to become ready. The default name is stargate-quic-tls.

7. Export or update the control-plane profile, register each compute plane again, and install the refreshed registration values.

8. Recreate the LLM functions and verify the certificate SAN, trust-bundle fingerprint, worker address, and worker arguments.

Use this order for a safe rollback:

1. Create and verify the replacement ClusterIssuer or namespaced Issuer.

2. Add both the current and replacement public roots to the compute-plane profile, calculate the canonical bundle fingerprint, register each compute plane again, and recreate the LLM workers with the combined trust bundle.

3. Set addons.llm.pki.issuerKind and issuerName to the replacement, set clusterIssuer.enabled: false, and apply the control-plane stack.

4. Wait for the Certificate named by addons.llm.pki.secretName to become ready, confirm a successful server_identity reload, and verify the replacement TLS data path with a new connection.

5. Remove the old root from the compute-plane profile, register each compute plane again, and recreate the workers.

6. Confirm that no Certificate references the old issuer:

   $
   kubectl --context <control-plane-context> get certificate -A \
   >
     -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,KIND:.spec.issuerRef.kind,ISSUER:.spec.issuerRef.name

7. Remove the old managed release after it is no longer part of the Helmfile state. Run helm status first and confirm the release is the superseded one, that the previous step listed no Certificate still referencing its issuer, and that the replacement data path is verified. Uninstalling is not reversible from the cluster; only proceed once all three hold.

   $
   helm status nvcf-pki \
   >
     --namespace cert-manager \
   >
     --kube-context <control-plane-context>

   After confirming, remove the release:

   $
   helm uninstall nvcf-pki \
   >
     --namespace cert-manager \
   >
     --kube-context <control-plane-context>

8. Delete the retained old ClusterIssuer only after all references are gone:

   $
   kubectl --context <control-plane-context> \
   >
     delete clusterissuer <old-issuer-name>

If no replacement issuer is available, undeploy the LLM functions and disable the LLM addon before removing the issuer. Keep LLM traffic stopped until a secure issuer and trust path are available.

The managed ClusterIssuer is retained when its Helm release is removed. Do not delete it before its certificates and consumers have moved to the replacement trust path. Do not use stargateQUICInsecure as a production rollback path.

Local Plaintext Transport

Use plaintext transport only in local or isolated test clusters. Set both plaintext controls:

1
addons:
2
  llm:
3
    enabled: true
4
    gateway:
5
      replicaCount: 1
6
      auth:
7
        grpcInsecure: true
8
    requestRouter:
9
      replicaCount: 1
10
11
agentConfig:
12
  mergeConfig: |
13
    workload:
14
      stargateQUICInsecure: true

addons.llm.gateway.auth.grpcInsecure: true configures the LLM API Gateway to talk to the NVCF API over plaintext gRPC.

workload.stargateQUICInsecure: true configures generated LLM workers to pass the plaintext QUIC setting to the pylon sidecar.

Troubleshooting

404 no_eligible_candidates from llm.invocation.<domain> means the request reached the LLM Gateway, but the requested function or model was unknown or was not registered on the selected request router. Similar 503 candidate errors mean the router knows the target but has no active eligible backend. Check:

• The LLM function is deployed and its pod is Running.
• The request model value uses <function-id>/<model-name>.
• The function’s models[].name matches the model suffix in the request.
• models[].llmConfig.uris includes the invoked path.
• When addons.llm.requestRouter.loadBalancer.config is set, it includes the algorithm selected by the function’s models[].llmConfig.routingMethod.
• The llm-worker sidecar connected to llm-request-router.
• The effective LLM request-router worker address is reachable from the worker cluster.
• Local clusters using plaintext transport include both grpcInsecure and stargateQUICInsecure.

For transport TLS failures, check:

• gRPC unknown issuer: for a public ACM NLB certificate, verify the system and public root store. For a private-CA NLB certificate, verify STARGATE_GRPC_TLS_CA_CERT_PATH and the CA chain in that PEM bundle.
• QUIC unknown issuer: inspect the Certificate Ready condition and verify issuerRef.kind, issuerRef.name, and the issuer namespace. A namespaced Issuer must be in nvcf.
• gRPC SAN mismatch: compare the external HTTPS dial hostname with the public DNS SANs on the NLB listener certificate. The internal pod authority is not the gRPC TLS identity.
• QUIC SAN mismatch: compare the request router’s --advertised-hostname-template with the SANs in the Secret named by addons.llm.pki.secretName. The default is Secret/stargate-quic-tls. The external UDP dial endpoint does not replace the advertised QUIC identity.
• Expired or not-yet-valid certificate: inspect the certificate dates and the cluster clock. Renew the certificate and use the transport TLS rotation runbook to verify that the replacement becomes active.
• System trust mode: confirm STARGATE_TLS_CERT_PATH and STARGATE_GRPC_TLS_CA_CERT_PATH are both unset so Pylon uses its enabled system and public roots.
• Bundle trust mode: verify ConfigMap/nvcf-transport-trust-bundle, compare its fingerprint with the compute-plane profile, and confirm STARGATE_TLS_CERT_PATH in the llm-worker container. The STARGATE_GRPC_TLS_CA_CERT_PATH override is optional; when unset, the gRPC registration and watch paths reuse the existing Stargate bundle. Restart the worker after changing either bundle.

Useful logs:

$
kubectl logs -n nvcf deploy/llm-api-gateway --tail=100
$
kubectl logs -n nvcf statefulset/llm-request-router \
>
  --all-pods=true --tail=100
$
kubectl logs -n nvcf-backend <function-pod> -c llm-worker --tail=100

In healthy routing, the request router logs show a reverse tunnel connection from the worker and at least one routing candidate for the requested function.