Credential Propagation

How credentials flow through the system when NeMo Helix runs jobs or serves inference.

Managed Job Workload Identity

When a user submits a job (customization, evaluation, data generation), the job runs in a workload container that needs to call NeMo Helix APIs to download datasets, upload results, and read secrets. Managed job credentials are propagated through workload identity token exchange, not by using a serialized principal as an API credential.

The flow:

1. The user submits a job via the API. The control plane authorizes job creation and records the submitter auth context on the job metadata.
2. When workload identity token exchange is enabled, the managed backend creates a workload with NHX_WORKLOAD_IDENTITY_TOKEN_FILE set to a subject-token file path. This is the only platform credential mounted into the workload for job OBO.
3. The backend owns that subject-token file. Kubernetes uses a projected, pod-bound service account token volume, while Docker uses a controller-managed volume containing a Docker workload proof token.
4. The SDK in the workload detects NHX_WORKLOAD_IDENTITY_TOKEN_FILE, reads the subject token, discovers the workload token exchange endpoint, and exchanges the subject token for a NeMo Helix access token using OAuth 2.0 Token Exchange (RFC 8693).
5. The auth service validates the workload proof, fetches the matching internal workload delegation row, and returns a delegated NeMo access token. The token top-level sub is the captured submitter, and the RFC 8693 act.sub claim is the workload actor.
6. API calls from the workload use the exchanged access token. When that access token nears expiry, the SDK rereads the subject-token file and performs another exchange.

Job containers still need to run inside the platform trust boundary. Network policies and gateway configuration should prevent workloads from reaching internal endpoints they do not need and should prevent external callers from forging trusted identity headers. For the full architecture, see Security Model.

Managed Agent Auth Proxy

Agent deployments use the same creator-auth-context model as jobs. When workload token exchange is enabled, an agent deployment must have creator auth context. If the creator context is missing, the deployment is rejected instead of falling back to an unscoped service principal.

Some agent clients cannot manage NeMo Helix bearer tokens themselves. For those clients, the platform injects an auth-proxy sidecar. The agent talks to the sidecar on loopback without sending credentials. In token-exchange mode, the sidecar receives NHX_WORKLOAD_IDENTITY_TOKEN_FILE, exchanges the subject token for a delegated NeMo Helix access token, refreshes it as needed, and forwards requests upstream with Authorization: Bearer <exchanged-token>. The sidecar forwards to the platform base URL. In token-exchange mode that URL must be safe for a bearer token: https:// for a remote endpoint, or loopback HTTP for local development.

When workload token exchange is disabled, the same auth-proxy sidecar runs in trusted-header mode and stamps X-NHX-Principal-* headers instead. That mode is for deployments where auth is enabled but workload tokens are not.

Workload Identity Token Files

For workload identity, managed backends inject NHX_WORKLOAD_IDENTITY_TOKEN_FILE. The file contains a workload identity subject token or proof token, not a final NeMo API access token.

The SDK reads that file immediately before OAuth 2.0 Token Exchange (RFC 8693), exchanges the subject token at the discovered workload token exchange endpoint, caches the returned access token until near expiry, and then rereads the file for the next exchange.

Backend ownership:

• Kubernetes uses a projected service account token volume. Kubelet rotates the file. The auth service validates the presented token with Kubernetes TokenReview and uses the verified authentication.kubernetes.io/pod-uid reference from TokenReview metadata to look up the internal delegation row.
• Docker uses a dedicated controller-managed workload identity volume. The Docker backend writes a NeMo opaque proof token bound to an internal delegation row. The token uses the private Docker subject-token type and is checked against the hash stored in the delegation row. Docker does not use TokenReview or external IdP-issued workload proof tokens.

Users must not provide NHX_PRINCIPAL, NEMO_WORKLOAD_TOKEN, NEMO_WORKLOAD_TOKEN_FILE, or NHX_WORKLOAD_IDENTITY_TOKEN_FILE in managed job requests. Direct SDK users may set NHX_WORKLOAD_IDENTITY_TOKEN_FILE only when they own the refreshed subject token file.

Workload delegation rows are internal auth state. NeMo Helix does not expose workload-delegation-specific API endpoints, generated public SDK resources, CLI commands, or UI flows. Platform administrators can still inspect generic entity store records through the normal entity administration surface.

Kubernetes labels, annotations, owner references, and Docker labels are reconciliation and cleanup aids only. They are not token-exchange authorization inputs. Token exchange authorizes against the verified subject token, the deterministic delegation row name, the stored submitter auth context, audience, expiry/revocation state, and any verifier-confirmed bound reference.

This follows the same shape as common cloud SDKs: AWS uses AWS_WEB_IDENTITY_TOKEN_FILE, Azure uses AZURE_FEDERATED_TOKEN_FILE, and Google Workload Identity Federation uses GOOGLE_APPLICATION_CREDENTIALS to point at a credential config that references a projected token file.

Inference Auth Context

When a model is deployed as an inference endpoint, incoming requests are authenticated at the gateway or service level before reaching the model. The user’s auth context is evaluated before the request reaches the model container.

Trust Implications

The workload identity subject-token file is not the final NeMo Helix access token, but it can be exchanged for one. Protect it as credential material, mount it read-only into the workload container where possible, and keep its lifetime short.

Related

• Security Model — Trust boundaries and job credential propagation.
• Auth Configuration — Platform auth configuration.