Credential Propagation

How credentials flow through the system when NeMo Platform runs jobs or serves inference.

Managed Job Workload Identity

When a user submits a job (customization, evaluation, data generation), the job runs in a workload container that needs to call NeMo Platform APIs to download datasets, upload results, and read secrets. Managed job credentials are propagated through workload identity token exchange, not by using a serialized principal as an API credential.

The flow:

1. The user submits a job via the API. The control plane authorizes job creation and records the submitter auth context on the job metadata.
2. When workload identity token exchange is enabled, the managed backend creates a workload with NMP_WORKLOAD_IDENTITY_TOKEN_FILE set to a subject-token file path. This is the only platform credential mounted into the workload for job OBO.
3. The backend owns that subject-token file. Kubernetes uses a projected, pod-bound service account token volume, while Docker uses a controller-managed volume containing a Docker workload proof token.
4. The SDK in the workload detects NMP_WORKLOAD_IDENTITY_TOKEN_FILE, reads the subject token, discovers the workload token exchange endpoint, and exchanges the subject token for a NeMo Platform access token using OAuth 2.0 Token Exchange (RFC 8693).
5. The auth service validates the workload proof, fetches the matching internal workload delegation row, and returns a delegated NeMo access token. The token top-level sub is the captured submitter, and the RFC 8693 act.sub claim is the workload actor.
6. API calls from the workload use the exchanged access token. When that access token nears expiry, the SDK rereads the subject-token file and performs another exchange.

Job containers still need to run inside the platform trust boundary. Network policies and gateway configuration should prevent workloads from reaching internal endpoints they do not need and should prevent external callers from forging trusted identity headers. For the full architecture, see Security Model.

Workload Identity Token Files

For workload identity, managed backends inject NMP_WORKLOAD_IDENTITY_TOKEN_FILE. The file contains a workload identity subject token or proof token, not a final NeMo API access token.

The SDK reads that file immediately before OAuth 2.0 Token Exchange (RFC 8693), exchanges the subject token at the discovered workload token exchange endpoint, caches the returned access token until near expiry, and then rereads the file for the next exchange.

Backend ownership:

• Kubernetes uses a projected service account token volume. Kubelet rotates the file. The auth service validates the presented token with Kubernetes TokenReview and uses the verified authentication.kubernetes.io/pod-uid reference from TokenReview metadata to look up the internal delegation row.
• Docker uses a dedicated controller-managed workload identity volume. The Docker backend writes a NeMo opaque proof token bound to an internal delegation row. The token uses the private Docker subject-token type and is checked against the hash stored in the delegation row. Docker does not use TokenReview or external IdP-issued workload proof tokens.

Users must not provide NMP_PRINCIPAL, NEMO_WORKLOAD_TOKEN, NEMO_WORKLOAD_TOKEN_FILE, or NMP_WORKLOAD_IDENTITY_TOKEN_FILE in managed job requests. Direct SDK users may set NMP_WORKLOAD_IDENTITY_TOKEN_FILE only when they own the refreshed subject token file.

Workload delegation rows are internal auth state. NeMo Platform does not expose workload-delegation-specific API endpoints, generated public SDK resources, CLI commands, or UI flows. Platform administrators can still inspect generic entity store records through the normal entity administration surface.

Kubernetes labels, annotations, owner references, and Docker labels are reconciliation and cleanup aids only. They are not token-exchange authorization inputs. Token exchange authorizes against the verified subject token, the deterministic delegation row name, the stored submitter auth context, audience, expiry/revocation state, and any verifier-confirmed bound reference.

This follows the same shape as common cloud SDKs: AWS uses AWS_WEB_IDENTITY_TOKEN_FILE, Azure uses AZURE_FEDERATED_TOKEN_FILE, and Google Workload Identity Federation uses GOOGLE_APPLICATION_CREDENTIALS to point at a credential config that references a projected token file.

Inference Auth Context

When a model is deployed as an inference endpoint, incoming requests are authenticated at the gateway or service level before reaching the model. The user’s auth context is evaluated before the request reaches the model container.

Trust Implications

The workload identity subject-token file is not the final NeMo Platform access token, but it can be exchanged for one. Protect it as credential material, mount it read-only into the workload container where possible, and keep its lifetime short.

Related

• Security Model — Trust boundaries and job credential propagation.
• Auth Configuration — Platform auth configuration.