Credential Propagation

How credentials flow through the system when NeMo Platform runs jobs or serves inference.

Managed Job Workload Identity

When a user submits a job (customization, evaluation, data generation), the job runs in a workload container that needs to call NeMo Platform APIs to download datasets, upload results, and read secrets. Managed job credentials are propagated through workload identity token exchange, not by using a serialized principal as an API credential.

The flow:

1. The user submits a job via the API. The control plane authorizes job creation and records the job metadata.
2. When workload identity token exchange is enabled, the managed backend creates a workload with NMP_WORKLOAD_IDENTITY_TOKEN_FILE set to a subject-token file path.
3. The backend owns that subject-token file. Kubernetes uses a projected service account token volume, while Docker uses a controller-managed volume that is refreshed by the jobs controller.
4. The SDK in the workload detects NMP_WORKLOAD_IDENTITY_TOKEN_FILE, reads the subject token, discovers the workload token exchange endpoint, and exchanges the subject token for a NeMo Platform access token using OAuth 2.0 Token Exchange (RFC 8693).
5. API calls from the workload use the exchanged access token. When that access token nears expiry, the SDK rereads the subject-token file and performs another exchange.

Job containers still need to run inside the platform trust boundary. Network policies and gateway configuration should prevent workloads from reaching internal endpoints they do not need and should prevent external callers from forging trusted identity headers. For the full architecture, see Security Model.

Workload Identity Token Files

For external IdP-backed workload identity, managed backends inject NMP_WORKLOAD_IDENTITY_TOKEN_FILE. The file contains a workload identity subject token, not a final NeMo API access token.

The SDK reads that file immediately before OAuth 2.0 Token Exchange (RFC 8693), exchanges the subject token at the discovered IdP token endpoint, caches the returned access token until near expiry, and then rereads the file for the next exchange.

Backend ownership:

• Kubernetes uses a projected service account token volume. Kubelet rotates the file.
• Docker uses a dedicated controller-managed workload identity volume. The Docker backend writes and refreshes the file.

Users must not provide NEMO_WORKLOAD_TOKEN, NEMO_WORKLOAD_TOKEN_FILE, or NMP_WORKLOAD_IDENTITY_TOKEN_FILE in managed job requests. Direct SDK users may set NMP_WORKLOAD_IDENTITY_TOKEN_FILE only when they own the refreshed subject token file.

This follows the same shape as common cloud SDKs: AWS uses AWS_WEB_IDENTITY_TOKEN_FILE, Azure uses AZURE_FEDERATED_TOKEN_FILE, and Google Workload Identity Federation uses GOOGLE_APPLICATION_CREDENTIALS to point at a credential config that references a projected token file.

Inference Auth Context

When a model is deployed as an inference endpoint, incoming requests are authenticated at the gateway or service level before reaching the model. The user’s auth context is evaluated before the request reaches the model container.

Trust Implications

The workload identity subject-token file is not the final NeMo Platform access token, but it can be exchanged for one. Protect it as credential material, mount it read-only into the workload container where possible, and keep its lifetime short.

Related

• Security Model — Trust boundaries and job credential propagation.
• Auth Configuration — Platform auth configuration.