> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Credential Propagation

How credentials flow through the system when NeMo Platform runs jobs or serves inference.

## Managed Job Workload Identity

When a user submits a job (customization, evaluation, data generation), the job
runs in a workload container that needs to call NeMo Platform APIs to download
datasets, upload results, and read secrets. Managed job credentials are
propagated through workload identity token exchange, not by using a serialized
principal as an API credential.

The flow:

1. The user submits a job via the API. The control plane authorizes job creation
   and records the submitter auth context on the job metadata.
2. When workload identity token exchange is enabled, the managed backend creates
   a workload with `NMP_WORKLOAD_IDENTITY_TOKEN_FILE` set to a subject-token file
   path. This is the only platform credential mounted into the workload for job
   OBO.
3. The backend owns that subject-token file. Kubernetes uses a projected,
   pod-bound service account token volume, while Docker uses a
   controller-managed volume containing a Docker workload proof token.
4. The SDK in the workload detects `NMP_WORKLOAD_IDENTITY_TOKEN_FILE`, reads the
   subject token, discovers the workload token exchange endpoint, and exchanges
   the subject token for a NeMo Platform access token using OAuth 2.0 Token
   Exchange (RFC 8693).
5. The auth service validates the workload proof, fetches the matching internal
   workload delegation row, and returns a delegated NeMo access token. The token
   top-level `sub` is the captured submitter, and the RFC 8693 `act.sub` claim
   is the workload actor.
6. API calls from the workload use the exchanged access token. When that access
   token nears expiry, the SDK rereads the subject-token file and performs
   another exchange.

Job containers still need to run inside the platform trust boundary. Network
policies and gateway configuration should prevent workloads from reaching
internal endpoints they do not need and should prevent external callers from
forging trusted identity headers. For the full architecture, see [Security Model](/documentation/access-control/security-model#job-credential-propagation).

## Workload Identity Token Files

For workload identity, managed backends inject
`NMP_WORKLOAD_IDENTITY_TOKEN_FILE`. The file contains a workload identity
subject token or proof token, not a final NeMo API access token.

The SDK reads that file immediately before OAuth 2.0 Token Exchange (RFC 8693),
exchanges the subject token at the discovered workload token exchange endpoint,
caches the returned access token until near expiry, and then rereads the file
for the next exchange.

Backend ownership:

* Kubernetes uses a projected service account token volume. Kubelet rotates the
  file. The auth service validates the presented token with Kubernetes
  TokenReview and uses the verified
  `authentication.kubernetes.io/pod-uid` reference from TokenReview metadata to
  look up the internal delegation row.
* Docker uses a dedicated controller-managed workload identity volume. The
  Docker backend writes a NeMo opaque proof token bound to an internal
  delegation row. The token uses the private Docker subject-token type and is
  checked against the hash stored in the delegation row. Docker does not use
  TokenReview or external IdP-issued workload proof tokens.

Users must not provide `NMP_PRINCIPAL`, `NEMO_WORKLOAD_TOKEN`,
`NEMO_WORKLOAD_TOKEN_FILE`, or `NMP_WORKLOAD_IDENTITY_TOKEN_FILE` in managed job
requests. Direct SDK users may set `NMP_WORKLOAD_IDENTITY_TOKEN_FILE` only when
they own the refreshed subject token file.

Workload delegation rows are internal auth state. NeMo Platform does not expose
workload-delegation-specific API endpoints, generated public SDK resources, CLI
commands, or UI flows. Platform administrators can still inspect generic entity
store records through the normal entity administration surface.

Kubernetes labels, annotations, owner references, and Docker labels are
reconciliation and cleanup aids only. They are not token-exchange authorization
inputs. Token exchange authorizes against the verified subject token, the
deterministic delegation row name, the stored submitter auth context, audience,
expiry/revocation state, and any verifier-confirmed bound reference.

This follows the same shape as common cloud SDKs: AWS uses
`AWS_WEB_IDENTITY_TOKEN_FILE`, Azure uses `AZURE_FEDERATED_TOKEN_FILE`, and
Google Workload Identity Federation uses `GOOGLE_APPLICATION_CREDENTIALS` to
point at a credential config that references a projected token file.

## Inference Auth Context

When a model is deployed as an inference endpoint, incoming requests are authenticated at the gateway or service level before reaching the model. The user's auth context is evaluated before the request reaches the model container.

## Trust Implications

The workload identity subject-token file is not the final NeMo Platform access
token, but it can be exchanged for one. Protect it as credential material, mount
it read-only into the workload container where possible, and keep its lifetime
short.

Ensure Kubernetes RBAC prevents unauthorized access to job pods. Network policies should restrict which pods can reach NeMo Platform internal endpoints.

## Related

* [Security Model](/documentation/access-control/security-model) — Trust boundaries and job credential propagation.
* [Auth Configuration](/documentation/access-control/deployment) — Platform auth configuration.