Skip to navigation

NVIDIA Workload Identity Keys

View as Markdown

dsx-identity-keys is the NVIDIA Workload Identity Keys process installed in each identity shard. It validates only local SPIRE JWT-SVIDs, transforms their SPIFFE subject to the NVIDIA Workload Identity contract, preserves the registered aud, exp, iat, jti, and nbf claims, writes the subject components as claims, and signs the exchanged DSX JWT with the shard’s active key. SPIRE registration determines the tenant in the local subject, and NVIDIA Workload Identity Keys preserves it while validating the common identity grammar. It obtains local JWT verification bundles through the node-local SPIFFE Workload API.

Tenant and service determine the exchanged trust domain. The resource path contains complete key/value pairs with unique keys. Each segment starts with an ASCII letter or digit, followed by ASCII letters, digits, ., _, or -. NVIDIA Workload Identity Keys preserves valid paths exactly, including case and order:

local: spiffe://<local-domain>/tenant/<tenant>/service/dsx/cluster/cluster-1/namespace/team-a/workload/api
exchanged: spiffe://<tenant>.<region>.dsx.<partition>/cluster/cluster-1/namespace/team-a/workload/api

For example, DPF can use this explicitly labelled DPUService suffix:

local: spiffe://<local-domain>/tenant/<tenant>/service/dsx/dpu/<serial>/dpuservice/<service-id>/namespace/<namespace>
exchanged: spiffe://<tenant>.<region>.dsx.<partition>/dpu/<serial>/dpuservice/<service-id>/namespace/<namespace>

DPF includes the namespace because DPU service IDs are unique only within a Kubernetes namespace. NVIDIA Workload Identity Keys only preserves that suffix.

Configure the external DPF identity templates to supply the local tenant and service segments and this DPUService suffix.

By default, each NVIDIA Workload Identity Keys instance writes only its local public keys to one shared NATS KV bucket and watches the complete bucket. It retains one aggregate keyed by globally unique kid, and its JWKS endpoint serves every valid local and remote public key. Startup requires the Event Bus connection, KV bucket, and initial history. Later connection failures leave NVIDIA Workload Identity Keys live but not ready while it reconnects. Instances using the same bucket converge on the same aggregate, so relying parties can use the JWKS from any reachable NVIDIA Workload Identity Keys.

--disable-nats skips the Event Bus connection and serves only the local public signing keys. Readiness then depends only on the local key registry. This mode does not distribute keys between shards, so each relying party must use the JWKS endpoint for the shard that issued its tokens.

Token claims

The exchanged DSX JWT carries each component of its subject as a claim, so relying parties can authorize on claims without parsing sub. Each value is the component encoded into sub. For spiffe://operator-1.ds9.dsx.nvid.id/bm/worker-1/pod/<pod-uid>/role/web:

{
"dsx.nvidia.com/tenant": "operator-1",
"dsx.nvidia.com/region": "ds9",
"dsx.nvidia.com/service": "dsx",
"dsx.nvidia.com/partition": "nvid.id",
"dsx.nvidia.com/resource": [
{"resource_type": "bm", "resource_id": "worker-1"},
{"resource_type": "pod", "resource_id": "<pod-uid>"},
{"resource_type": "role", "resource_id": "web"}
]
}

resource lists every path pair in order, starting with the resource itself. A rule on one entry, such as {bm, worker-1}, matches the host and every workload on it, like a prefix match on sub. To match only the host, the rule must also check the length of resource, like an exact match on sub.

Flat key distribution

one aggregate JWKS local keys out, all public keys in local keys out, all public keys in local keys out, all public keys in External relying party NVIDIA Workload IdentityKeys Deploymentshard-a NVIDIA Workload IdentityKeys Deploymentshard-b NVIDIA Workload IdentityKeys Deploymentshard-c External DSX Event BusNATS KV bucket

The local E2E fixture uses two NVIDIA Workload Identity Keys instances in the same subject space:

local public keys local public keys initial replay and watched updates initial replay and watched updates k1 NVIDIA Workload IdentityKeys Pod NATS KV bucket k2 NVIDIA Workload IdentityKeys Pod k1 aggregate JWKS k2 aggregate JWKS

Public keys use NATS KV keys of the form shard.<shard-id>.kid.<kid>. Each value is one public JWK. The bucket keeps one revision per key and applies the configured TTL. Each instance refreshes its local entries every quarter of that TTL, leaving three further write opportunities before expiry. Each instance starts an updates-only watch before loading History, so concurrent writes queue while current values are loaded. If the watcher closes, its replacement uses the same ordering. Each valid entry replaces the globally unique kid it addresses. NATS connections use either no authentication or OAuth2 client credentials. OAuth2 mode reads a JSON credentials file containing non-empty client_id and client_secret fields, discovers the token endpoint from the configured issuer, and rereads the credentials when the cached access token expires. OAuth scopes are configurable, and an empty scope list sends no scope parameter. The access token is sent through native NATS token authentication. Production deployments place NVIDIA Workload Identity Keys clients in a dedicated NATS account, which provides the account boundary and JetStream capacity used by the KV bucket. Basic authentication is not supported. Private, malformed, unsupported, and mismatched keys do not replace a valid aggregate entry.

The identity chart deploys two active replicas per shard through charts/dsx-identity/templates/identity-keys-deployment.yaml. Both replicas load the same shard-local private keyset and active-kid Secret. Only public keys are published to the Event Bus.

External secret automation requests /v1/transit/export/signing-key/<name>?format=pem and renders the exported data.keys values into the private keyset file:

{"keys":["-----BEGIN EC PRIVATE KEY-----\n...\n-----END EC PRIVATE KEY-----\n"]}

NVIDIA Workload Identity Keys derives each kid from the SHA-256 public JWK thumbprint and accepts SEC 1 EC, PKCS#1 RSA, and PKCS#8 asymmetric private keys. The separate active-kid.json Secret entry selects one loaded key and the JOSE algorithm for the Transit key ring. Every loaded key must support that algorithm:

{"active_kid":"<kid>","alg":"ES256"}

Each instance watches both mounted sources with filesystem notifications and polls for content changes every 30 seconds. --file-watch-poll-interval sets the polling interval. Both watchers detect Kubernetes Secret symlink updates. Each file change validates the keyset and active-key pair. A valid pair replaces the active signer and local public keys without a restart and triggers NATS publication. Missing, malformed, or mismatched sources retain the last valid signer and keys until both sources validate. Invalid sources at startup prevent the process from starting.

Rotation first adds and publishes the new key while the old kid remains active. Activation changes only the active-kid Secret. Removing an expired old key stops its lease refresh, after which it disappears from aggregate JWKS responses.

HTTP API

EndpointPurpose
POST /v1/exchangeUnauthenticated RFC 8693 exchange of a local SPIRE JWT-SVID
GET /.well-known/jwks.jsonPublic JWKS. Aggregate with NATS enabled, local-only with NATS disabled
GET /.well-known/openid-configurationIssuer, JWKS URL, and advertised signing algorithms
GET /liveProcess liveness
GET /readyPublic-key registry readiness. Includes NATS connection and KV replay when enabled

The exchange endpoint is intentionally unauthenticated because the submitted local SPIRE JWT-SVID is the credential being exchanged.

The exchange form requires grant_type, subject_token, and subject_token_type. It accepts the optional requested_token_type, caps request bodies at 1 MiB. Audience is preserved from the validated SVID. NVIDIA Workload Identity Keys does not accept an audience override or apply a relying-party audience allowlist. Error responses do not expose parser or verifier details.

Flags

FlagDefaultPurpose
--listen:8080HTTP listen address
--socketWorkload API defaultLocal SPIFFE Workload API socket URL
--jwks-urlrequiredAbsolute public JWKS URL advertised by discovery
--issuerrequiredHTTPS DSX issuer with a DNS hostname
--regionrequiredRegion encoded in exchanged DSX identities
--partitionrequiredPartition encoded in exchanged DSX identities
--signing-keyset/run/secrets/dsx-identity-keys/keyset.jsonJSON file containing private PEM keys
--file-watch-poll-interval30sPositive interval for checking file contents when filesystem notifications are missed
--active-kid/run/secrets/dsx-identity-keys-active-kid/active-kid.jsonJSON file containing the active signing kid and algorithm
--shard-idNATS enabledGlobally unique shard ID in the Event Bus subject space
--disable-natsfalseServe local public keys without NATS synchronization
--nats-serverNATS enabledNATS server URL
--nats-client-namedsx-identity-keys-<shard-id>NATS connection name
--public-key-ttl10mNATS KV entry TTL
--nats-kv-bucketDSX_IDENTITY_KEYSShared NATS KV bucket
--nats-kv-replicas1KV bucket replica count
--nats-auth-modenoneNATS authentication mode: none or oauth2
--nats-credentials-fileOAuth2 modeJSON file containing OAuth2 client credentials
--nats-oauth-issuer-urlOAuth2 modeOAuth2 issuer whose discovery document advertises the token endpoint
--nats-oauth-scopesnatsComma-separated scopes; empty requests none
--log-levelinfoJSON log level: debug, info, warn, or error