Machine Identity (Day 1)

View as Markdown

Operator guide for per-organization machine identity configuration: JWT-SVID issuance for tenant workloads, optional RFC 8693 token delegation, discovery endpoints, verification, and signing-key rotation.

This is a Day 1 (Configuration) activity. Complete Day 0 Machine Identity first — site secrets, [machine_identity] in site config, and a healthy nico-api with enabled = true.

The primary management surface is the NICo REST API (nico-rest-api). Use nicocli (TUI or REST-backed commands where available), curl, or your automation against the endpoints below.

Design reference: SPIFFE JWT-SVID SDD.


Before You Start

You should already have:

  • Day 0 machine identity enabled ([machine_identity].enabled = true and valid encryption keys).
  • A tenant org with at least one instance in READY (for end-to-end signing tests).
  • A user with TENANT_ADMIN in the target org (REST management APIs). KEK re-wrap is site-operator scope: gRPC only today, REST API planned — see KEK rotation.

Verify REST connectivity:

$nicocli site list
$nicocli user get

Resolve your site id (UUID) — required in all tenant-identity URLs:

$nicocli site list --output json

Overview

ConcernWhere it lives
Global enable, algorithm, KEK id, TTL bounds, egress proxySite [machine_identity]Day 0 guide
Issuer, audiences, TTL, per-org enable, signing keysPer-org tenant-identity/config (REST)
RFC 8693 token exchange callbackPer-org tenant-identity/token-delegation (REST)
Workload token fetchIMDS GET …/meta-data/identity on the instance (DPU agent)
Direct signingCore gRPC SignMachineIdentity (DPU machine cert)
Public keysREST .well-known/jwks.json, .well-known/spiffe/jwks.json, .well-known/openid-configuration

1. Create or Update Per-Org Identity Config

Endpoint: PUT /v2/org/{org}/nico/site/{siteID}/tenant-identity/config

Role: TENANT_ADMIN in {org}.

On first PUT, Core generates a new ES256 per-org signing keypair, encrypts the private key with the site KEK, and stores it. Subsequent PUTs reuse the keypair unless you set rotateKey: true.

Minimal example (direct signing)

Replace {org}, {site-id}, issuer host, and audience strings for your environment. The issuer is typically the NICo REST base URL for that org/site (it becomes the JWT iss claim):

$curl -sS -X PUT \
> -H "Authorization: Bearer $TOKEN" \
> -H "Content-Type: application/json" \
> "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/tenant-identity/config" \
> -d '{
> "enabled": true,
> "issuer": "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}",
> "defaultAudience": "tenant-api",
> "allowedAudiences": ["tenant-api"],
> "tokenTtlSeconds": 300
> }'
FieldRequiredNotes
issuerYesJWT iss / OIDC issuer — https://, http://, or spiffe:// with DNS host
defaultAudienceYesUsed when a caller does not specify an audience
tokenTtlSecondsYesMust fall within site token_ttl_min_sectoken_ttl_max_sec
enabledNoDefaults to true; set false to pause issuance while keeping config
allowedAudiencesNoEmpty/omitted → stored as [defaultAudience] only
subjectPrefixNoSPIFFE prefix for JWT sub; if omitted, derived from issuer trust domain

Returns 201 Created on first call, 200 OK on update.

nicocli TUI

Interactive flow:

$nicocli tui
$> tenant-identity update

The TUI prompts for issuer, audiences, TTL, optional subjectPrefix, and optional signing-key rotation.

Read and Delete

$# GET config + signing key metadata
$curl -sS -H "Authorization: Bearer $TOKEN" \
> "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/tenant-identity/config"
$
$# DELETE (removes config and keys for the org)
$curl -sS -X DELETE -H "Authorization: Bearer $TOKEN" \
> "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/tenant-identity/config"

DELETE is appropriate when offboarding an org from machine identity entirely. It does not disable site-level [machine_identity].


2. Optional — Token Delegation

When token delegation is configured, NICo issues a short-lived intermediate JWT-SVID to your RFC 8693 token exchange server instead of signing the final workload token directly. Use this when the tenant layer must control final token shape or audience mapping.

Prerequisite: tenant-identity/config must exist (404 otherwise).

Endpoint: PUT /v2/org/{org}/nico/site/{siteID}/tenant-identity/token-delegation

Recommendation: Token delegation causes nico-api to call the org-configured tokenEndpoint over HTTP(S). For external token exchange URLs, configure site-level egress controls in [machine_identity] during Day 0:

  • token_endpoint_http_proxy — route outbound token-exchange HTTP through a controlled egress proxy
  • token_endpoint_domain_allowlist — restrict which hostnames tenants may register on tokenEndpoint

Together these mitigate SSRF-style risk if a tenant admin supplies an endpoint the API should not reach. They are optional at install time but strongly recommended for production sites that delegate to external hosts.

Example (adjust fields to match your STS):

$curl -sS -X PUT \
> -H "Authorization: Bearer $TOKEN" \
> -H "Content-Type: application/json" \
> "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/tenant-identity/token-delegation" \
> -d '{
> "tokenEndpoint": "https://sts.example.com/oauth/token",
> "subjectTokenAudience": "tenant-exchange",
> "clientSecretBasic": {
> "clientId": "nico-delegation",
> "clientSecret": "<secret>"
> }
> }'

PUT is full replace: omitting clientSecretBasic on an update clears stored credentials. Re-supply secrets on every update that should keep basic auth.

Note: tokenEndpoint may use http:// with an IP address (for example a node-local sidecar). NICo allows this for in-instance STS; use allowlists and network policy in production where appropriate.

$# Remove delegation (return to direct signing)
$curl -sS -X DELETE -H "Authorization: Bearer $TOKEN" \
> "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/tenant-identity/token-delegation"

3. Discovery and Verification

Full step-by-step checks (OIDC discovery, JWKS alignment, gRPC signing, IMDS, JWT claim validation): Machine Identity Verification.

Quick reference:

$curl -sS "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/.well-known/openid-configuration"
$curl -sS "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/.well-known/jwks.json"
$curl -sS "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/.well-known/spiffe/jwks.json"

nicocli TUI: tenant-identity openid-configuration get, tenant-identity jwks get, tenant-identity spiffe-jwks get.


4. Per-Org JWT Signing Key Rotation

Rotates the org’s JWT signing keypair with JWKS overlap (distinct from KEK rotation).

Full runbook: JWT Signing Key Rotation.

Requirements summary: rotateKey: true, signingKeyOverlapSecondstokenTtlSeconds and ≤ site signing_key_overlap_max_sec; omit overlap when not rotating.

5. Troubleshooting

SymptomLikely causeAction
PUT config → 503Site [machine_identity] disabled or invalidFix Day 0 config; restart nico-api
PUT config → 400Invalid issuer, TTL out of bounds, bad overlapCheck OpenAPI validation messages; align with site TTL/overlap max
PUT delegation → 404No identity config for orgCreate config first
SignMachineIdentity → NotFoundNo/disabled org config, instance not READY, wrong machineGET config; check instance; confirm DPU cert matches instance
SignMachineIdentity → invalid audienceAudience not in allowlistUpdate allowedAudiences
IMDS 403/404/503Agent limits, missing config, or signing failureCheck agent logs; verify Core reachable from DPU
JWKS missing second key during rotationOverlap expired or rotation not committedRe-check GET config signingKeys
Token delegation HTTP errorsToken exchange service unreachable, proxy misconfigured, allowlist blockVerify token_endpoint_http_proxy and domain allowlist; test token exchange service from API network namespace

Admin inspection (Core gRPC)

Site operators with a Forge Admin CLI mTLS certificate can inspect config with grpcurl (no REST or nico-admin-cli subcommand for this RPC):

$grpcurl -cacert … -cert … -key … \
> -d '{"organization_id": "<org>"}' \
> carbide-api.forge:443 forge.Forge/GetTenantIdentityConfiguration

Client cert setup: Generating client certificates.


TaskDocument
Enable site-level machine identityDay 0 Machine Identity
Verify issuance end-to-endMachine Identity Verification
Rotate org JWT signing keysJWT Signing Key Rotation
Rotate site master encryption key (KEK)Master Encryption Key Rotation
Provision instances for tenantsTenant Management
DPU agent IMDS limitsDay 0 — DPU agent section
Full API and data modelSPIFFE JWT-SVID SDD

REST API details: Tenant Identity tag in the REST API Reference.