Machine Identity Verification

Runbook for verifying that machine identity (SPIFFE JWT-SVID issuance) is configured correctly: discovery documents, JWKS, and the workload IMDS path.

Use this after Day 0 and Day 1 configuration, or when troubleshooting token issuance.

Prerequisites

• Site [machine_identity].enabled = true and valid encryption keys (Day 0).
• Per-org tenant-identity/config exists with enabled: true (Day 1).
• At least one instance in READY assigned to the org under test.
• Network access from a workload on that instance to 169.254.169.254 (IMDS).

Collect {org}, {site-id}, and an allowed audience (for example tenant-api) from your identity config:

$
curl -sS -H "Authorization: Bearer $TOKEN" \
>
  "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/tenant-identity/config"

Verification Checklist

IMDS is the operator-facing check. The DPU agent calls Core gRPC SignMachineIdentity internally on that path — you do not need to invoke that RPC directly for routine verification. See DPU-side debugging when isolating agent vs Core failures.

1. OIDC Discovery

$
curl -sS "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/.well-known/openid-configuration" | jq .

nicocli TUI: tenant-identity openid-configuration get

Confirm:

• issuer matches the value in your tenant identity config.
• jwks_uri points at the org/site JWKS URL under the same REST base.
• Document is reachable from wherever your verifiers run (ingress, mTLS, or internal DNS as designed).

2. JWKS Endpoints

OIDC JWKS (JWT verifiers):

$
curl -sS "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/.well-known/jwks.json" | jq .

SPIFFE JWKS:

$
curl -sS "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/.well-known/spiffe/jwks.json" | jq .

nicocli TUI: tenant-identity jwks get, tenant-identity spiffe-jwks get

Confirm:

• Response is a JWK Set with at least one key.
• Key type/curve matches site algorithm (ES256 → kty: EC, crv: P-256).
• During signing-key rotation, two keys may appear until the overlap window ends.

3. Align Config with JWKS

Compare GET config signingKeys with JWKS:

$
curl -sS -H "Authorization: Bearer $TOKEN" \
>
  "https://<nico-rest>/v2/org/{org}/nico/site/{site-id}/tenant-identity/config" | jq '.signingKeys'

Mismatch between config and JWKS usually indicates a stale cache, incomplete rotation, or REST/Core sync delay — re-fetch after a short wait; if persistent, check Core logs and repeat GET/JWKS.

4. IMDS (Workload Path)

From a workload network namespace on the instance:

$
# JSON (default)
$
curl -sS -H 'Metadata: true' \
>
  'http://169.254.169.254/latest/meta-data/identity?aud=<allowed-audience>'
$
$
# Plain JWT
$
curl -sS -H 'Metadata: true' -H 'Accept: text/plain' \
>
  'http://169.254.169.254/latest/meta-data/identity?aud=<allowed-audience>'

Success (JSON): HTTP 200 with access_token, token_type, expires_in, and issued_token_type.

Notes:

• The Metadata: true header is required (AWS IMDS compatibility).
• Multiple audiences: repeat aud= query parameters; URL-encode values with special characters.
• Rate limits come from the DPU agent [machine-identity] section — see Day 0. HTTP 429 indicates throttling.

Token delegation path

If the org uses token delegation, IMDS still returns a workload token, but Core may exchange via the configured STS. Verification steps are the same at the IMDS layer; additionally confirm your STS receives the intermediate token and returns the final token (STS logs/metrics).

5. Decode and Validate JWT Claims

Extract the JWT from IMDS JSON (access_token) or the plain IMDS body, then decode (offline):

$
# Header
$
echo '<jwt>' | cut -d. -f1 | base64 -d 2>/dev/null | jq .
$
$
# Payload
$
echo '<jwt>' | cut -d. -f2 | base64 -d 2>/dev/null | jq .

Or use jwt.io in non-production environments only — do not paste production tokens into third-party sites.

Signature verification: fetch JWKS (§2), locate the key by kid in the JWT header, verify ES256 signature with your JWT library or:

$
# Example with jwt-cli (if installed): jwt verify --alg ES256 --jwks <url> '<jwt>'

Troubleshooting

DPU-side debugging (optional)

When IMDS fails, check the DPU agent first: logs, [machine-identity] rate limits, and sign-timeout-secs (Day 0).

If the agent has sign-proxy-url set, IMDS forwards to that HTTP service instead of calling Core directly. Test the proxy from the DPU with the same request IMDS would send:

$
curl -sS -H 'Metadata: true' \
>
  --cacert /etc/forge/sign_proxy_root.pem \
>
  'https://sign-proxy.example.com/prefix/latest/meta-data/identity?aud=<allowed-audience>'

Use --cacert when sign-proxy-tls-root-ca is configured; omit it for http: URLs or when the proxy uses a public CA.

Reference: SignMachineIdentity gRPC (optional)

This is not part of routine operator verification — IMDS (§4) is sufficient for end-to-end checks on the default path.

Keep it as a reference if you operate a custom HTTP sign proxy (sign-proxy-url) whose implementation calls forge.Forge/SignMachineIdentity on the backend. Use it to validate the gRPC leg independently while developing or troubleshooting proxy code.

From a host that holds the DPU machine certificate (/opt/forge/machine_cert.pem, paths may vary):

$
grpcurl \
>
  -cacert /opt/forge/forge_root.pem \
>
  -cert /opt/forge/machine_cert.pem \
>
  -key /opt/forge/machine_cert.key \
>
  -d '{"audience": ["<allowed-audience>"]}' \
>
  carbide-api.forge:443 forge.Forge/SignMachineIdentity

If this call succeeds but IMDS via your proxy fails, the issue is in the proxy HTTP layer (URL, TLS, headers, timeouts) — not Core signing.

Related Documentation

• Machine Identity (Day 1) — configure issuer, audiences, delegation
• JWT Signing Key Rotation — rotate org signing keys and re-verify JWKS
• Master Encryption Key Rotation (KEK) — site master key (does not change JWKS)
• Day 0 Machine Identity — site enablement