DocumentationAccess Control

Troubleshooting

View as Markdown

When something goes wrong with authentication or authorization, start here. Problems are organized by symptom.

”I Get 401 Unauthorized on Every Request”

Cause: Auth is enabled but no valid credentials are being sent.

Diagnosis:

  1. Check if auth is enabled:
$curl ${BASE_URL}/apis/auth/discovery

If auth_enabled is true, all requests must include credentials.

  1. Check your token status:
$nemo auth status

The CLI and SDK automatically refresh expired tokens. If automatic refresh fails (e.g., the refresh token itself has expired), re-login:

$nemo auth login
  1. Verify the token is being sent. For SDK/curl, ensure the Authorization: Bearer <token> header is present.

Common causes:

  • Refresh token has expired — re-login with nemo auth login
  • OIDC issuer mismatch — the token’s iss claim doesn’t match auth.oidc.issuer in the config
  • Audience mismatch — the token’s aud claim doesn’t match auth.oidc.audience (or client_id)
  • Using email-as-API-key (X-NMP-Principal-Id) when OIDC is enabled and native validation rejects it

”I Get 403 Forbidden but I Should Have Access”

Cause: Authentication succeeded, but authorization failed — the user lacks the required role or the token lacks the required scope.

Diagnosis:

  1. Check your role binding in the workspace:
$nemo workspaces members list --workspace <workspace-name>

Verify your email appears and has the expected role (Viewer, Editor, or Admin).

  1. Check your token scopes. Decode the JWT and verify the required scopes are present:
$nemo auth token --decode

Look for scp or scope — ensure platform:write is present for write operations.

  1. Account for role propagation delay. After adding a member, wait up to 30 seconds (or up to policy_data_refresh_interval) for the change to take effect.

Common causes:

  • Missing role binding — you’re not a member of the workspace
  • Insufficient role — you have Viewer but need Editor for create/update/delete
  • Token scope too narrow — logged in with --scope "platform:read" but trying to write
  • Role not yet propagated — just added as a member; wait and retry

”nemo auth login Fails with an IdP Error”

AADSTS70011 (Azure AD — Invalid Scope)

The requested scope is not configured or admin consent has not been granted.

Fix: In Azure Portal → App Registration → API Permissions, ensure:

  • Custom scopes (platform:read, platform:write) are defined under “Expose an API”
  • Admin consent is granted for the scopes

”Device flow not enabled” or “Public client flows not allowed”

Your IdP doesn’t have device flow enabled for the NeMo Platform application.

Fix:

  • Azure AD: App Registration → Authentication → “Allow public client flows” → Yes
  • Okta: Application → General → “Device Authorization” grant type must be enabled
  • Keycloak: Client → Settings → “OAuth 2.0 Device Authorization Grant” → On

Client ID Mismatch

The client_id in NeMo Platform config doesn’t match the application in your IdP.

Fix: Verify auth.oidc.client_id matches the client ID in your IdP exactly.

”I Can’t Delete a Workspace”

Cause: The workspace contains resources, or you don’t have the Admin role.

Diagnosis:

  1. Check the error message — it lists which entity types exist:
1{
2  "detail": "Cannot delete workspace 'ml-team': workspace contains entities that must be deleted first: project (3), dataset (5)"
3}

  1. Delete all resources in the workspace first, then retry.

  2. Verify you have the Admin role:

$nemo workspaces members list --workspace <workspace-name>

“Gateway Bearer-Auth Callout Isn’t Working”

Cause: Headers are not being set or stripped correctly by the gateway.

Diagnosis:

  1. Check that the gateway calls /apis/auth/authenticate with the original Authorization header before forwarding protected requests.

  2. Check that the gateway forwards the trusted X-NMP-Principal-* and X-NMP-Scopes headers returned by the auth service on successful authentication.

  3. Check that auth headers are stripped from external requests. Try sending a request with X-NMP-Principal-Id or X-NMP-Scopes from outside the cluster — those headers should be stripped by the gateway.

  4. Verify the PDP is reachable from platform services. If using external OPA, check that the OPA sidecar/service is running and the bundle endpoint is accessible.

”Role Change Doesn’t Take Effect”

Cause: Policy data propagation delay.

The PDP caches role binding data. After adding or removing a member:

  • Embedded PDP: Wait up to policy_data_refresh_interval seconds (default: 30).
  • External OPA: Wait up to 2 × bundle_cache_seconds (default: ~10 seconds).

The CLI and SDK members commands wait for propagation by default. If you’re using the API directly with wait_role_propagation=false, account for the delay.

”PlatformAdmin Can’t Access a Workspace”

Cause: The principal identity doesn’t match admin_email exactly.

Diagnosis:

  1. Check the admin_email in your config — it’s case-sensitive.
  2. Verify the email in your token matches. Decode the JWT and check the email/upn claim.
  3. If using Azure AD with email_claim: "upn", ensure the UPN matches admin_email exactly.

Getting Help

If these steps don’t resolve the issue:

  1. Check service logs for authorization error details
  2. Check the PDP health — if using embedded PDP, verify the auth service is running; if using external OPA, verify OPA is running and fetching bundles
  3. Decode your JWT token and verify claims match the platform configuration (issuer, audience, email claim)