Configuring NVKE API Authentication

How to make NVKE require a token on its API, using an identity provider you already run.

This is the operator’s side. For getting a token and making a call once it is on, see Call the API in Secure the NVKE API.

What you are setting up

NVKE can require a signed JWT on every /api/* request, in one of these states:

Permissive exists so you can turn this on somewhere that already has callers and find out who would break before they do. Use it.

A token is accepted when its issuer is one you configured, its signature verifies, it names your deployment in its aud claim, it has not expired, and its subject matches a rule you wrote. The first four failing gives a 401. The last one gives a 403: the token was real, the caller just isn’t on the list.

NVKE does no authorization. Every admitted caller can call every endpoint.

issuers is a list. Add an entry for each identity provider. name and issuer are each unique in the file. An issuer’s audiences replaces the top-level list for that issuer. An issuer carries one of serviceAccounts, spiffe, or groups.

Before you start

You need permission to set Helm values on the NVKE release, and a cluster running a chart version with an apiAuth section:

CHART=...   # your NVKE chart reference
helm show values "$CHART" | grep apiAuth

On an existing site, where the controller runs as cloud-workflow-engine, replace nke-controller with cloud-workflow-engine in every command on this page.

The commands below use $NS for the namespace NVKE runs in:

NS=nke-controller   # your release namespace

Recipe: your cluster’s own API server

This is the common case, and it needs no extra infrastructure. Kubernetes already signs tokens for service accounts and publishes the keys to verify them.

1. Read your cluster’s issuer

Read it rather than assuming it:

kubectl get --raw /.well-known/openid-configuration

{
  "issuer": "https://kubernetes.default.svc.cluster.local",
  "jwks_uri": "https://10.0.0.1:6443/openid/v1/jwks"
}

Take the issuer value.

Do not take jwks_uri. It is built from the API server’s advertise address, so it is a node IP or a load-balancer address that can change. Use the in-cluster name instead, which is stable and serves the same thing:

https://kubernetes.default.svc/openid/v1/jwks

2. Create the identity your callers will use

NVKE admits callers by name, so the account has to exist before you name it in the configuration. The Helm chart does not create these objects.

kubectl create namespace nke-clients
kubectl -n nke-clients create serviceaccount nke-client

nvkectl mints this pair and no other. The names nke-clients / nke-client are fixed in the CLI, and it looks for the authentication ConfigMap under two fixed names: nvke-auth-config, then nke-auth-config for a site that keeps the older name by pinning apiAuth.configMap.name. It searches both in namespace nke-controller, then both in cloud-workflow-engine, and uses the first it can read.

A ConfigMap it is not permitted to read does not stop the command: the search moves to the next candidate, so a kubeconfig scoped to the pinned name alone still works. The denial is reported only if no candidate turns out to be readable. Any other error reading a ConfigMap stops the command. In any other namespace, nvkectl finds no ConfigMap and sends no token. Other names work for curl and hack/nvke-token.sh if you override them; they will not work for nvkectl. One account per caller is still easier to revoke when those callers are not the CLI.

Minting a token for that account needs create on serviceaccounts/token in its namespace. A cluster administrator already has it. To let something else mint it, such as nvkectl under a least-privilege kubeconfig, a CI job, or an application that renews its own token, grant it:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: nke-client-token-minter
  namespace: nke-clients
rules:
  - apiGroups: [""]
    resources: ["serviceaccounts/token"]
    resourceNames: ["nke-client"]
    verbs: ["create"]

resourceNames holds that grant to the one account, so it does not become a way to mint tokens for anything else in the namespace. Bind it to whoever is doing the minting: the user or group in the kubeconfig nvkectl uses.

3. Write the configuration

audiences: ["https://nke.example.com"]
issuers:
  - name: local-kubernetes
    issuer: "https://kubernetes.default.svc.cluster.local"
    jwks:
      uri: "https://kubernetes.default.svc/openid/v1/jwks"
      caBundlePath: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearerTokenPath: /var/run/secrets/kubernetes.io/serviceaccount/token
    algorithms: [RS256, ES256]
    admit:
      serviceAccounts:
        - namespace: nke-clients
          name: nke-client

audiences is what callers must name in their token. Pick something unique to this deployment; its external URL is a good choice. Do not reuse your issuer or a kubernetes.default.svc address: every pod in the cluster already carries those, so the check would admit everything.

admit is the allow-list, and it is what makes this safe. Anyone who can create a pod can have Kubernetes mint a token for their own service account naming any audience they like. Verifying the signature is not enough on its own; the allow-list is what stops every service account in the cluster from being an NVKE administrator. Omitting name admits every service account in that namespace.

The two CA and token paths are files Kubernetes puts in every pod, so they work as written.

4. Set the chart values

apiAuth:
  enabled: true
  permissive: true
  configMap:
    create: true
    name: nvke-auth-config
  config:
    # the YAML from step 3 goes here
    audiences: ["https://nke.example.com"]
    issuers:
      - name: local-kubernetes
        # ...

Start permissive. The next section covers going enforcing.

If you would rather manage the config yourself, with a sealed secret or an existing GitOps pipeline, leave create at its default of false and put the document in a ConfigMap named by configMap.name, under the key auth.yaml:

kubectl -n "$NS" create configmap nvke-auth-config \
  --from-file=auth.yaml=auth.yaml \
  --dry-run=client -o yaml | kubectl apply -f -

Set one or the other. Setting config without create, or create without config, fails the install rather than silently ignoring one of them.

5. Check it came up

kubectl -n "$NS" logs deploy/nke-controller | grep 'API authentication'

{"msg":"fetched issuer signing keys","issuer":"local-kubernetes","keys":1}
{"msg":"API authentication is enabled","mode":"permissive","issuers":["local-kubernetes"]}

You want one fetched issuer signing keys line per issuer. If one is missing, fix it now: while permissive, a failed fetch is only a warning; enforcing, it stops NVKE from starting at all.

6. Tell callers what to ask for

nvkectl mints on its own once config set-site --kubeconfig points at this cluster. The kubeconfig identity needs the Role above, the service account must exist, and the admit rule must list nke-clients/nke-client.

For curl and scripts:

kubectl create token nke-client -n nke-clients \
  --audience=https://nke.example.com --duration=30m

This has to be kubectl create token. The older approach, reading a ServiceAccount Secret, produces a token with no audience, which NVKE refuses.

Recipe: SPIRE

Use this if you already run SPIRE. The shape is the same; the differences are that subjects are SPIFFE IDs rather than service accounts, and you have to supply a CA certificate yourself.

1. Read the issuer, and check the certificate

kubectl -n spire-server exec deploy/spire-spiffe-oidc-discovery-provider -- \
  curl -sk https://127.0.0.1/.well-known/openid-configuration

As above, take issuer and be careful about the keys URL. Whatever address you choose has to appear in the certificate SPIRE serves, or the connection fails:

kubectl -n spire-server get secret spire-spiffe-oidc-discovery-provider-cert \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -ext subjectAltName

If the only usable name is an IP address, use it, but know you’re pinning something that may move.

2. Get the CA certificate into NVKE

SPIRE’s endpoint uses its own certificate, not one from a public CA, so NVKE needs the CA to trust it:

kubectl -n spire-server get secret oidc-provider-upstream-ca \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > spire-ca.crt

NVKE reads it from a file inside its container. It goes beside the configuration, under extraFiles: the whole ConfigMap is mounted as a directory, so anything listed here is a file next to auth.yaml:

apiAuth:
  configMap:
    create: true
    extraFiles:
      spire-ca.crt: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----

It is then at /etc/nke/auth/spire-ca.crt, which is what caBundlePath points at below. A CA bundle is a public certificate, not a secret.

If you manage the ConfigMap yourself instead, put the file in it directly:

kubectl -n "$NS" create configmap nvke-auth-config \
  --from-file=auth.yaml=auth.yaml \
  --from-file=spire-ca.crt=spire-ca.crt \
  --dry-run=client -o yaml | kubectl apply -f -

3. Write the issuer

issuers:
  - name: spire
    issuer: "https://spire.example.com"
    jwks:
      uri: "https://spire.example.com:30443/keys"
      caBundlePath: /etc/nke/auth/spire-ca.crt
    algorithms: [RS256, ES256, ES384]
    admit:
      spiffe:
        trustDomain: example.org
        pathPrefixes: ["/nke-service/"]

pathPrefixes is required and cannot be /. A trust domain on its own would admit every workload in it, which is rarely what you want.

An issuer uses one of serviceAccounts, spiffe, or groups.

4. Tell callers what to ask for

spire-server jwt mint \
  -spiffeID spiffe://example.org/nke-service/my-service \
  -audience https://nke.example.com \
  -ttl 10m

-ttl takes a duration like 10m, not a number of seconds.

Recipe: an external identity provider

Use this when callers get tokens from an identity provider you already run. NVKE reads group membership only from the token’s groups claim.

1. Write the issuer

Add this issuer to a configuration that already contains at least one top-level audiences value. The block below is not a complete configuration. NVKE refuses to start when audiences is missing.

issuers:
  - name: example-idp
    issuer: "https://idp.example.com/realms/example"
    jwks:
      uri: "https://idp.example.com/keys"
    algorithms: [RS256]
    admit:
      groups:
        - nke-operators

A token is admitted when its groups claim contains any listed group. The group names are matched exactly.

When the identity provider’s certificate is signed by a private CA, put the bundle in extraFiles and set jwks.caBundlePath, as in the SPIRE recipe. Omit caBundlePath when a public CA signs the certificate.

Going enforcing without breaking callers

Do not skip permissive. It is the only way to find callers you did not know about.

Watch the counter, not the log. Successful reads are logged at debug level, so the log will undercount them:

kubectl -n "$NS" exec deploy/nke-controller -- \
  curl -s http://127.0.0.1:9090/metrics | grep '^nvke_auth_requests_total'

nvke_auth_requests_total{issuer="local-kubernetes",outcome="authenticated"} 42
nvke_auth_requests_total{issuer="unknown",outcome="would_reject",reason="no_token"} 7

Every would_reject series is a caller that enforcing would break. The reason label says what each one needs: usually a token, or a subject added to your allow-list. Rejections are also logged in full:

kubectl -n "$NS" logs deploy/nke-controller | grep '"logger":"auth"'

When would_reject stops climbing, set apiAuth.permissive: false.

Add new issuers while still permissive. Enforcing NVKE will not start if it cannot reach every issuer’s keys, so a bad issuer becomes an outage rather than a warning.

Reference

Configuration fields

The file is read once at startup, so restart NVKE after you change it:

kubectl -n "$NS" rollout restart deploy/nke-controller

Values are matched exactly: leading or trailing whitespace is an error, not something NVKE trims.

Chart values

Why a request was rejected

Everything is a 401 except not_admitted, which is a 403.

Responses name the reason and nothing else. Issuer names and configuration stay in the log.

Endpoints that need no token

/healthz, /readyz, /api/version, /openapi.yaml, /openapi-public.yaml, /docs, and the Swagger UI under /api/docs/.

Everything else needs one, including paths that route nowhere, so a caller cannot probe for endpoints without authenticating. This is also why a successful call to /api/version tells you nothing about whether authentication is on.

Troubleshooting

NVKE will not start, and the log says it could not set up authentication. When enforcing, every issuer’s keys must be reachable at startup. The message names the issuer and the error. Check the keys URL is reachable from inside the pod, and that any caBundlePath points at a file that is actually mounted. If it sits under /etc/nke/auth, it has to be listed in extraFiles.

Everything returns 401, including tokens that look right. Check the audience first: it’s the usual cause. A token minted for the wrong audience is rejected identically to no token at all. Compare its aud claim against the audiences NVKE logs at startup.

A caller gets 403. Its token is fine and its subject is not on the allow-list. The log names the subject; compare it to your admit rules. Re-minting the token will not help.

NVKE rejected the configuration at startup. The error names the offending field. Usual causes: an audience that matches one of your issuers, a value with stray whitespace, serviceAccounts and spiffe on the same issuer, or a keys URL that is not https.

Config changes have no effect. It is read once at startup. Restart the deployment.

You cannot tell whether authentication is on. Read it off the deployment rather than probing the API:

kubectl -n "$NS" get deploy nke-controller \
  -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="AUTH_CONFIG_PATH")].value}{"\n"}'

See also

• Secure the NVKE API: the operator path, and the caller’s side including nvkectl.