Janitor Provider Configuration

Overview

The Janitor Provider is the CSP-specific backend that executes node operations (reboot, terminate) on cloud infrastructure. The Janitor connects to the Janitor Provider over gRPC with TLS and ServiceAccount token authentication. This document covers all Helm configuration options for system administrators.

Configuration Reference

Module Enable/Disable

Controls whether the janitor-provider module is deployed in the cluster.

global:
  janitorProvider:
    enabled: true

Logging

Sets the verbosity level for janitor-provider logs.

janitor-provider:
  logLevel: info  # Options: debug, info, warn, error

Replica Count

janitor-provider:
  replicaCount: 1

Resources

Defines CPU and memory resource requests and limits for the janitor-provider pod. No defaults are set; configure for production use.

janitor-provider:
  resources: {}

TLS

The Janitor Provider serves gRPC over TLS. The certificate is issued and renewed by cert-manager, which is a required dependency.

janitor-provider:
  tls:
    enabled: true
    certDir: "/etc/nvsentinel/janitor-provider/tls"
    issuerName: "janitor-selfsigned-issuer"
    secretName: ""

enabled

When true, the gRPC server requires TLS. Must match janitor.config.cspProvider.tls.enabled.

certDir

Path inside the container where the TLS certificate and key are mounted.

issuerName

Name of the cert-manager ClusterIssuer used to issue the gRPC server certificate.

secretName

Name of the Kubernetes Secret where the issued certificate is stored. When empty, the chart generates a name automatically.

Authentication

janitor-provider:
  auth:
    enabled: true
    audiences:
      - "nvsentinel-csp-provider"

enabled

When true, the Janitor Provider validates the ServiceAccount token presented by the Janitor on every gRPC call.

audiences

List of accepted token audiences. Must include the value set in janitor.config.cspProvider.auth.audience.

Cloud Provider Selection

janitor-provider:
  csp:
    provider: "kind"  # Options: kind, kwok, aws, gcp, azure, oci, nebius, lambda, akamai, generic, label

Only one provider is active at a time. kind and kwok are for development only; they simulate reboots without contacting any cloud API.

AWS

Uses IRSA (IAM Roles for Service Accounts) for authentication. The cluster must have OIDC federation configured, and the IAM role must trust the Janitor Provider’s Kubernetes ServiceAccount.

janitor-provider:
  csp:
    provider: "aws"
    aws:
      region: ""
      accountId: ""
      iamRoleName: ""

region

AWS region where the EC2 instances are running (e.g. us-west-2).

accountId

12-digit AWS account ID hosting the EKS cluster.

iamRoleName

Name of the IAM role used for IRSA. The role must include ec2:RebootInstances and ec2:TerminateInstances permissions. See CSP Health Monitor IAM Setup for the general Workload Identity pattern; apply the same approach for the Janitor Provider ServiceAccount.

GCP

Uses Workload Identity to authenticate the Janitor Provider’s Kubernetes ServiceAccount as a GCP Service Account.

janitor-provider:
  csp:
    provider: "gcp"
    gcp:
      project: ""
      zone: ""
      serviceAccount: ""

project

GCP project ID where the GKE cluster and Compute Engine instances reside.

zone

GCP zone of the instances (e.g. us-central1-a).

serviceAccount

GCP Service Account name, without the @project.iam.gserviceaccount.com suffix. The SA must have compute.instances.reset and compute.instances.stop permissions. See CSP Health Monitor IAM Setup for the Workload Identity binding pattern.

Azure

Uses Azure Workload Identity to authenticate the Janitor Provider’s Kubernetes ServiceAccount as a Managed Identity.

janitor-provider:
  csp:
    provider: "azure"
    azure:
      subscriptionId: ""
      resourceGroup: ""
      location: ""
      clientId: ""

subscriptionId

Azure subscription ID containing the AKS cluster and VM resources.

resourceGroup

Azure resource group where the VM Scale Set or individual VMs are managed.

location

Azure region of the resources (e.g. eastus).

clientId

Client ID of the Managed Identity used for Workload Identity authentication. The identity must have Microsoft.Compute/virtualMachines/restart/action and Microsoft.Compute/virtualMachines/delete/action permissions on the relevant resource group.

OCI

Uses OCI Workload Identity or a credentials file for authentication.

OCI maps all RebootNode resources to the RESET action. RESET powers off the instance immediately without waiting for the operating system. Drain workloads before you create the resource because an immediate power-off can cause data corruption.

janitor-provider:
  csp:
    provider: "oci"
    oci:
      region: ""
      compartment: ""
      credentialsFile: ""
      profile: "DEFAULT"
      principalId: ""

region

OCI region identifier (e.g. us-phoenix-1).

compartment

OCID of the OCI compartment containing the compute instances.

credentialsFile

Path to an OCI credentials file inside the container. Leave empty to use Workload Identity (recommended for production).

profile

Profile name within the credentials file. Defaults to DEFAULT. Ignored when credentialsFile is empty.

principalId

OCI principal OCID used for Workload Identity. Required when credentialsFile is empty.

Nebius

For Nebius Managed Kubernetes (MK8s) clusters. The provider uses the official Nebius Go SDK and authenticates with a service account key or an IAM token. The identity needs the compute.instances.stop and compute.instances.start permissions.

janitor-provider:
  csp:
    provider: "nebius"
    nebius:
      serviceAccountKeySecret: "nebius-sa-credentials"

Configure exactly one credential. Prefer a service account key: IAM tokens expire and need a manual refresh, so they suit testing rather than production.

serviceAccountKeySecret

Secret holding the service account key file under the key sa-credentials.json. The chart mounts it at /etc/nebius and sets NEBIUS_SA_KEY_FILE for you. This is the recommended production option.

serviceAccountKeyFile

Path to a service account key file already present inside the container, in the JSON format the Nebius SDK documents. Passed as NEBIUS_SA_KEY_FILE. Use this when you mount the credential yourself.

iamToken

A Nebius IAM token supplied directly, passed as NEBIUS_IAM_TOKEN. Obtain one with nebius iam get-access-token. For testing only — the token expires and remediation stops working when it does.

iamTokenSecretRef

Secret holding an IAM token, as an alternative to writing the token into Helm values. Keeps the token out of your values file, but it still expires.

janitor-provider:
  csp:
    nebius:
      iamTokenSecretRef:
        name: nebius-janitor-token
        key: token

Lambda

Reboot maps to the Lambda power-cycle operation (a host-level power cycle, not a guest restart) and terminate maps to the Lambda terminate operation. Nodes must carry spec.providerID in the form lambda://<instanceID>.

Authenticate either with a static API key or with workload identity. Workload identity is preferred: no Lambda credential is stored in the cluster, and the token it uses is short-lived and rotated automatically.

janitor-provider:
  csp:
    provider: "lambda"
    lambda:
      apiKeySecretRef:
        name: "lambda-api-key"
        key: "LAMBDA_API_KEY"

apiEndpoint

Optional. Overrides the base URL of the Lambda Cloud API, passed as LAMBDA_API_ENDPOINT. Defaults to https://cloud.lambda.ai when unset. Must be https, and must not carry userinfo, a query string, or a fragment. The host is checked against a fixed allowlist built into the provider, so an unapproved host fails at startup rather than on the first remediation.

apiKeySecretRef

Secret holding the Lambda API key, injected as LAMBDA_API_KEY. name is required when csp.provider is lambda, unless workloadIdentity.identityLRN is set; the install fails without one of the two. key defaults to LAMBDA_API_KEY. The same Secret can be shared with the CSP Health Monitor, which reads the key the same way. The key needs permission to power cycle and terminate instances.

workloadIdentity.identityLRN

The service identity to assume, in the form lrn:iam:identity:<id>. Setting it annotates the janitor-provider ServiceAccount with lambda.ai/identity-lrn. The pod is then given a short-lived Kubernetes token, which the provider exchanges for a Lambda API key and refreshes before it expires — so no Lambda credential is stored in the cluster.

janitor-provider:
  csp:
    provider: "lambda"
    lambda:
      workloadIdentity:
        identityLRN: "lrn:iam:identity:3cd2d107c6a347eeb0ef9498820d637d"

When set, apiKeySecretRef is not required and no LAMBDA_API_KEY is placed in the pod. If both are configured, workload identity wins and the static key is ignored; the provider logs which one it selected at startup (authMode).

On the Lambda side, the identity must exist, hold permission to power cycle and terminate instances, and trust this cluster’s ServiceAccount.

Setting up the identity

Run once per cluster, with an admin API key. $WS is the workspace the cluster’s instances belong to.

# Stop at the first failed call: a half-applied setup prints an identityLRN
# that looks usable but has no role or no trust behind it.
set -euo pipefail
KEY=<admin-api-key>
export WS=<workspace-id>
lambda_curl() {
  printf 'Authorization: Bearer %s\n' "$KEY" | curl -sf -H @- "$@"
}
# 1. Create a service identity for the Janitor Provider.
SID=$(lambda_curl "https://cloud.lambda.ai/api/v1/identities" \
  -H 'Content-Type: application/json' \
  -d '{"display_name":"nvsentinel-janitor-provider"}' | jq -r .data.id)
[ -n "$SID" ] && [ "$SID" != "null" ] || { echo "identity was not created" >&2; exit 1; }
# 2. Add it to the workspace. A workspace-scoped role assignment needs membership.
lambda_curl "https://cloud.lambda.ai/api/v1/workspaces/$WS/memberships" \
  -H 'Content-Type: application/json' \
  -d "{\"member_type\":\"identity\",\"member_id\":\"$SID\"}"
# 3. Assign the built-in roles, scoped to that workspace.
for role in instance-reader instance-power-cycle instance-terminate; do
  ROLE=$(lambda_curl "https://cloud.lambda.ai/api/v1/roles" \
    | jq -r --arg n "$role" '.data[] | select(.name==$n) | .id')
  [ -n "$ROLE" ] && [ "$ROLE" != "null" ] || { echo "role $role not found" >&2; exit 1; }
  lambda_curl "https://cloud.lambda.ai/api/v1/identities/$SID/role-assignments" \
    -H 'Content-Type: application/json' \
    -d "{\"role_id\":\"$ROLE\",\"scope\":{\"type\":\"workspace\",\"workspace_id\":\"$WS\"}}"
done
# 4. Trust this cluster's ServiceAccount to assume the identity. Registering the
#    issuer is an idempotent upsert keyed on the issuer URL, so re-running is safe.
SUBJECT=$(
  kubectl get deployments --all-namespaces \
    -l app.kubernetes.io/name=janitor-provider -o json |
    jq -er '
      .items
      | if length == 1 then .[0] else error("expected exactly one janitor-provider Deployment") end
      | "system:serviceaccount:\(.metadata.namespace):\(.spec.template.spec.serviceAccountName)"
    '
)
ISS=$(kubectl get --raw /.well-known/openid-configuration |
  jq -er '.issuer | select(type == "string" and length > 0)') ||
  { echo "OIDC discovery returned no valid issuer" >&2; exit 1; }
kubectl get --raw /openid/v1/jwks > /tmp/jwks.json
lambda_curl "https://cloud.lambda.ai/api/v1/oidc-providers" \
  -H 'Content-Type: application/json' \
  -d "$(jq -n --arg iss "$ISS" --arg lrn "lrn:iam:identity:$SID" \
        --arg subject "$SUBJECT" --slurpfile jwks /tmp/jwks.json \
        '{issuer_url:$iss, jwks:$jwks[0],
          trusts:[{identity_lrn:$lrn, subject:$subject}]}')"
echo "identityLRN: lrn:iam:identity:$SID"

The printed LRN is what goes in the chart value above.

All three are workspace-scoped, so the identity can only act on instances in $WS.

Drop instance-terminate from the loop if your remediation policy only ever reboots — the roles are additive, so granting just the two leaves terminate unauthorized.

Verifying

The identity is attached when the pod is created, so it only appears on pods created after the annotation lands. Restart the deployment if you added it to a running install.

(
  set -euo pipefail
  NAMESPACE=$(
    kubectl get deployments --all-namespaces \
      -l app.kubernetes.io/name=janitor-provider -o json |
      jq -er '
        .items
        | if length == 1 then .[0] else error("expected exactly one janitor-provider Deployment") end
        | .metadata.namespace
      '
  )
  # The identity the pod received.
  kubectl -n "$NAMESPACE" get pod -l app.kubernetes.io/name=janitor-provider \
    -o jsonpath='{.items[0].spec.containers[0].env[?(@.name=="LAMBDA_IDENTITY_LRN")]}'
  # Which credential the provider selected.
  kubectl -n "$NAMESPACE" logs -l app.kubernetes.io/name=janitor-provider | grep authMode
)

Expect authMode=workload-identity. If it says api-key, the pod never received an identity — check that the annotation is on the ServiceAccount and that the pod was created after it landed.

Failures in the exchange itself surface on the first remediation, not at startup, because the token is minted lazily. When the exchange endpoint rejects the token it returns 401 with no detail by design, so an unauthenticated caller cannot probe for which identities exist; that means a missing trust, a wrong identity LRN and a disabled account all look identical from the client. Check the trust and the identity LRN first.

Akamai

Reboot maps to the Linode reboot operation. Terminate recycles the node through LKE when the instance belongs to an LKE cluster, which deletes it and provisions a replacement so the node pool keeps its size, and deletes the instance directly otherwise. Nodes must carry spec.providerID in the form linode://<instanceID>, which the Linode cloud controller manager sets on LKE nodes.

Warning: a terminate replaces the node, so everything on the node’s own disk is lost. This includes hostPath and emptyDir volumes and local PersistentVolumes. Move any data you need off the node before you let the janitor terminate it.

Authenticate with a static API token. A reboot is reported complete only when both conditions hold: the instance reports running, and the kubelet’s bootID differs from the one recorded before the reboot. The bootID check is the one that matters, because the instance can still read running for a while after the API accepts the request.

global:
  janitor:
    enabled: true
  janitorProvider:
    enabled: true
janitor-provider:
  csp:
    provider: "akamai"
    akamai:
      apiTokenSecretRef:
        name: "akamai-api-token"
        key: "LINODE_TOKEN"

apiEndpoint

Optional. Overrides the base URL of the Akamai Cloud API, passed as AKAMAI_API_ENDPOINT. Defaults to https://api.linode.com/v4 when unset. Must be https, and must not carry userinfo, a query string, or a fragment. The host must be api.linode.com and the path must be /v4, so any other host or path fails at startup rather than on the first remediation.

apiTokenSecretRef

Secret holding the Akamai Cloud API token, injected as LINODE_TOKEN, the variable the Linode CLI and SDKs read. name is required when csp.provider is akamai; the install fails without it. key defaults to LINODE_TOKEN. The token needs the linodes:read_write scope to reboot and delete instances and the lke:read_write scope to recycle LKE nodes. A recycle also reads the instance, so lke:read_write alone is not enough: the token needs a linodes scope as well, and linodes:read_write covers it. A restricted identity that Akamai limits by permission needs view_linode, because the provider reads the instance before every reboot, delete and recycle and again while it waits for a reboot to finish, and then reboot_linode for reboots, delete_linode to delete an instance that is not an LKE node, and the lkecluster_admin role to recycle LKE nodes. The scopes were checked against the live API. The permission and role names come from the Akamai API reference and were not tested. The pod reads the token from its environment when it starts, so after you rotate the Secret, restart the janitor-provider pods, for example with kubectl rollout restart deployment/janitor-provider -n nvsentinel. The CSP Health Monitor reads the same variable but only lists maintenance events, so give it its own read-only token rather than sharing this one.

Generic / Bare-Metal

The generic provider spawns a privileged Kubernetes Job on the target node that runs chroot /host reboot to reboot the node. Use this for bare-metal clusters, on-premises deployments, or any environment not covered by the cloud provider backends.

The provider records the node boot ID when it creates the Job and passes it to the container in the PRE_REBOOT_BOOT_ID environment variable. Before any other action, the container compares this value with /proc/sys/kernel/random/boot_id. If the values are different, the node has already rebooted, so the container exits without a reboot and without a syslog entry. This check stops a second reboot when the kubelet starts the same pod again after the node boots. If the container cannot read either boot ID, it exits with an error and does not reboot the node.

janitor-provider:
  csp:
    provider: "generic"
    generic:
      rebootImage: "public.ecr.aws/docker/library/busybox:1.37.0"
      useSysrqReboot: false
      rebootJobNamespace: ""
      rebootJobTTLSeconds: 3600
      imagePullSecrets: ""
      writeSyslog: false

rebootImage

Container image used by the reboot Job. Must include sh, cat and chroot. Every reboot method runs through sh -c, because the boot ID check needs a shell. If the image has no sh, the container does not start and the node does not reboot.

useSysrqReboot

When false (default), the Job runs chroot /host reboot to trigger a clean OS shutdown and restart.

When true, the Job writes b to /proc/sysrq-trigger, triggering an immediate kernel reboot via the Linux Magic SysRq interface. Use this only when chroot /host reboot is unavailable — for example, when the node OS has a read-only root filesystem or a custom OS image that does not expose a reboot binary at the standard path.

rebootJobNamespace

Kubernetes namespace where the reboot Job is created. Defaults to the janitor-provider namespace when empty.

rebootJobTTLSeconds

Time in seconds after Job completion before Kubernetes deletes the Job and its pod. Defaults to 3600.

imagePullSecrets

Name of an image pull secret to attach to the reboot Job, if rebootImage is pulled from a private registry.

writeSyslog

When true, the Job writes an attribution entry to the node’s syslog via logger before executing the reboot command. Defaults to false.

Label provider

Requests reboot and terminate by labeling the Node. An external controller running on the Kubernetes control plane can watch those labels and perform the required action. After reboot completion, that controller MUST remove the reboot label. The node is considered ready once the reboot label is removed and the node’s boot ID has changed.

janitor-provider:
  csp:
    provider: "label"
    label:
      rebootKey: "nke.nvidia.com/reboot=requested-by-nvsentinel"
      terminateKey: "nke.nvidia.com/terminate=requested-by-nvsentinel"

rebootKey

Node label spec used to request a reboot, in key=value form so reboot and terminate can use different values. Defaults to nke.nvidia.com/reboot=requested-by-nvsentinel.

terminateKey

Node label spec used to request termination, in key=value form. Defaults to nke.nvidia.com/terminate=requested-by-nvsentinel.