NEK Rotation and Re-encryption#

Overview#

A Namespace Encryption Key (NEK) is an AES-256-GCM key that directly encrypts user secrets in the Encrypted Secret Store (ESS). Each NVCF namespace has its own NEK. The NEK is stored in Cassandra in encrypted form, wrapped by the Master Encryption Key (MEK) from OpenBAO.

Rotate the NEK on a regular schedule (for example, every 90 days) or when required by your security policy. After a MEK rotation, you must also re-encrypt all NEKs so they are wrapped with the new MEK.

For MEK rotation, see MEK (Master Encryption Key) Rotation.

When to Rotate vs. Re-encrypt#

Operation	When
NEK Rotation	Periodically (e.g. every 90 days) to limit the impact of a potential key compromise. A new NEK is generated and all future secrets are encrypted with it.
NEK Re-encryption	After a MEK rotation. Existing NEKs in Cassandra must be re-wrapped with the new MEK.

Prerequisites#

kubectl configured for your NVCF control plane cluster
Access to a Cassandra pod in the cassandra-system namespace (or direct cqlsh connectivity)
Cassandra credentials (stored in the cassandra secret in cassandra-system)
Access to OpenBAO in the vault-system namespace to read the MEK
A maintenance window is recommended; rotation can briefly affect ESS availability if pods are restarted

Where Keys are Stored#

NEKs in Cassandra:

The ESS keyspace in Cassandra (ess_api) contains three NEK-related tables:

Table	Purpose
`encryption_keys_by_kid_and_encrypted_at`	Primary NEK table. Stores NEKs keyed by namespace, key ID, and encryption timestamp. Includes a `status` column and a `current_kid` static column per namespace.
`encryption_keys_by_kid`	Lookup table keyed by namespace and key ID.
`encryption_keys_by_timestamp`	Lookup table keyed by namespace and creation timestamp (newest first).

MEK in OpenBAO:

The MEK is stored in the services/all/kv/ KV v2 mount in OpenBAO at path encryption/keys/stored_data. See MEK (Master Encryption Key) Rotation for details.

Inspecting Current State#

Use the following commands to view the current NEK and MEK state before making changes.

Retrieve Cassandra credentials:

kubectl get secret -n cassandra-system cassandra \
  -o jsonpath='{.data.cassandra-password}' | base64 -d

List current NEKs:

kubectl exec -n cassandra-system cassandra-0 -- \
  cqlsh -u cassandra -p <CASSANDRA_PASSWORD> \
  -e "SELECT namespace, kid, status, encrypted_by_kid, encrypted_at
      FROM ess_api.encryption_keys_by_kid_and_encrypted_at;"

Read the MEK’s current key ID (to verify which MEK is wrapping the NEKs):

VAULT_TOKEN=$(kubectl get secret -n vault-system openbao-server-root-token \
  -o jsonpath='{.data.root_token}' | base64 -d)

kubectl exec -n vault-system openbao-server-0 -c openbao -- \
  sh -c "VAULT_TOKEN=$VAULT_TOKEN bao kv get \
    -field=current_kid services/all/kv/encryption/keys/stored_data"

The encrypted_by_kid column in Cassandra should match the current_kid from OpenBAO, confirming the NEKs are wrapped by the active MEK.

NEK Rotation Procedure#

ESS generates and rotates NEKs internally. When ESS starts, it checks whether a valid NEK exists for each namespace and creates one if needed. To trigger a NEK rotation:

Verify the current NEK state using the inspection commands above.

Restart the ESS deployment to trigger NEK lifecycle processing:

kubectl rollout restart deployment -n ess ess-api-helm-nvcf-ess-api-deployment

Wait for the rollout to complete:

kubectl rollout status deployment -n ess ess-api-helm-nvcf-ess-api-deployment

Verify that a new NEK was created:

kubectl exec -n cassandra-system cassandra-0 -- \
  cqlsh -u cassandra -p <CASSANDRA_PASSWORD> \
  -e "SELECT namespace, kid, status, encrypted_at
      FROM ess_api.encryption_keys_by_kid_and_encrypted_at;"

Confirm a new row with status = 'CREATION_VALIDATED' exists.

Check ESS logs for any errors:

kubectl logs -n ess -l app.kubernetes.io/name=helm-nvcf-ess-api \
  -c helm-nvcf-ess-api --tail=100 | grep -i -E "key|encrypt|error"

NEK Re-encryption (After MEK Rotation)#

After rotating the MEK (see MEK (Master Encryption Key) Rotation), all NEKs in Cassandra are still wrapped with the old MEK. You must re-encrypt them with the new MEK.

Complete the MEK rotation first. Verify the new MEK is active in OpenBAO.

Restart ESS so it picks up the new MEK from OpenBAO via its vault-agent sidecar:

kubectl rollout restart deployment -n ess ess-api-helm-nvcf-ess-api-deployment
kubectl rollout status deployment -n ess ess-api-helm-nvcf-ess-api-deployment

Verify re-encryption by checking that the encrypted_by_kid now matches the new MEK’s current_kid:

kubectl exec -n cassandra-system cassandra-0 -- \
  cqlsh -u cassandra -p <CASSANDRA_PASSWORD> \
  -e "SELECT namespace, kid, encrypted_by_kid, encrypted_at
      FROM ess_api.encryption_keys_by_kid_and_encrypted_at;"

Test secret access by reading an existing secret through the NVCF API to confirm decryption still works.

Verification#

After any rotation or re-encryption:

All ESS pods are Running and Ready:
```
kubectl get pods -n ess
```

No decryption or key-loading errors in ESS logs:

kubectl logs -n ess -l app.kubernetes.io/name=helm-nvcf-ess-api \
  -c helm-nvcf-ess-api --tail=200 | grep -i error

Secrets can still be read and written through the NVCF API.

Rollback#

If rotation or re-encryption fails:

Check ESS pod logs for specific error messages.
If the new NEK is not yet promoted (status is not CREATION_VALIDATED), restarting ESS will cause it to re-attempt key creation.
If re-encryption fails, the old MEK must still be present in the OpenBAO keys array (it should not be removed until re-encryption is verified). Restore the old MEK as the current_kid if needed and restart ESS.
Do not delete old keys from either Cassandra or OpenBAO until the new keys are fully verified.