Manage Secrets

NeMo Platform provides an interface to manage user-defined secrets. This allows you to securely store API keys to integrate with other providers. For example, you might want to store a Hugging Face API token as a secret to access models, or a Weights & Biases API key for experiment tracking.

Concepts

A secret is a named resource that holds sensitive information, such as API keys or tokens.

• Secrets are unique within a workspace, meaning they exist within a specific workspace in NeMo Platform. Operations which utilize secrets must specify the workspace in which the secret resides.

• All secrets are encrypted at rest. Once a secret is created, its value cannot be retrieved through the API—not even by workspace Admins. Users can view secret metadata (name, description, timestamps) but can only update or delete the secret, never read its value back.

• Only Platform Administrators can retrieve secret values. This restricted access ensures that sensitive credentials remain protected even from users with full workspace administrative privileges.

• Platform services access secrets internally as needed to facilitate user-requested operations, for example, when a job requires access to a Hugging Face model, or when reporting metrics to Weights & Biases.

• Secrets referenced by other resources (for example, customization configs) are referenced by their name and workspace.

  • A reference to a secret by another resource is stored in the format workspace/secret_name.

Access Control

When authentication is enabled, secrets follow role-based access control:

Role	List	View Metadata	Create	Update	Delete	Read Value
Viewer	✓	✓	✗	✗	✗	✗
Editor	✓	✓	✓	✓	✓	✗
Admin	✓	✓	✓	✓	✓	✗
Platform Admin	✓	✓	✓	✓	✓	✓

Important

Even workspace Admins cannot read secret values. This is by design—secrets contain sensitive credentials that should not be exposed through the API. Only Platform Administrators have the elevated permissions required to retrieve secret values when absolutely necessary.

Naming Requirements

Secret names must follow these rules:

• Allowed characters: letters (a-z, A-Z), digits (0-9), hyphens (-), underscores (_), and dots (.)

• Maximum length: 255 characters

• Must contain at least one character (cannot be empty)

• Pattern: ^[\w\-.]+$ (word characters, dots, and hyphens)

Tip

Best practice: Use lowercase letters, numbers, and hyphens only (for example, my-api-key). This ensures compatibility with Kubernetes naming conventions if your secrets are used in job environments.

Example valid names: hf-token, nim-api-key, wandb.key.production, my_secret_1

Understanding Workspaces

In NeMo Platform, a workspace is used to organize and isolate resources. Workspaces function similarly to namespaces - they provide logical separation between different sets of resources.

• The default workspace is named default

• Most API operations require specifying a workspace parameter

• Secrets, model providers, jobs, and other resources exist within a workspace

• Resources in one workspace cannot directly access resources in another workspace

Tip

If you are just getting started, use the default workspace. You can create additional workspaces later to organize resources by team, project, or environment.

Operations

To manage secrets, you can use the NeMo Platform SDK. Below are examples of how to create, list, update, and delete secrets.

To work with secrets, you can use the following import and initialization:

import os
from nemo_platform import NeMoPlatform

sdk = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)

Creating Secrets

Creating a secret involves specifying a name, workspace, and the initial secret data.

secret = sdk.secrets.create(
    name="my-secret",
    workspace="default",
    data="<your-secret-data>"
)

Listing Secrets

You can list all secrets within a specific workspace. This returns metadata only.

# List all secrets in the "default" workspace
secrets = sdk.secrets.list(workspace="default")
for secret in secrets.data:
    print(secret.name)

Retrieving a Secret

To retrieve a single secret by name, use the retrieve method. This returns the secret’s metadata only.

# Get a specific secret's metadata
secret = sdk.secrets.retrieve(
    name="my-secret",
    workspace="default"
)
print(f"Secret: {secret.name}, Created: {secret.created_at}")

Updating Secrets

To update a secret, you can change its data field.

# Update the content of the secret
updated_secret = sdk.secrets.update(
    name="my-secret",
    workspace="default",
    data="<your-new-secret-data>"
)

Deleting Secrets

Deleting a secret will remove the secret from the platform.

# Delete a secret
sdk.secrets.delete(
    name="my-secret",
    workspace="default"
)

Use Cases

Hugging Face API Tokens

A Hugging Face API token secret allows NeMo Platform to authenticate with the Hugging Face Hub for accessing models and repositories.

How are Hugging Face API Tokens Used?

NeMo Platform uses Hugging Face api tokens to authenticate requests to the Hugging Face Hub. This enables users to download private models and upload customized models to their Hugging Face repositories.

Creating a Hugging Face Secret

To create a Hugging Face API key secret, use the NeMo Platform SDK to create a new secret with your Hugging Face token.

import os
from nemo_platform import NeMoPlatform

sdk = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)

hf_secret = sdk.secrets.create(
    name="my-hugging-face-token",
    workspace="default",
    data="<your-hugging-face-token>"
)

Weights & Biases Keys

NeMo Platform users can integrate with Weights & Biases by providing their API key as a secret. This allows the NeMo Customizer service to report training metrics to the user’s W&B account.

How are Weights & Biases Keys Used?

When a Weights & Biases key is provided, the NeMo Customizer service sends telemetry data to W&B including:

• Job ID

• Training loss

• Validation loss

• Other relevant metrics

All metrics are stored under the user’s nvidia-nemo-customizer project in their W&B account.

Creating a Weights & Biases Secret

To create a W&B secret, use the NeMo Platform SDK to create a new secret with your W&B API key.

import os
from nemo_platform import NeMoPlatform

sdk = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)

wandb_secret = sdk.secrets.create(
    name="my-wandb-key",
    workspace="default",
    data="<your-wandb-api-key>"
)

Encryption Configuration

Important

For production deployments, it is critical to configure secure encryption for secrets at rest. The default local encryption provider is intended for initial deployment or evaluation.

Rotating or Migrating Encryption Keys

NeMo Platform supports rotation of encryption keys used to secure secrets at rest. Key rotation can be performed by Platform Administrators to comply with security policies or to migrate between encryption providers.

To rotate or migrate encryption keys:

1. Add a new provider configuration to the platform config’s secrets encryption settings. You can name the new provider as desired (for example, v2) and set it as the current_provider.

   secrets:
     encryption:
       current_provider: new_provider_name
       providers:
         secret_key:
           # Keep the existing provider for decryption of old secrets
           v1:
             value: "<old-encryption-key>"
           # Add the new provider for encrypting new secrets
           v2:
             value: "<new-encryption-key>"
             # from_env: "NEW_ENCRYPTION_KEY_ENV_VAR" # Optional alternative to load from environment variable
   

2. Re-deploy the platform services to apply the new configuration.

3. As the platform administrator role, call the /v2/secrets/rotate-encryption-keys endpoint to initiate the rotation process. This will re-encrypt all existing secrets using the new provider. You can also initiate the rotation using the NeMo Platform SDK:

   import os
   from nemo_platform import NeMoPlatform
   
   sdk = NeMoPlatform(
       base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
       workspace="default",
   )
   
   # Initiate the key rotation process
   sdk.secrets.admin.rotate_encryption_keys()
   

   This operation can take some time depending on the number of secrets stored. During this process, both the old and new encryption providers must be available to ensure seamless access to secrets. Do not remove the old provider until rotation is confirmed completed.

   If any errors occur during rotation, they will be logged for review. You can retry the rotation operation as needed until all secrets are successfully re-encrypted.

4. To validate the rotation completed successfully, you can check the logs for the following confirmation message:

   Completed rotation of encryption keys for all secrets
   

5. After rotation is confirmed, you can optionally remove the old provider configuration from the platform settings to prevent its further use.