NeMo Platform Helm Chart

View as Markdown

Type: application

For deployment guide, see Self-Managed Deployment in the NeMo Platform documentation.

Platform Secrets Encryption Key

The platform secrets service reads NMP_SECRETS_DEFAULT_ENCRYPTION_KEY from the API env Secret. The value must be base64-encoded and decode to at least 32 bytes.

Set secrets.defaultEncryptionKey.value to provide your own key. When that value is empty and neither envFromSecret nor secrets.defaultEncryptionKey.existingSecret.name is set, the chart runs a pre-install hook that creates <fullname>-api-env with a per-install random key. The hook is install-only and refuses to patch or rotate an existing Secret.

Set secrets.defaultEncryptionKey.existingSecret.name to use an existing Secret for only the secrets service encryption key. After Kubernetes decodes the Secret data, the value loaded from secrets.defaultEncryptionKey.existingSecret.key must be a base64-encoded key that decodes to at least 32 bytes.

Set envFromSecret to use a fully user-managed API env Secret. In that mode the chart does not create or generate the API env Secret.

On upgrade, the generated Secret must already exist and contain NMP_SECRETS_DEFAULT_ENCRYPTION_KEY. If it is missing, restore the original Secret instead of generating a replacement key; existing encrypted platform secrets will not decrypt with a new key.

Intake and ClickHouse

Intake is included in the platform API service group. By default, the chart deploys a single-node embedded ClickHouse 26.3 LTS service, and Intake creates and migrates its own intake database on first use.

The embedded ClickHouse is intended for development, evaluation, and non-critical single-node installations. It does not provide replication or automatic backups. For a production deployment that requires high availability, set clickhouse.enabled to false and provide a separately managed ClickHouse:

First, create the credentials Secret in the Helm release namespace. The Secret key must match externalClickhouse.existingSecretPasswordKey:

$kubectl create secret generic clickhouse-credentials \
> --namespace <release-namespace> \
> --from-literal=password='<clickhouse-password>'

Then configure the external connection:

1clickhouse:
2 enabled: false
3
4externalClickhouse:
5 host: clickhouse.example.internal
6 port: 8443
7 secure: true
8 user: nemo
9 database: intake
10 existingSecret: clickhouse-credentials
11 existingSecretPasswordKey: password

The external user must be allowed to create the configured database, tables, materialized views, and indexes because Intake owns its ClickHouse migrations.

ClickHouse sizing

Use retained span count as an initial operational threshold, not as a disk-size estimate. Span input, output, and attribute payloads vary substantially. Measure bytes_on_disk from representative traffic before setting production storage:

1SELECT
2 table,
3 sum(rows) AS physical_rows,
4 formatReadableSize(sum(bytes_on_disk)) AS disk
5FROM system.parts
6WHERE active AND database = 'intake'
7GROUP BY table
8ORDER BY table;

The following are starting points for the current Intake schema and interactive query workload:

Retained Intake spansTopologyClickHouse resourcesStorage
Up to 1 millionEmbedded single node for non-critical workloads2 vCPU, 8 GiB RAMFast SSD, at least 20 GiB and 2× measured active data
1–10 millionExternal preferred; embedded only when downtime and data loss are acceptable4–8 vCPU, 16–32 GiB RAMProvisioned-IOPS SSD, at least 100 GiB and 2× measured active data
More than 10 million, or any HA requirementManaged ClickHouse or an operator-managed replicated clusterStart at 8 vCPU and 32 GiB RAM per replica, then load-test the actual ingest/read mixSize from measured compression, retention, replication, and merge headroom

Intake currently retains spans and its trace index for 90 days. Evaluator results and annotations do not have a time-based TTL, so include their continuing growth in capacity planning. ReplacingMergeTree retries also leave physical row versions until background merges complete.

For large or frequently queried deployments, ClickHouse recommends:

  • At least 8 GiB RAM even at low data volumes.
  • A general-purpose starting ratio of 4 GiB RAM per CPU core.
  • Provisioned-IOPS SSDs for latency-sensitive workloads.
  • Roughly 1:30 to 1:50 RAM-to-storage for frequently accessed large datasets.
  • Replication for production durability; vertically scale replicas before adding shards.

Validate CPU, query peak memory, active parts, merge backlog, disk latency, and free space under representative batched OTLP ingestion before promoting a tier. See the upstream ClickHouse sizing guide and OSS operational recommendations.

Values

The following is the complete values.yaml for the NeMo Platform Helm Chart. All configuration options are documented inline with comments.