Observability Stack Configuration#

The BCM cm-kubernetes-setup wizard deploys the foundational observability components in Kubernetes by default. Additional configuration enables time-series and logging data sources from BCM.

Configure storage and retention for Prometheus#

Create a new YAML file on the BCM head node for this setup:

cat << EOF > prom_config.yaml
prometheus:
  prometheusSpec:
    retention: 90d
    additionalArgs:
      - name: storage.tsdb.no-lockfile
        value: ""
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          storageClassName: local-path
          resources:
            requests:
              storage: 300Gi
EOF

Note

The preceding prom_config.yaml sets a 90-day retention period. The local-path StorageClass uses the shared storage pool selected during Kubernetes installation. Estimate your storage requirements before setting a longer retention period.

Install/upgrade CRDs:

helm pull oci://master.cm.cluster:5000/helm-charts/kube-prometheus-stack --untar \
    --ca-file /cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt
kubectl apply -f kube-prometheus-stack/charts/crds/crds/ --server-side

With the configuration file in place, apply these values to the K8s cluster:

helm upgrade kube-prometheus-stack ./kube-prometheus-stack \
    -n prometheus -f prom_config.yaml --reuse-values

Configure storage and retention for Loki#

This will require a redeployment of the Helm chart for Loki as we need to set a new storage value and we can’t resize this live.

Create a new YAML file on the BCM head node for this setup:

cat << EOF > loki_config.yaml
global:
  image:
    registry: master.cm.cluster:5000
memcached:
  image:
    repository: library/memcached
backend:
  replicas: 3
chunksCache:
  writebackSizeLimit: 500MB
commonConfig:
  replication_factor: 3
compactor:
  replicas: 0
deploymentMode: SimpleScalable
distributor:
  maxUnavailable: 0
  replicas: 0
indexGateway:
  maxUnavailable: 0
  replicas: 0
ingester:
  replicas: 0
loki:
  limits_config:
    ingestion_rate_mb: 75
    ingestion_burst_size_mb: 150
    per_stream_rate_limit: 16MB
    per_stream_rate_limit_burst: 64MB
    retention_period: 90d
  auth_enabled: false
  commonConfig:
    replication_factor: 3
  schemaConfig:
    configs:
    - from: '2025-09-04'
      index:
        period: 24h
        prefix: loki_index_
      object_store: s3
      schema: v13
      store: tsdb
  storage:
    type: s3
    bucketNames:
     chunks: loki-chunks-bucket
     ruler: loki-ruler-bucket
     admin: loki-admin-bucket
minio:
  enabled: true
  persistence:
    size: 500Gi
  image:
    repository: master.cm.cluster:5000/pgsty/minio
  mcImage:
    repository: master.cm.cluster:5000/pgsty/mc
querier:
  maxUnavailable: 0
  replicas: 0
queryFrontend:
  maxUnavailable: 0
  replicas: 0
queryScheduler:
  replicas: 0
read:
  replicas: 3
singleBinary:
  replicas: 0
write:
  replicas: 3
EOF

Uninstall the existing Helm chart:
```
helm uninstall loki -n loki
```
Delete any lingering PVCs:
```
kubectl delete pvc --all -n loki
```

Install the Helm chart for Loki, using the new values:

helm install loki oci://master.cm.cluster:5000/helm-charts/loki --ca-file \
    /cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt \
    -n loki -f loki_config.yaml

Configure Observability#

Select the Ubuntu release of the DGX software image:

Ubuntu 24.04

The datacenter-gpu-manager-exporter package is already installed. No package download or installation is required.

Ubuntu 26.04

On an internet-connected host, run the command for the target DGX node architecture, regardless of the download host architecture. Replace the example package version in the URL if your deployment requires a different version.

For x86_64 DGX nodes:

curl -fLO "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/datacenter-gpu-manager-exporter_4.8.4.3955-1_amd64.deb"

For aarch64 DGX nodes:

curl -fLO "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/sbsa/datacenter-gpu-manager-exporter_4.8.4.3955-1_arm64.deb"

Transfer the downloaded package alongside the bundle to the air-gapped environment. Keep this separately downloaded file outside bundle: nmc-airgap verify bundle checks only artifacts recorded in the bundle manifest. The download host does not need to run BCM or match the DGX node architecture.

On the BCM head node, copy the transferred package to /cm/images/<image name>/tmp/dcgm-exporter.deb. Then install it inside that image before configuring DCGM metric scraping:

cm-chroot /cm/images/<image name>
dpkg -i /tmp/dcgm-exporter.deb
exit

Enable the dcgm-exporter service in the DGX software image for either Ubuntu release:

cm-chroot-sw-img /cm/images/<image name> << EOF
  systemctl enable nvidia-dcgm-exporter.service
  exit
EOF

Follow the instructions in the following sections in Observability Stack Configuration:

Install NMC Grafana dashboards#

As part of NVIDIA Mission Control, a set of Grafana dashboards are provided that consolidate monitoring data across the cluster. These dashboards help track ongoing operations and support troubleshooting. This dashboard serves as a starting point for visualizing infrastructure telemetry. It is encouraged to modify or extend it based on the facility’s capabilities.

Prerequisites#

To take advantage of all dashboard features, install and configure the Infinity plugin and data sources to process data from the BCM REST API and UFM REST API.

Create a user to access the BCM REST API. Create user apiuser with a unique password of your choosing (do not reuse a shared or previously published value) with a read-only BCM profile:
```
cmsh -c "user; add apiuser; set password <your-bcm-api-password>; set profile readonly; commit"
```
Generate a Kubernetes secret for the registry CA certificate so Grafana can access the local registry to install plugins.
```
kubectl create secret generic registry-ca -n prometheus --from-file \
    ca.crt=/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt
```
Note

The Grafana plugin zips (including yesoreyeram-infinity-datasource) were already downloaded into bundle/files/ by the grafana-plugins component of the nmc-airgap fetch. The script in the next step scans that directory and pushes each plugin to the local registry. See Manifest and bundle structure for the bundle layout.

Save the Grafana plugin upload script to a file called push-plugins.sh in the current directory

#!/usr/bin/env bash
set -euo pipefail
PROG="$(basename "$0")"
die() { echo "$PROG: error: $*" >&2; exit 1; }
log() { echo "$PROG: $*" >&2; }
REGISTRY="master.cm.cluster:5000"
SCHEME="https"
CACERT="/cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt"
PLUGIN_PATTERNS=("grafana-metricsdrilldown-app" "grafana-lokiexplore-app" "yesoreyeram-infinity-datasource")
VERBOSE="${VERBOSE:-0}"
BUNDLE=""
while [[ $# -gt 0 ]]; do
    case "$1" in
        --bundle) BUNDLE="$2"; shift 2 ;;
        -v) VERBOSE=1; shift ;;
        -h|--help) echo "Usage: $PROG --bundle <dir> <repository>" >&2; exit 0 ;;
        --) shift; break ;;
        -*) die "Unknown option: $1" ;;
        *) break ;;
    esac
done
[[ $# -eq 1 ]]    || { echo "Usage: $PROG --bundle <dir> <repository>" >&2; exit 1; }
[[ -n "$BUNDLE" ]] || die "--bundle is required"
REPOSITORY="$1"
[[ -d "$BUNDLE" ]]       || die "Bundle directory not found: $BUNDLE"
[[ -d "$BUNDLE/files" ]] || die "No files/ subdirectory in bundle: $BUNDLE"
[[ -f "$CACERT" ]]       || die "CA certificate not found: $CACERT"
sha256_file() { command -v sha256sum &>/dev/null && sha256sum "$1" | awk '{print $1}' || shasum -a 256 "$1" | awk '{print $1}'; }
sha256_str()  { command -v sha256sum &>/dev/null && printf '%s' "$1" | sha256sum | awk '{print $1}' || printf '%s' "$1" | shasum -a 256 | awk '{print $1}'; }
file_size()   { stat -c%s "$1" 2>/dev/null || stat -f%z "$1"; }
TMPDIR_WORK="$(mktemp -d)"
trap 'rm -rf "$TMPDIR_WORK"' EXIT
_curl() { curl -s --cacert "$CACERT" -o "$TMPDIR_WORK/response_body" -D "$TMPDIR_WORK/response_headers" -w "%{http_code}" "$@" 2>/dev/null; }
_hdr()  { grep -i "^${1}:" "$TMPDIR_WORK/response_headers" 2>/dev/null | head -1 | sed 's/^[^:]*:\s*//' | tr -d '\r'; }
_api()  { local m="$1" u="$2"; shift 2; local c; c=$(_curl -X "$m" "$@" "$u") || c="000"; [[ "${VERBOSE:-0}" == "1" ]] && echo "$PROG: [debug] $m $u → $c" >&2; echo "$c"; }
blob_exists() { [[ "$(_api HEAD "${SCHEME}://${REGISTRY}/v2/${REPOSITORY}/blobs/$1")" == "200" ]]; }
push_blob_file() {
    local file="$1" digest="$2" size="$3" label="${4:-blob}"
    if blob_exists "$digest"; then log "$label already present, skipping"; return 0; fi
    log "Uploading $label (${size} bytes)..."
    local code loc
    code=$(_api POST "${SCHEME}://${REGISTRY}/v2/${REPOSITORY}/blobs/uploads/")
    [[ "$code" == "202" ]] || die "Failed to initiate blob upload (HTTP $code)"
    loc=$(_hdr "Location")
    [[ -n "$loc" ]] || die "No Location header in upload response"
    [[ "$loc" == http* ]] || loc="${SCHEME}://${REGISTRY}${loc}"
    local sep="?"; [[ "$loc" == *"?"* ]] && sep="&"
    code=$(_curl -X PUT -H "Content-Type: application/octet-stream" -H "Content-Length: ${size}" --data-binary "@${file}" "${loc}${sep}digest=${digest}") || code="000"
    [[ "$code" == "201" ]] || die "Blob upload failed (HTTP $code): $(cat "$TMPDIR_WORK/response_body" 2>/dev/null)"
    log "$label uploaded"
}
push_blob_str() {
    blob_exists "$2" && return 0
    printf '%s' "$1" > "$TMPDIR_WORK/blob_str"
    push_blob_file "$TMPDIR_WORK/blob_str" "$2" "$3" "config blob"
}
push_manifest() {
    printf '%s' "$1" > "$TMPDIR_WORK/manifest.json"
    local code
    code=$(_curl -X PUT -H "Content-Type: application/vnd.oci.image.manifest.v1+json" --data-binary "@$TMPDIR_WORK/manifest.json" "${SCHEME}://${REGISTRY}/v2/${REPOSITORY}/manifests/$2") || code="000"
    [[ "$code" == "201" ]] || die "Manifest push failed (HTTP $code): $(cat "$TMPDIR_WORK/response_body" 2>/dev/null)"
}
push_plugin() {
    local zipfile="$1" plugin_name="$2"
    local filename; filename="$(basename "$zipfile")"
    local tag="${filename%.zip}"; tag="${tag//+/-}"
    log "=== ${filename} ==="
    local layer_digest="sha256:$(sha256_file "$zipfile")"
    local layer_size; layer_size=$(file_size "$zipfile")
    local cfg='{}' cfg_digest cfg_size
    cfg_digest="sha256:$(sha256_str "$cfg")"
    cfg_size=${#cfg}
    local manifest
    manifest="{\"schemaVersion\":2,\"mediaType\":\"application/vnd.oci.image.manifest.v1+json\",\"artifactType\":\"application/vnd.oci.artifact.v1\",\"config\":{\"mediaType\":\"application/vnd.oci.empty.v1+json\",\"digest\":\"${cfg_digest}\",\"size\":${cfg_size}},\"layers\":[{\"mediaType\":\"application/zip\",\"digest\":\"${layer_digest}\",\"size\":${layer_size},\"annotations\":{\"org.opencontainers.image.title\":\"${filename}\"}}]}"
    push_blob_str "$cfg" "$cfg_digest" "$cfg_size"
    push_blob_file "$zipfile" "$layer_digest" "$layer_size" "$filename"
    push_manifest "$manifest" "$tag"
    echo "${SCHEME}://${REGISTRY}/v2/${REPOSITORY}/blobs/${layer_digest};${plugin_name}"
}
command -v curl &>/dev/null || die "curl is required"
log "Scanning ${BUNDLE}/files/ for matching plugins..."
urls=()
while IFS= read -r -d '' zipfile; do
    name="$(basename "$zipfile")"
    matched=""
    for p in "${PLUGIN_PATTERNS[@]}"; do [[ "$name" == *"${p}"* ]] && matched="$p" && break; done
    [[ -n "$matched" ]] && urls+=("$(push_plugin "$zipfile" "$matched")")
done < <(find "$BUNDLE/files" -maxdepth 1 -name "*.zip" -print0 | sort -z)
[[ ${#urls[@]} -gt 0 ]] || die "No matching plugin zips found in ${BUNDLE}/files/"
log "Done. Pushed ${#urls[@]} plugin(s)."
GRAFANA_IMAGE="$(kubectl get deploy kube-prometheus-stack-grafana -n prometheus \
    -o jsonpath='{range .spec.template.spec.containers[?(@.name=="grafana")]}{.image}{end}' 2>/dev/null || true)"
[[ -n "$GRAFANA_IMAGE" ]] || die "Could not determine Grafana image; is kube-prometheus-stack-grafana deployed in namespace prometheus?"
{
  echo "grafana:"
  echo "  extraInitContainers:"
  echo "    - name: install-airgap-plugins"
  echo "      image: ${GRAFANA_IMAGE}"
  echo '      command: ["/bin/sh", "-c"]'
  echo "      args:"
  echo "        - |"
  echo "            set -e"
  echo "            PDIR=/var/lib/grafana/plugins"
  echo "            install() {"
  echo '              if [ -d "$PDIR/$1" ]; then echo "$1 already present, skipping"; return; fi'
  echo '              echo "Installing $1 from $2"'
  echo '              grafana cli --insecure --pluginUrl "$2" --pluginsDir "$PDIR" plugins install "$1"'
  echo "            }"
  for entry in "${urls[@]}"; do
    echo "            install ${entry##*;} ${entry%;*}"
  done
  echo "      volumeMounts:"
  echo "        - name: storage"
  echo "          mountPath: /var/lib/grafana"
} > grafana_plugins.yaml

Run the plugin upload script

Note

These steps assume that the bundle is present in the current directory. If it is elsewhere, update the --bundle argument to the correct path. The script also requires kubectl access to the cluster, which it uses to detect the deployed Grafana image for the init container.
```
chmod +x push-plugins.sh && ./push-plugins.sh --bundle ./bundle grafana-plugins
```

The script pushes each plugin to the local registry and generates a file called grafana_plugins.yaml. This file defines a Grafana init container that installs the plugins from the local registry into Grafana’s plugin directory (/var/lib/grafana/plugins) at startup, so Grafana loads them without contacting grafana.com. It is applied in a later step. If successful, the script’s output should look like the following:

push-plugins.sh: Scanning ./bundle/files/ for matching plugins...
push-plugins.sh: === grafana-lokiexplore-app-1.0.40.zip ===
push-plugins.sh: Uploading config blob (2 bytes)...
push-plugins.sh: config blob uploaded
push-plugins.sh: Uploading grafana-lokiexplore-app-1.0.40.zip (10480908 bytes)...
push-plugins.sh: grafana-lokiexplore-app-1.0.40.zip uploaded
push-plugins.sh: === grafana-metricsdrilldown-app-1.0.34.zip ===
push-plugins.sh: Uploading config blob (2 bytes)...
push-plugins.sh: config blob uploaded
push-plugins.sh: Uploading grafana-metricsdrilldown-app-1.0.34.zip (4146320 bytes)...
push-plugins.sh: grafana-metricsdrilldown-app-1.0.34.zip uploaded
push-plugins.sh: === yesoreyeram-infinity-datasource-3.7.4.zip ===
push-plugins.sh: Uploading config blob (2 bytes)...
push-plugins.sh: config blob uploaded
push-plugins.sh: Uploading yesoreyeram-infinity-datasource-3.7.4.zip (81446703 bytes)...
push-plugins.sh: yesoreyeram-infinity-datasource-3.7.4.zip uploaded
push-plugins.sh: Done. Pushed 3 plugin(s).

Create a new file with the following values, substituting the BCM password you set above and your UFM REST API credentials. Keep each Infinity datasource’s allowed host scoped to its own API path:

cat << EOF > grafana_infinity.yaml
grafana:
  grafana.ini:
    plugins:
      preinstall_disabled: true
      public_key_retrieval_disabled: true
    analytics:
      check_for_updates: false
      check_for_plugin_updates: false
  enabled: true
  extraSecretMounts:
    - name: registry-ca
      mountPath: /etc/ssl/certs/ca-cert-registry.crt
      secretName: registry-ca
      readOnly: true
      optional: false
      subPath: ca.crt
  additionalDataSources:
    - name: BCM API
      type: yesoreyeram-infinity-datasource
      access: proxy
      isDefault: false
      basicAuth: true
      basicAuthUser: apiuser
      jsonData:
        auth_method: 'basicAuth'
        allowedHosts:
          - https://master:8081/rest/v1/
        tlsSkipVerify: true
        timeoutInSeconds: 60
      secureJsonData:
        basicAuthPassword: <your-bcm-api-password>
    - name: UFM API
      type: yesoreyeram-infinity-datasource
      access: proxy
      isDefault: false
      basicAuth: true
      basicAuthUser: <your-ufm-api-user>
      jsonData:
        auth_method: 'basicAuth'
        allowedHosts:
          - https://master:11443/ufmRest/
        tlsSkipVerify: true
        timeoutInSeconds: 60
      secureJsonData:
        basicAuthPassword: <your-ufm-api-password>
    - name: Loki
      type: loki
      uid: loki
      url: http://loki-gateway.loki.svc.cluster.local:80
      access: proxy
      isDefault: false
EOF

Run the following command to apply the values:

helm upgrade kube-prometheus-stack ./kube-prometheus-stack -n prometheus \
  --reuse-values -f grafana_infinity.yaml -f grafana_plugins.yaml

To access the UFM REST API, we need to install a port-forwarding rule within BCM as well. To configure this, you need to gather the IP address of the UFM appliance and replace it in the following command sequence:
```
cmsh -c "device; portforward create UFM_IP_ADDRESS 443 11443"
```
This should be repeated on your secondary BCM head node, if you have one configured.

Note

The port-forwarding rules are not persisted on restarts of the BCM daemon process. If your BCM head node reboots or the daemon restarts for any reason, you will need to rerun this port-forwarding sequence.

Installation#

Note

Run these commands from the directory that contains the uploaded dashboard chart.

Upload the dashboard chart (.tgz file) to the BCM head node.

Enable dashboard folders in Grafana:

helm upgrade -n prometheus kube-prometheus-stack ./kube-prometheus-stack \
  --set grafana.sidecar.dashboards.provider.foldersFromFilesStructure=true \
  --set grafana.sidecar.dashboards.enabled=true --reuse-values

DGX B200