Observability Software#

Base Command Manager Monitoring#

NVIDIA Mission Control utilizes NVIDIA Base Command Manager’s built-in monitoring infrastructure for comprehensive system observability. This guide assumes administrators are familiar with BCM monitoring concepts, including metrics collection, measurable definitions, trigger and alert configurations, and health checks.

For detailed coverage of observability and monitoring in Base Command Manager, administrators should refer directly to:

Base Command Manager (BCM) Administrator Guide:

  • Monitoring Cluster Devices (Section 10)

  • Metrics, Health Checks, Enummetrics, and Actions (Appendix G)

For customizing monitoring alerts, health checks, and related configurations, consult these sections before proceeding.

Prometheus and Grafana#

Validate metric collection using Grafana#

Grafana should be available on your head node listening on HTTPS on the /grafana page.

By default, the administrative account is admin with password prom-operator.

Grafana login screen

Once authenticated, click on Explore on the left. Once in the Explore interface, set the query editor to code and query the metric memorytotal.

Grafana explore interface showing memorytotal query

BCM provides many metrics out of the box, you can find a list of these in Grafana by using the Metric browser selecting all metrics which include the label job with the value external-bcmexporter.

Grafana metric browser showing BCM metrics

This can also be found by inspecting the output of the BCM Prometheus exporter endpoint. Here using curl with some light processing.

curl -sk https://localhost:8081/exporter | grep -Ev '# HELP|# TYPE' | cut -d'{' -f1 | sort -u

alertlevel
...
writetime_total

Build a Grafana dashboard#

With BCM exporting metrics, you can create dashboards. In this example, you’ll build a device state history dashboard. To build this dashboard, follow these steps:

  1. Select Dashboards on the left panel, select New and select New dashboard.

    Creating a new Grafana dashboard

  2. Click Add visualization.

    Adding a visualization to the dashboard

  3. Select Prometheus as your datasource.

    Selecting Prometheus as the datasource

  4. Start by setting the visualization to Status history. Query the metric devicestatus.

    Setting up status history visualization with devicestatus metric

  5. On the right pane, scroll down to Value mappings and click on Add value mappings. This metric returns either 0 which maps to OK or 1 which maps to NOK. Add value conditions for these with a desired color. In this step, you’ll set 0 as green and 1 as red.

    Configuring value mappings for device status

  6. Delete the threshold for the value 80.

    Deleting the default threshold

  7. Expand options in the query window and set the Legend to Custom and set the value to {{hostname}}.

    Setting custom legend with hostname

  8. You may get this error:

    Too many points to visualize properly.

  9. You can work around this by setting the time window of the dashboard from Last 6 hours to Last 15 minutes.

    Adjusting the time window to resolve visualization issues

  10. To allow for you to use this over a longer period of time, set the Min step for the query to 5m. Set the Title of the panel to Node status.

    Setting minimum step and panel title

    Click on Back to dashboard. Resize the panel to a desired size.

    Returning to dashboard view

    Set the window of time to the Last 1 hour, take note that the dashboard rendered more status boxes.

    Resizing the panel and adjusting time window

  11. Select Save dashboard. Set the Title to BCM node status and click on Save.

    Saving the dashboard with a title

Configure email contact points for Grafana alerts#

Grafana defines an email contact point for alerts by default. Edit this contact point so that alerts reach the administrator’s email address.

The default email contact point in the Grafana contact points list

To change the default email address, follow these steps:

  1. Get the current kube-prometheus-stack Helm chart values.

    helm get values -n prometheus kube-prometheus-stack > /tmp/prometheus-kube-prometheus-stack-values.yaml

  2. Add an smtp section to the grafana.ini values in the file. Replace the host, from_address, and from_name values with the details of your SMTP relay.

    grafana.ini:
  analytics:
    check_for_updates: true
  log:
    mode: console
  paths:
    data: /var/lib/grafana/
    logs: /var/log/grafana
    plugins: /var/lib/grafana/plugins
    provisioning: /etc/grafana/provisioning
  server:
    domain: '{{ if (and .Values.ingress.enabled .Values.ingress.hosts) }}{{ tpl
      (.Values.ingress.hosts | first) . }}{{ else if (and .Values.route.main.enabled
      .Values.route.main.hostnames) }}{{ tpl (.Values.route.main.hostnames | first)
      . }}{{ else }}''''{{ end }}'
    root_url: '%(protocol)s://%(domain)s:%(http_port)s/grafana/'
    serve_from_sub_path: true
  smtp:
    enabled: true
    from_address: admin@nmc-obs.cm.cluster
    from_name: nmc-obs
    host: 10.141.255.254:25
    skip_verify: true
  unified_storage:
    index_path: /var/lib/grafana-search/bleve

  3. Apply the updated values file to upgrade kube-prometheus-stack.

    helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack -n prometheus -f /tmp/prometheus-kube-prometheus-stack-values.yaml --version 87.18.0 --wait --timeout 5m

  4. In Grafana, click Alerting on the left panel.

  5. Click Notification configuration under Alerting.

  6. Click the Contact points tab.

  7. Click Edit for the grafana-default-email contact point.

    The Edit button for the grafana-default-email contact point

  8. Enter the administrator’s email address in the Addresses field.

    Administrator email address entered in the Addresses field

  9. Click Save contact point to save the address before you test it.

  10. Click Test.

    The Test button on the email contact point

  11. Click Send test notification.

    The Send test notification button in the test window

  12. Confirm that a Test notification sent successfully message appears and that the test email arrives.

    Test notification sent successfully message in Grafana The test email as received in the administrator mailbox

  13. Click X in the top right corner to close the window.

  14. Click Save contact point.

    The Save contact point button on the email contact point

Configure Slack contact points for Grafana alerts#

To send alerts to a Slack channel, add a Slack contact point. Follow these steps:

  1. In Grafana, click Alerting on the left panel.

  2. Click Notification configuration under Alerting.

  3. Click the Contact points tab.

  4. Click New contact point.

    The New contact point button in the Grafana contact points list

  5. Enter a name for the contact point in the Name field.

  6. Select Slack from the Integration drop-down menu.

  7. Enter the Slack channel in the Recipient field.

  8. Enter the token for the Slack app.

    Slack integration fields for a new contact point

  9. Click Test to open the test window.

    The test window for the Slack contact point

  10. Click Send test notification and confirm that a Test notification sent successfully message appears in the same window.

    Test notification sent successfully message for the Slack contact point

  11. Click X in the top right corner to close the window.

  12. Click Save contact point.

    The Save contact point button on the Slack contact point

Note

For instructions on creating a Slack app, refer to Quickstart app settings in the Slack documentation.

Configure Grafana managed alerts#

After you configure contact points, configure the alert rules that send notifications to them. To configure an alert rule, follow these steps:

  1. Open the Grafana panel you want to alert on and click Edit. This example uses the Storage >= 80% usage panel.

    The Storage >= 80% usage panel in Grafana

  2. Click the Alert tab, then click New alert rule.

    The New alert rule button on the Alert tab

  3. Enter or adjust the query that fires the alert, and set its condition.

    Query and condition settings for a new alert rule

  4. Select an existing folder and evaluation group for the alert, or create new ones.

    Folder and evaluation group settings for a new alert rule

  5. Select the contact point for the alert, add a custom notification message if you need one, then click Save.

    Contact point and notification message settings for a new alert rule

The following image shows an example Slack notification sent after the alert rule fires.

Example Slack notification produced by a Grafana alert rule

Logs#

With BCM, in-built logging aggregation comes out of the box. Each node that BCM deploys by default forwards logs to the BCM head nodes for centralization of logs. From a BCM head node, you can view these logs in /var/log/syslog.

Counting Xid errors#

When a GPU fault has occurred, these are logged to syslog. These can indicate user driven exceptions, errors with the driver, or in some circumstances a hardware fault. It’s good to understand the Xids encountered and the counts of these.

Xid values and descriptions can be found in this document: Xid errors

grep "Xid (PCI" /var/log/syslog | awk -v cutoff="$(date -d '12 hours ago' '+%b %d %H:%M:%S')" '
  $1 " " $2 " " $3 >= cutoff {
    split($4, host, ".");
    if (match($0, /Xid \(PCI:[^)]+\): ([0-9]+)/, id)) {
      print host[1], id[1]
    }
  }
' | sort | uniq -c | awk '{print $2 ":", $1, "occurrences of error ID", $3}'

Example output

a07-p1-dgx-03-c08: 10158 occurrences of error ID 143
b06-p1-dgx-06-c01: 4 occurrences of error ID 149
b06-p1-dgx-06-c01: 4 occurrences of error ID 154
b06-p1-dgx-06-c02: 4 occurrences of error ID 149
b06-p1-dgx-06-c02: 3 occurrences of error ID 154
b06-p1-dgx-06-c04: 4 occurrences of error ID 149
b06-p1-dgx-06-c04: 4 occurrences of error ID 154
b06-p1-dgx-06-c05: 4 occurrences of error ID 149
b06-p1-dgx-06-c05: 4 occurrences of error ID 154
b06-p1-dgx-06-c06: 4 occurrences of error ID 149
b06-p1-dgx-06-c06: 4 occurrences of error ID 154
b06-p1-dgx-06-c07: 4 occurrences of error ID 149
b06-p1-dgx-06-c07: 2 occurrences of error ID 154
b06-p1-dgx-06-c08: 4 occurrences of error ID 149
b06-p1-dgx-06-c08: 4 occurrences of error ID 154
b06-p1-dgx-06-c09: 4 occurrences of error ID 149
b06-p1-dgx-06-c09: 4 occurrences of error ID 154
b06-p1-dgx-06-c11: 4 occurrences of error ID 149
b06-p1-dgx-06-c11: 4 occurrences of error ID 154
b06-p1-dgx-06-c12: 4 occurrences of error ID 149
b06-p1-dgx-06-c12: 3 occurrences of error ID 154
b06-p1-dgx-06-c13: 4 occurrences of error ID 149
b06-p1-dgx-06-c13: 4 occurrences of error ID 154
b06-p1-dgx-06-c13: 112 occurrences of error ID 45
b06-p1-dgx-06-c14: 4 occurrences of error ID 149
b06-p1-dgx-06-c14: 4 occurrences of error ID 154
b06-p1-dgx-06-c14: 112 occurrences of error ID 45
b06-p1-dgx-06-c15: 4 occurrences of error ID 149
b06-p1-dgx-06-c15: 4 occurrences of error ID 154
b06-p1-dgx-06-c15: 112 occurrences of error ID 45
b06-p1-dgx-06-c16: 4 occurrences of error ID 149
b06-p1-dgx-06-c16: 4 occurrences of error ID 154
b06-p1-dgx-06-c16: 112 occurrences of error ID 45
b06-p1-dgx-06-c17: 4 occurrences of error ID 149
b06-p1-dgx-06-c17: 4 occurrences of error ID 154
b06-p1-dgx-06-c17: 112 occurrences of error ID 45
b06-p1-dgx-06-c18: 4 occurrences of error ID 149
b06-p1-dgx-06-c18: 4 occurrences of error ID 154