Monitoring Data Producers | NVIDIA Base Command Manager

This chapter covers how to add a new metrics and health checks scripts with cmsh.

Five different types of Monitoring Data Producers can be added:

metric: a script which produces a single value.
health check: a script which produces a PASS, FAIL, UNKNOWN, or no data value.
collection: a script that produces zero or more metrics, health checks, or a combination of both.
perpetual a script that is started once over the lifetime of the BCM cmd process. The script produces zero or more metrics, health checks, or a combination of both on its own timing mechanism.
prometheus one or more URLs to Prometheus metric exporters.

A monitoring data producer cannot be plotted in cmsh or Base View, because it contains no data. A producer defines measurables: metrics and/or health checks. It also generates data for these measurables, which can be plotted.

2.1 Measurables

There are three types of measurable:

metric: a numeric value, or no data.
health check: PASS/FAIL/UNKNOWN/no data.
enum metric: one of a set of user-defined string based values, or no data.

2.2 Measurables Classes

All measurables are grouped into classes. A class is a user-defined free string field, with / as delimiters. Base View uses this class to build a tree for easy search and access.

2.3 Metric Monitoring Data Producers

A metric data producer script generates one data point.

For example, as in the following script:

Example

[root@basecm10 ~]# cat /path/to/my/metric
#!/bin/bash
info_fd=${CMD_INFO_FD:=3}    #set CMD_INFO_FD to 3 by default only if nothing set
# CMD_INFO_FD:=3    <--- same as: if [ -z "$CMD_INFO_FD" ]; then CMD_INFO_FD=3; fi
echo $((RANDOM))
# Optionally provide extra information
echo "Extra information" >&$info_fd

The script can be defined as a metric script via the monitoring setup mode of cmsh:

Example

[basecm10]% monitoring setup
[basecm10->monitoring->setup]% add metric my-metric
[...my-metric]% set script /path/to/my/metric
[...my-metric]% set class My/Class
[...my-metric]% set unit B
[...my-metric]% set interval 1m
[...my-metric]% commit

All nodes then execute the script every minute, and produce a random number.

2.4 Health Check Monitoring Data Producers

A health check data producer script generates one data point. The data point can be one of four possible values expected of it: PASS, FAIL, UNKNOWN, or no data. Other file descriptors can be used to provide extra information.

For example, as in the following script:

Example

[root@basecm10 ~]# cat /path/to/my/health-check
#!/bin/bash
info_fd=${CMD_INFO_FD:=3}    #set CMD_INFO_FD to 3 by default if nothing set
if [ $((RANDOM)) -gt 8000 ]; then
  echo "PASS"
else
  echo "FAIL"
  # Optionally provide extra information
    echo "Extra information" >&$info_fd
fi

The script can be defined as a health check script via the monitoring setup mode of cmsh:

Example

[basecm10]% monitoring setup
[basecm10->monitoring->setup]% add healthcheck my-health-check
[...my-check]% set script /path/to/my/health-check
[...my-check]% set class My/Class
[...my-check]% set interval 1m
[...my-check]% commit

All nodes then execute the script every minute, and produce data values with roughly 75% PASS and 25% FAIL.

2.5 Collection Monitoring Data Producers

A collection data producer script can generate multiple data points in one run. Data points can be a combination of metrics and health checks. Collection scripts are also allowed to produce no data.

A collection script has two modes: initialize mode and sample mode.

initialize: defines the measurables that data values are generated for.
sample: returns the data values for all the measurables defined in initialize mode.

During normal cluster operation the initialize mode is called only once, during boot. Afterwards, the script is called in sample mode at the desired interval.

The following example combines both of the metric and health check examples from earlier on. However, this time it is written as a single script, using JSON as the output format:

Example

[root@basecm10 ~]# cat /path/to/my/collection
#!/usr/bin/python
import sys
import json
import random
def initialize():
    metric = {"metric": "my.collection.metric",
              "unit": "B",
              "class": "My/Collection"}
    check = {"check": "my.collection.check",
             "class": "My/Collection"}
    return [metric, check]
def sample():
    metric = {"metric": "my.collection.metric",
              "value": random.randint(0, 32767)}
    check = {"check": "my.collection.check",
             "info": "random with 25% failure rate",
             "value": 'PASS' if random.randint(0, 32767) > 8000 else 'FAIL'}
    return [metric, check]
def main():
    if len(sys.argv) > 1 and sys.argv[1] == "--initialize":
        data = initialize()
    else:
        data = sample()
    print (json.dumps(data, indent=4))
if __name__ == '__main__':
    main()

The script can be defined as a collection script via the monitoring setup mode of cmsh:

Example

[basecm10]% monitoring setup
[basecm10->monitoring->setup]% add collection my-collection
[...my-collection]% set script /path/to/my/collection
[...my-collection]% set format JSON
[...my-collection]% set interval 1m
[...my-collection]% commit

All nodes then execute the script every minute and produce two data points upon each execution. That is, one metric and one health check per execution.

2.6 Perpetual Monitoring Data Producers

A perpetual data producer script is a special case of a collection data producer script. It is intended to be used if the script needs permanent memory storage.

Example

[root@basecm10 ~]# cat /path/to/my/perpetual
#!/usr/bin/python
import my_sampler_module
import json
import time
# create single instance
sampler = my_sampler_module.MySampler()
# load important data into memory
sampler.load()
# Infinite loop with its own timing
delay = 0
while True:
    time.sleep(delay)
    (definitions, values, delay) = sampler.process()
    if definitions:
        # Print new measurables
        print (json.dumps(definitions))
    # Print data
    print (json.dumps(values))

The my_sampler_module is the part which does the important work.

Example

[root@basecm10 ~]# cat /path/to/my/my_sampler_module.py
class MySampler:
    def __init__(self):
        self.initialized = False
        self.definitions = None
    def load(self):
        # Do time consuming work here
        metric = {"metric": "my.collection.metric",
                  "unit": "B",
                  "class": "My/Collection"}
        check = {"check": "my.collection.check",
                 "class": "My/Collection"}
        self.definitions = [metric, check]
    def process(self):
        metric = {"metric": "my.collection.metric",
                  "value": random.randint(0, 32767)}
        check = {"check" : "my.collection.check",
                 "value" : 'PASS' if random.randint(0, 32767) > 8000 else 'FAIL'}
        values = metric, check
        # return definitions once, afterwards they never change
        # but new definitions could be added this way
        definitions = self.definitions
        self.definitions = None
        return definitions, values, 60

The script can be defined as a perpetual script via the monitoring setup mode of cmsh:

Example

[basecm10]% monitoring setup
[basecm10->monitoring->setup]% add perpetual my-perpetual
[...my-perpetual]% set script /path/to/my/perpetual
[...my-perpetual]% set format JSON
[...my-perpetual]% commit

2.7 Prometheus Monitoring Data Producers

Prometheus is a monitoring and alerting toolkit (https://prometheus.io). A Prometheus monitoring data producer script parses data from a Prometheus exporter (https://prometheus.io/docs/instrumenting/exporters/)

The script can be defined as a Prometheus script via the monitoring setup mode of cmsh:

Example

[basecm10]% monitoring setup
[basecm10->monitoring->setup]% add prometheus my-prometheus-exporter
[...my-prometheus-exporter]% set urls http://my.prometheus.exporter:80
[...my-prometheus-exporter]% set interval 1m
[...my-prometheus-exporter]% commit

If multiple URLs are defined, then only the data values from the first successful HTTP GET are used.

2.8 Node Execution Filters

By default a monitoring data producer script is executed on every node. When this is not desirable, a node execution filter should be created. A node execution filter defines the nodes on which the producer script should be executed.

For example, a filter to execute the script only on cloud nodes can be configured as follows:

Example

[basecm10]% monitoring setup use my-check
[...my-check]% nodeexecutionfilters
[...nodeexecutionfilters]% add type Cloud
[...nodeexecutionfilters*[Cloud*]]% set cloudnode yes
[...nodeexecutionfilters*[Cloud*]]% show
Parameter          Value
------------------ ----------------------------
Base type          MonitoringExecutionFilter
Name               Cloud
Type               Type
Head node          no
Physical node      no
Cloud node         yes
Virtual node       no
Lite node          no
[...nodeexecutionfilters*[Cloud*]]% commit

It is also possible to filter based on the specific resources associated with a node:

Example

[basecm10]% monitoring setup use my-IB-check
[...my-IB-check]% nodeexecutionfilters
[...nodeexecutionfilters]% add resource IB
[...nodeexecutionfilters*[IB*]]% set resources IB
[...nodeexecutionfilters*[IB*]]% commit

Because of high availability, a special resource, active, is defined for the active head node.

Example

[basecm10]% monitoring setup use my-metric
[...my-metric]% nodeexecutionfilters
[...nodeexecutionfilters]% active
Added active resource filter
[...nodeexecutionfilters*]% commit

2.9 Execution Multiplexers

By default a monitoring data producer script is executed once: the node executes the script only for itself.

However, some scripts, such as BMC samplers, must be sampled from the active head node for all nodes.

In the following example a BMC script is run on each node that has the ipmi or drac resource:

Example

[basecm10]% monitoring setup use my-ipmi-collection
[...my-ipmi-collection]% executionmultiplexers
[...executionmultiplexers]% add resource ipmi
[...executionmultiplexers*[ipmi*]]% set resources ipmi drac
[...executionmultiplexers*[ipmi*]]% set operator OR
[...executionmultiplexers*[ipmi*]]% commit

If an execution multiplexer <multiplexer> is defined, then there should also be a node execution filter <filter> associated with it to restrict the number of nodes on which the script runs.

This is because having the script run on many nodes for many other nodes is unlikely to be a desired configuration.

The combination of the execution filter and the multiplexer should be read as:

for every node that matches filter, run script, for each node that matches multiplexer.

A more specific example, using two of the preceding examples, with a filter based on the resource IB, and multiplexers based on the IPMI/Drac resources, the combination should be read as:

for every node that matches IB, run script, for each node that matches ipmi or drac.

2.10 Monitoring Resources

Every device in BCM has one or more resources. These resources are automatically calculated from:

Roles
Hardware
Settings

Resources for a specific node can be viewed as follows:

Example

[basecm10]% device use node001
[basecm10]% monitoringresources
CentOS7u5
Ethernet
category:default

It is possible to add one or more custom resources to a device:

Example

[basecm10]% device use node001
[basecm10]% add userdefinedresources MyResource
[basecm10]% append userdefinedresources MyOtherResource
[basecm10]% # wait ~10 seconds for the settings to propagate
[basecm10]% monitoringresources
CentOS7u5
Ethernet
category:default
MyResource
MyOtherResource

Any of these resources can be used to filter and multiplex monitoring data producers.

If a resources changes because of a settings change, then monitoring automatically stops or starts sampling.

2.11 Collection Monitoring Data Producers With Filter And Multiplexer

If a script has an execution multiplexer set, then it needs to determine for which nodes the script runs:

Example

[root@basecm10~]# cat /path/to/my/collection
#!/usr/bin/python
import sys
import json
import random
def initialize(entity):
    metric = {"metric": "my.collection.metric",
              "entity": entity,
              "unit": "B",
              "class": "My/Collection"}
    check = {"check": "my.collection.check",
             "entity": entity,
             "class": "My/Collection"}
    return [metric, check]
def sample(entity):
    metric = {"metric": "my.collection.metric",
              "entity": entity,
              "value": random.randint(0, 32767)}
    check = {"check" : "my.collection.check",
             "entity": entity,
             "value" : 'PASS' if random.randint(0, 32767) > 8000 else 'FAIL'}
    return [metric, check]
def main():
    try:
        # determine for which node we are sampling
        entity = os.environ['CMD_HOSTNAME']
    except:
        sys.stderr.write('Target device not specified in environment\n')
        return
    if len(sys.argv) > 1 and sys.argv[1] == "--initialize":
        data = initialize(entity)
    else:
        data = sample(entity)
    print (json.dumps(data, indent=4))
if __name__ == '__main__':
    main()

It can be defined with a filter to run on the active head for all nodes in the GPU category:

Example

[basecm10]% monitoring setup
[basecm10->monitoring->setup]% add collection my-collection
[...my-collection]% set script /path/to/my/collection
[...my-collection]% set format JSON
[...my-collection]% set interval 1m
[...my-collection]% nodeexecutionfilters
[...nodeexecutionfilters]% active
Added active resource filter
[...nodeexecutionfilters]% exit
[...my-collection]% executionmultiplexers
[...executionmultiplexers]% add category
[...executionmultiplexers*[GPU*]% add category GPU
[...executionmultiplexers*[GPU*]% commit

The script is then executed on the head, once for each node in the category of GPU.

2.12 Collection Monitoring Data Producers For Standalone Entities

Sometimes monitoring data does not belong to a BCM entity.

For this reason the standalone monitored entity was added in NVIDIA Base Command Manager 8.0.

This entity can be anything with a name and custom type.

BCM does nothing with this kind of entity, except allow it to store monitoring data.

Each standalone entity which needs to be monitored should be added:

Example

[basecm10]% monitoring standalone
[basecm10->monitoring->standalone]% add MSD.0
[...standalone*[MSD.0*]]% set type Lustre
[...standalone*[MSD.0*]]% commit
[...standalone*[MSD.0*]]% add MSD.1
[...standalone*[MSD.1*]]% set type Lustre
[...standalone*[MSD.1*]]% commit

A script can be created that produces data for all MSD entities:

Example

[root@basecm10 ~]# cat /path/to/my/collection
#!/usr/bin/python
import sys
import json
def initialize():
    msd_0 = {"metric": "lustre.free.space",
             "entity": "MSD.0",
             "unit": "B",
             "class": "Lustre"}
    msd_1 = {"metric": "lustre.free.space",
             "entity": "MSD.1",
             "unit": "B",
             "class": "Lustre"}
    return [msd_0, msd_1]
def sample():
    msd_0 = {"metric": "lustre.free.space",
             "entity": "MSD.0",
             "value": 12345,
             "class": "Lustre"}
    msd_1 = {"metric": "lustre.free.space",
             "entity": "MSD.1",
             "value": 54321}
    return [msd_0, msd_1]
def main():
    if len(sys.argv) > 1 and sys.argv[1] == "--initialize":
        data = initialize()
    else:
        data = sample()
    print (json.dumps(data, indent=4))
if __name__ == '__main__':
    main()

It can be defined to run on only the active head node:

Example

[basecm10]% monitoring setup
[basecm10->monitoring->setup]% add collection my-collection
[...my-collection]% set script /path/to/my/collection
[...my-collection]% set format JSON
[...my-collection]% set interval 5m
[...my-collection]% nodeexecutionfilters
[...nodeexecutionfilters]% active
Added active resource filter
[...nodeexecutionfilters]% commit

The script is then executed on the active head every 5 minutes and collects one data point for each MSD.

Data for a standalone script can be viewed with the same commands as for regular BCM nodes.

Example

[basecm10]% monitoring standalone
[basecm10->monitoring->standalone]% use MSD.0
[...standalone*[MSD.0*]]% latestmetricdata
...
lustre.free.space        12345         3m 47s

2.13 Debugging Standalone Scripts

Here describes how debugging information can be obtained when running samplenow with the --debug option with the ntp healthcheck script.

Many scripts under /cm/local/apps/cmd/scripts/ can have their debug output inspected with samplenow --debug.

A recursive grep on the head node, similar to the following, should show which scripts have a settable debug environment:

grep -r CMD_DEBUG /cm/local/apps/cmd/scripts/

The debug output in the script can be specified along the lines of the following code snippet:

Example

import os
debug = os.environ.get('CMD_DEBUG', '0') == '1'
info_fd = int(os.environ.get('CMD_INFO_FD', '3'))
if debug:
    os.write(info_fd, 'debug message')