Skip to navigation

NVIDIA Base Command Manager Python API

View as Markdown

This chapter introduces the Python API of NVIDIA Base Command Manager.

The Python API package was completely overhauled in NVIDIA Base Command Manager 8.2.

The cmdaemon-pythoncm package now provides a pure Python connection to the cluster manager, making it possible for cluster administrators to automate cluster operations via Python.

It also makes it possible to run Python code on any operating system that supports Python 3.5 and later.

The BCM Python API uses the following extra modules:

  1. pyOpenSSL

  2. ply

  3. lxml

  4. tabulate

  5. monotonic

  6. humanfriendly

  7. pyYAML

  8. six

1.1 Getting Started

On the cluster head node itself the python3 module can simply be loaded:

Example

[root@basecm11 ~]# module load python3

To execute or develop the Python code on any other machine usually requires some extra steps:

  • Python 3 (3.5, 3.6, 3.7, 3.8 will work) should be installed

  • The 8 extra modules listed previously should be installed using Pip

  • The /cm/local/apps/cmd/pythoncm/lib/python3.9/site-packages/pythoncm directory should be copied over to the site-packages directory of the development machine. The version is enforced in lib/python3.9/site-packages/pythoncm/__init__.py, so version consistency would be required here.

The pythoncm module should then be loaded, to confirm everything was set up correctly:

Example

[alice@desktop ~]# python -c "import pythoncm"

If connecting from outside the cluster, then port 8081 must not be blocked by a firewall.

A certificate is needed by the Python API to identify itself to CMDaemon.

The existence of the certificate on the head node should be checked. It should be copied over to the development machine, if it is needed there.

Example

[root@basecm11 ~]# ls -al .cm/
-rw------- 1 root root 1708 Dec 11 09:25 admin.key
-rw------- 1 root root 1269 Dec 11 09:25 admin.pem

Example

[alice@basecm11 ~]$ ls -al .cm/
-rw------- 1 root root 1708 Dec 11 09:25 cert.key
-rw------- 1 root root 1269 Dec 11 09:25 cert.pem

The developer many need to contact the cluster administrator to get a certificate.

1.2 Connecting To A Cluster

The first step when working with the Python API is to establish a connection to the CMDaemon process on the cluster:

#!/usr/bin/env python
from pythoncm.cluster import Cluster
from pythoncm.settings import Settings
cluster = Cluster()

If working outside the cluster, then the settings for connecting to the cluster must be specified:

settings = Settings(host='<head-node-hostname>',
                    port=8081,
                    cert_file='/some/path/cert.pem',
                    key_file='/some/path/cert.key',
                    ca_file='.../site-packages/pythoncm/etc/cacert.pem')
if not settings.check_certificate_files():
    print('Unable to load certificates')
else:
    cluster = Cluster(settings)

1.3 Inspecting Settings

All settings in BCM are stored inside an entity.

Each entity has a type and a unique name among the entities of the same type.

To inspect an entity it should first be found inside the cluster:

node001 = cluster.get_by_name('node001')

If the name node001 was also given a different entity, then the type must be specified to ensure that the correct entity is returned:

node001 = cluster.get_by_name('node001', pythoncm.entity.Node)
node001 = cluster.get_by_name('node001', 'Node')

Once the node entity is found, inspecting the settings is a matter of printing the desired field:

print(node001.hostname)
print(node001.mac)

Complex settings, such as network interfaces, have their own settings:

for interfaces in node001.interfaces:
    print(interface.name, interface.ip)

Because many nodes could have a network interface called eth0, such a setting cannot be found from the cluster: The following code will return None.

eth0 = cluster.get_by_name('eth0')

To find all eth0 interfaces, all nodes need to be found, and then iterated over:

nodes = cluster.get_by_type(pythoncm.entity.Node)
all_eth0 = [interface
            for node in nodes
            for interface in node.interfaces
            if interface.name == 'eth0']

1.4 Modifying Settings

Basic entity settings are exported as Python properties and can simply be changed:

node001.mac = '00:00:00:00:00:00'
node001.category = cluster.get_by_name('gpu', 'Category')

Similarly interfaces settings can be accessed and changed directly:

node001.interfaces[0].ip = '1.2.3.4'
node001.interfaces[0].network = cluster.get_by_name('ib', 'Network')

Removing an interface from a node can be done in various Pythonic ways:

node001.interfaces.remove(0)
del node001.interfaces[0]
node001.interfaces = [interface for interface in node001.interfaces
                      if interface.name != 'eth0']

To add a new interface, the entity instance needs to be created first, and then added to the node:

eth1 = pythoncm.entity.NetworkPhysicalInterface()
eth1.name = 'eth1'
eth1.ip = '1.2.3.4'
eth1.network = cluster.get_by_name('ib', 'Network')
node001.interfaces.append(eth1)

All changes are made on a local copy of the entity. The cluster has no knowledge of the changes until they are committed.

It is recommended to make many changes locally, and only commit once at the end.

The return value of the commit operation should always be checked.

Committing a badly-configured node will be blocked by the head node:

commit_result = node001.commit()
if not commit_result.good:
    print(commit_result)

An entity found using the cluster object is removed differently.

As with commit, the result should always be checked: a removal can fail if a node is UP:

remove_result = node001.remove()
if not remove_result.good:
    print(remove_result)

1.5 Inspecting The Entire Cluster

The example directory contains a script to inspect the entire cluster.

Example

[root@basecm11 ~]# cd /cm/local/examples/cmd/pythoncm/
[root@basecm11 pythoncm]# module load python3
[root@basecm11 pythoncm]# ./print-all.py

The example directory also contains a script that prints all metadata for all available entities in BCM:

Example

[root@basecm11 pythoncm]# ./entity_info.py

1.6 Performing Operations On Entities

All Python API functionality is contained in /cm/local/apps/cmd/pythoncm/lib/python3.9/site-packages/pythoncm.

Methods are documented inside the python code itself.

node001 = cluster.get_by_name('node001')
node001.power_on()

When operating on multiple entities, it is possible to iterate over them and do each operation individually.

nodes = cluster.get_by_type('Node')
for node in nodes:
    node.power_on()

However the same can also be done with a parallel version of the operation. When possible the parallel version should be used, because it is faster and requires less network traffic.

nodes = cluster.get_by_type('Node')
cluster.parallel.power_on(nodes)

1.7 Monitoring

All monitoring data can be accessed using the Python API.

Monitoring is a set of operations performed on entities.

For example, to get latest data for a single entity:

print(node001.get_latest_monitoring_data())

Monitoring operations on multiple operations should be done using the monitoring module:

data = cluster.monitoring.get_latest_monitoring_data([node001,
                                                      node002,
                                                      node003])

1.8 Examples

The best way to get going is by looking at the examples. These can be found on the head node, at /cm/local/examples/cmd/pythoncm:

Example

[root@basecm11 pythoncm]# ls
add-collection.py                    dump-job-monitoring-data.py       power-history.py
add-healthcheck.py                   dump-monitoring-data.py           power-parallel-status.py
add-metric.py                        entity_info.py                    power-status.py
add-node-group.py                    execute.py                        print-all.py
add-role.py                          fabric-bindings.py                range-expander-test.py
add-user.py                          fake-file-write.py                remove-many-nodes.py
all-background-tasks.py              free_port.py                      remove-node-group.py
all-nodes.py                         get-all-wlm-jobs.py               sample-all-checks.py
arch_os_image_info.py                get-status.py                     sample-now-checks.py
certificate-info.py                  health-overview.py                sample-now-parallel.py
charge-back.py                       instance_by_name.py               select-devices.py
clone-many-nodes.py                  instant-query.py                  service.py
clone-node-group.py                  key_value_pair.py                 service-status.py
clone-node.py                        latest-counter-data.py            set-node-image.py
cm-job-analytics.py                  latest-health-data.py             test-add-update-remove.py
cm-job-gpu.py                        monitoring-push.py                tftpboot-file-information.py
cm-network-traffic-monitor-setup.py  move-to-new-pdu.py                total-job-power-usage.py
config-writer.py                     new-nodes.py                      up-percentage.py
cookie.py                            parallel-execute-async.py         user-data.py
create-certificate.py                parallel-execute-check-status.py  wait-for-provisioning.py
create-ramdisk-task.py               parallel-execute.py               wait-for-up.py

The examples can be tried out after loading the Python environment:

Example

[root@basecm11 ~]# cd /cm/local/examples/cmd/pythoncm/
[root@basecm11 pythoncm]# module load python3
[root@basecm11 pythoncm]# ./power-status.py
INFO     (25-May-2020 18:29:25) [cluster.py          :207] Follow redirection to active head IP:
10.141.255.254
INFO     (25-May-2020 18:29:25) [cluster.py          :298] Start event thread for session
42949672967
success: True
[
    "uniqueKey": 1125899906842642,
    "oldLocalUniqueKey": 0,
    "baseType": "PowerStatus",
    "childType": "",
    "revision": "",
    "modified": false,
    "toBeRemoved": false,
    "readonly": false,
    "not_set_fields": [],
    "device": 38654705666,
    "host": 38654705665,
    "powerDistributionUnit": 0,
    "gpu": -1,
    "prt": 0,
    "name": "custom",
    "state": "ON",
    "msg": "",
    "extendedMsg": "",
    "indexes": [
        2
    ],
    "tracker": 0,
    "retries": 0
]
INFO     (25-May-2020 18:29:28) [entity_change.py    : 38] Stop event change watcher
[root@basecm11 pythoncm]#