Skip to navigation

CMDaemon REST API

View as Markdown

Some data from CMDaemon can be accessed via its REST API.

The REST API typically allows data only to be retrieved for most calls. Exceptions are:

  • the Status call, which can generate status notifications (section 4.2.1), and

  • the Event call, which can generate events (section 4.2.9),

and which can take POST input to specify their calls.

4.1 Authentication, And Definition Of curlauth

Two forms of authentication are supported:

  • Basic: HTTP authentication (--basic option of curl)

  • Certificate: Certificate-based authentication (--cert option of curl). Certificate-based authentication is covered in section 6.4.2 of the Administrator Manual.

The following two commands give identical results:

[alice@basecm10 ~]$ curl -k --basic --user "alice:password" "https://master:8081/rest"
[alice@basecm10 ~]$ curl --cert ~/.cm/cert.pem --key ~/.cm/cert.key -k "https://master:8081/rest"

For security, it is best to use the certificate key-based version.

For convenience, the command and authority parts of the preceding two commands —that is the string in the line that includes the text from curl to 8081 in the two curl commands—is designated by <curlauth> in this chapter. Thus, each of the commands can be represented by:

Example

[alice@basecm10 ~]$ <curlauth>/rest"

This allows the reader to focus on the path segment and variables part of the API.

4.2 Browsing The API

A summary diagram of the REST API is shown in figure 4.1:

REST API summary tree

Figure 4.1: REST API summary tree

The remainder of this section elaborates upon the diagram.

The API directory structure is documented within the directory itself.

A GET operation on the main /rest entry point can list all subdirectories:

Example

[alice@basecm10 ~]$ <curlauth>/rest"
["v1"]

New lines are not part of the output by default. Setting a parameter of 1, 2, or more, for the indent variable uses newlines and an indentation of one, two, or more spaces. This makes the API output more readable for all API resource paths:

Example

[alice@basecm10 ~]$ <curlauth>/rest?indent=1"
[
"v1"
]

Appending /v1 to the URL gives the functionality available in the first version of the REST API.

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1?indent=1"
[
"monitoring",
"chargeback",
"status",
"session",
"check",
"version",
"license",
"sysinfo",
"device",
"workload",
"event"
]

Appending /monitoring to the URL lists the subdirectory functionality available for monitoring.

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring?indent=1"
[
"entity",
"measurable",
"latest",
"dump",
"usage"
]

4.2.1 Returning A Status, Or Generating A Status Message, Using /v1/status

4.2.1.1 Returning A Status Using /v1/status

The status resource path returns the UP/DOWN status for all devices:

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/status?indent=2"
[
{
"hostname": "basecm10",
"status": "UP"
},
{
"hostname": "node001",
"status": "UP"
},
{
"hostname": "node002",
"status": "DOWN"
}
]

The status can also be requested for a single device:

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/status?name=node001&indent=2"
[
{
"hostname": "node001",
"status": "UP"
}
]

The “two dots” list specification format (section 2.5.5.10 of the Administrator Manual) used in Base View and cmsh can also be used in the API:

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/status?name=node001..node002&indent=2"
[
{
"hostname": "node001",
"status": "UP"
},
{
"hostname": "node002",
"status": "DOWN"
}
]

For more detailed information, the verbose parameter can be added (output truncated):

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/status?verbose=1&indent=2"
[
{
"health_check_failed": true,
"health_check_unknown": false,
"hostname": "basecm10",
"provisioning_failed": false,
"restart_required": false,
"status": "UP"
},
{
"health_check_failed": true,
"health_check_unknown": false,
"hostname": "node001",
"provisioning_failed": false,
...

4.2.1.2 Generating A Status Message Using /v1/status

A message can be associated with the status. An input can be as in the following rest.in file:

Example

[alice@basecm10 ~]$ cat /tmp/rest.in
[
{
"hostname": "node001",
"user": "user message1",
"info": "info message1",
"tool": tool message1"
},
{
"hostname": "node002",
"user": "user message2",
"info": "info message2",
"tool": tool message2"
}
]

The usual curl authentication string used so far, <curlauth> (section 4.1) is slightly modified from its value of:

curl --cert ~/.cm/cert.pem --key ~/.cm/cert.key -k "https://master:8081

to

curl --cert ~/.cm/cert.pem --key ~/.cm/cert.key -k --data "@/tmp/rest.in" "https://master:8081

This modified version allows POST data to be entered. The modified version can be called <curlauthpost>, and can be used as follows, returning a vector with the components having an integer value of 0, up to 3:

Example

[alice@basecm10 ~]$ <curlauthpost>/rest/v1/status"
[
0,
3
]

The dimension of the vector is in the current example is 2, and corresponds to the number of hostnames. Thus, the first component is associated with the hostname node001, and the second component is associated with hostname node002.

The value of each component returns the number of fields in the POST file that were modified by the API call. The fields that are evaluated are the optional fields user, info, and tool.

The event is logged in the event logger, by default at /var/spool/cmd/events.log, as:

Example

Mon Jun 17 18:15:50 2024 [notice] basecm10: node001, status: UP, reported: UP, time: 1718113341752,
info message: info node1, user message: user node1, tool message: tool node1 (index: 27, display: 1)
Mon Jun 17 18:20:30 2024 [notice] basecm10: node002, status: UP, reported: UP, time: 1718113666566,
info message: info node2, user message: user node2, tool message: tool node2 (index: 28, display: 1)

An entry is also made in /var/log/cmdaemon:

Example

Jun 17 18:51:57 basecm10 cmd[2586]: [ CMD ] Info: [Service::post_v1_status], update node001, changes: 1
Jun 17 18:51:57 basecm10 cmd[2586]: [ CMD ] Info: [Service::post_v1_status], update node002, changes: 1

With the default settings of cmsh, a window running cmsh shows:

Example

[root@basecm10 ~]# cmsh
[basecm10]%
Mon Jun 17 18:51:58 2024 [notice] basecm10: node001 [ UP ] (info node1) (user node1) (tool node1)
Mon Jun 17 18:51:58 2024 [notice] basecm10: node002 [ UP ] (info node2) (user node2) (tool node2)

Messaging via the REST API is somewhat similar to the event bucket InfoMessages feature (section 12.10.4 of the Administrator Manual) but developers should find the REST API version cleaner.

4.2.2 Monitoring Using /v1/monitoring

4.2.2.1 Entities Via /v1/monitoring/entity

The entity resource returns information about the entities that are known to the monitoring system. It is possible for an entity known to the monitoring system to have no data.

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/entity?indent=1"
{
"entities": [
{
"key": 12884901889,
"name": "default",
"type": "Category"
},
{
"key": 17179869185,
"name": "globalnet",
"type": "Network"
},
{
"key": 17179869186,
"name": "internalnet",
"type": "Network"
},
...
[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/entity?name=node001&indent=1"
{
"entities": [
{
"key": 38654705666,
"name": "node001",
"type": "PhysicalNode"
}
]
}
[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/entity?type=node&indent=1"
{
"entities": [
{
"key": 38654705665,
"name": "basecm10",
"type": "HeadNode"
},
{
"key": 38654705666,
"name": "node001",
"type": "PhysicalNode"
},
{
"key": 38654705667,
"name": "node002",
"type": "PhysicalNode"
}
]
}
[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/entity?name=node001..node002&indent=1"
{
"entities": [
{
"key": 38654705666,
"name": "node001",
"type": "PhysicalNode"
},
{
"key": 38654705667,
"name": "node002",
"type": "PhysicalNode"
}
]
}

A regex matcher can be used to find entities based on a name match:

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/entity?like=lobal&indent=1"
{
"entities": [
{
"key": 17179869185,
"name": "globalnet",
"type": "Network"
}
]
}
[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/entity?like=nod.0*1&indent=1"
{
"entities": [
{
"key": 38654705666,
"name": "node001",
"type": "PhysicalNode"
}
]
}

Regexes are based on the ECMAScript specification (https://en.cppreference.com/w/cpp/regex/ecmascript).

4.2.2.2 Measurables Via /v1/monitoring/measurable

This entry returns information about the defined measurables.

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/measurable?indent=1"
{
"measurables": [
{
"key": 261993005057,
"name": "IpForwDatagrams",
"type": "metric"
},
{
"key": 261993005058,
"name": "IpFragCreates",
"type": "metric"
},
{
"key": 261993005059,
"name": "IpFragFails",
"type": "metric"
},
...typically hundreds more lines...
[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/measurable?name=loadone&indent=1"
{
"measurables": [
{
"key": 261993005138,
"name": "LoadOne",
"type": "metric"
}
]
}
[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/measurable?like=load&indent=1"
{
"measurables": [
{
"key": 261993005136,
"name": "LoadFifteen",
"type": "metric"
},
{
"key": 261993005137,
"name": "LoadFive",
"type": "metric"
},
{
"key": 261993005138,
"name": "LoadOne",
"type": "metric"
}
]
}

4.2.2.3 Data Usage Via /v1/monitoring/usage

The usage resource is intended to show which (entity, measurable) pairs have data. For example, nodes with only 1 disk do not have data, if their associated measurables have the string sdb in their name.

To get the complete usage:

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/usage?indent=1"
{
"data": [
{
"entity": "default",
"measurable": "CoresTotal"
},
{
"entity": "default",
"measurable": "CoresUp"
},
...typically hundreds more lines...

It is also possible to get all the measurables for which a specific entity, such as node001, has data.

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/usage?entity=node001&indent=1"
{
"data": [
{
"entity": "node001",
"measurable": "IpForwDatagrams"
},
{
"entity": "node001",
"measurable": "IpFragCreates"
},
{
"entity": "node001",
"measurable": "IpFragFails"
},
...typically hundreds more lines...

Or all entities which have data for a specific measurable such as loadone:

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/usage?measurable=loadone&indent=1"
{
"data": [
{
"entity": "basecm10",
"measurable": "LoadOne"
},
{
"entity": "node001",
"measurable": "LoadOne"
}
]
}

4.2.2.4 The Latest Monitoring Data Via /v1/monitoring/latest

The latest resource can be used to retrieve the last known sampled data points.

It is possible to get the latest monitoring data for all (entity, measurable) pairs.

This may result in a lot of information: about 125 bytes per (entity, measurable) pair.

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/latest?indent=1"
{
"data": [
{
"age": 47.868,
"entity": "default",
"measurable": "CoresTotal",
"raw": 1.0,
"time": 1540476088861,
"value": "1"
},
{
"age": 47.868,
"entity": "default",
"measurable": "CoresUp",
"raw": 1.0,
"time": 1540476088861,
"value": "1"
},
{
"age": 47.868,
"entity": "default",
"measurable": "NodesClosed",
"raw": 0.0,
"time": 1540476088861,
"value": "0"
},
{
"age": 47.868,
"entity": "default",
"measurable": "NodesDown",
"raw": 0.0,
"time": 1540476088861,
"value": "0"
},
...typically thousands more lines...

The latest data can be requested for a selection of entities and measurables.

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/latest?entity=node001&indent=1"
{
"data": [
{
"age": 138.625,
"entity": "node001",
"measurable": "IpForwDatagrams",
"raw": 0.0,
"time": 1540476100389,
"value": "0/s"
},
{
"age": 138.625,
"entity": "node001",
"measurable": "IpFragCreates",
"raw": 0.0,
"time": 1540476100389,
"value": "0/s"
},
{
"age": 138.625,
"entity": "node001",
"measurable": "IpFragFails",
"raw": 0.0,
"time": 1540476100389,
"value": "0/s"
},
...typically hundreds more lines...
[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/latest?entity=node001..node004&indent=1"
...as for previous output but for the range of nodes001..node004...
[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/latest?measurable=LoadOne&indent=1"
{
"data": [
{
"age": 114.099,
"entity": "basecm10",
"measurable": "LoadOne",
"raw": 0.03,
"time": 1540476351361,
"value": "0.03"
},
{
"age": 155.07,
"entity": "node001",
"measurable": "LoadOne",
"raw": 0.0,
"time": 1540476310390,
"value": "0"
}
]
}
[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/latest?entity=node001&measurable=LoadOne&indent=1"
{
"data": [
{
"age": 106.706,
"entity": "node001",
"measurable": "LoadOne",
"raw": 0.0,
"time": 1540476790390,
"value": "0"
}
]
}

4.2.2.5 Historic Data Dump Via /v1/monitoring/dump

Dumping historic data can be done using the entry point:

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/dump?<options>"

The dump resource has several options:

  • entity: name or range of entities

  • measurable: name of the measurable

  • start: time to be plotted (default: -1h)

  • end: end to be plotted (default: now)

  • intervals: number of interpolation intervals (default: 0, raw data)

  • epoch: display timestamps as unix epoch (default: 0)

The time specification format is the same one used for the dumpmonitoringdata command (section 12.6.4.1 of the Administrator Manual).

To prevent gigabytes of data being retrieved when no options are specified, entity and measurable must be specified.

If there is a need to dump all the monitoring data, then it can be done by specifying empty strings for both entity and measurable. For example, the following command dumps all raw data for the default last hour:

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/monitoring/dump?entity=&measurable=&indent=1"
{
"data": [
{
"entity": "default",
"measurable": "CoresTotal",
"raw": 1.0,
"time": "2018/10/25 13:15:28",
"value": "1"
},
{
"entity": "default",
"measurable": "CoresTotal",
"raw": 1.0,
"time": "2018/10/25 16:35:28",
"value": "1"
},
{
"entity": "default",
"measurable": "CoresUp",
"raw": 1.0,
"time": "2018/10/25 13:49:28",
"value": "1"
},
...typically thousands more lines...

4.2.3 Session Using /v1/session

The response to the sessions endpoint is similar to the output from listing in session mode of cmsh (cmsh -c "session list")

The endpoint lists the sessions that the cluster manager is involved with.

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/session?indent=1"
[
{
"address": "127.0.0.1",
"group": "admin",
"node": "basecm10",
"type": "node",
"username": ""
},
{
"address": "10.141.255.254",
"group": "admin",
"type": "node",
"username": ""
},
{
"address": "10.141.0.1",
"group": "node",
"node": "node001",
"type": "node",
"username": ""
},
{
"address": "10.141.0.2",
"group": "node",
"node": "node002",
"type": "node",
"username": ""
}

4.2.4 Version Using /v1/version

The version endpoint returns version parameters.

[alice@basecm10 ~]$ <curlauth>/rest/v1/version?indent=1"
{
"build_hash": "daf30669f1",
"build_index": 152175,
"cm_version": "9.2",
"cmd_version": "2.2",
"database_version": 36280
}

4.2.5 License Using /v1/license

The license endpoint returns license parameters.

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/license?indent=1"
{
"acceleratorNodeCount": 0,
"accountingAndReporting": true,
"baseType": "LicenseInfo",
"burstNodeCount": 0,
"childType": "",
"edgeSites": true,
"edition": "Advanced",
"endTime": 2177449140,
"licenseType": "Commercial",
"licensedAcceleratorNodes": 80,
"licensedBurstNodes": 1000,
"licensedNodes": 100,
"licensee": "/C=US/ST=None/L=None/O=None/OU=None/CN=basecm10",
"macAddress": "FA:16:3E:3B:94:98",
"message": "",
"modified": false,
"nodeCount": 3,
"oldLocalUniqueKey": 0,
"refPartitionUniqueKey": 21474836481,
"revision": "",
"serial": 1017214,
"startTime": 1508108400,
"toBeRemoved": false,
"uniqueKey": 281474976710653,
"version": "7.0 and above"
}

4.2.6 Sysinfo Using /v1/sysinfo

The sysinfo endpoint is similar to the sysinfo command in the device mode of cmsh. It returns information about some basic system hardware parameters.

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/sysinfo?indent=1"
{
"node001": {
"baseType": "SysInfoCollector",
"biosDate": "04/01/2014",
"biosVendor": "SeaBIOS",
"biosVersion": "1.13.0-1ubuntu1.1",
"bootIf": "ens3",
"childType": "",
"clusterRandomNumber": 6332472641088672013,
"diskCount": 2,
"diskTotalSpace": 10745806848,
"disks": [
{
"baseType": "DiskInfo",
"childType": "",
"ioScheduler": "[mq-deadline] kyber bfq none",
"model": "virtio",
"modified": false,
"name": "vda",
"oldLocalUniqueKey": 0,
"rev": "",
"revision": "",
"size": 8388608,
"toBeRemoved": false,
"uniqueKey": 281474976710948,
"vendor": ""
},
{
"baseType": "DiskInfo",
"childType": "",
"ioScheduler": "[mq-deadline] kyber bfq none",
"model": "virtio",
"modified": false,
"name": "vdb",
"oldLocalUniqueKey": 0,
"rev": "",
"revision": "",
"size": 10737418240,
"toBeRemoved": false,
"uniqueKey": 281474976710949,
"vendor": ""
}
],
"extra": null,
"fabric": false,
"fips": false,
"fpgas": [],
"gpus": [],
"ibGUIDs": [],
"interconnects": [],
"memory": [
{
"IDs": [
"0/0"
],
"baseType": "MemoryInfo",
"childType": "",
"description": "DIMM RAM",
"locations": [
"DIMM 0"
],
"modified": false,
"oldLocalUniqueKey": 0,
"revision": "",
"size": 1073741824,
"speed": 0,
"toBeRemoved": false,
"uniqueKey": 281474976710950
}
],
"memorySwap": 0,
"memoryTotal": 1016152064,
"modified": false,
"motherboardManufacturer": "",
"motherboardName": "",
"nics": [
"ens3"
],
"oldLocalUniqueKey": 0,
"osFlavor": "Rocky8u5",
"osName": "Linux",
"osVersion": "4.18.0-348.el8.0.2.x86_64",
"parentUniqueKey": 85899345921,
"processors": [
{
"IDs": [
0
],
"baseType": "Processor",
"bogomips": 4190.15,
"cacheSize": 16777216,
"childType": "",
"coreIDs": [
0
],
"cores": 1,
"model": "Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz",
"modified": false,
"oldLocalUniqueKey": 0,
"physicalIDs": [
0
],
"revision": "",
"speed": 2095078000.0,
"toBeRemoved": false,
"uniqueKey": 281474976710947,
"vendor": "GenuineIntel"
}
],
"raidControllers": [],
"refDeviceUniqueKey": 38654705666,
"revision": "",
"seLinux": false,
"systemManufacturer": "OpenStack Foundation",
"systemName": "OpenStack Nova",
"timestamp": 1651158566,
"toBeRemoved": false,
"uniqueKey": 85899345921,
"updateCount": 5,
"vendorTag": "5bf2a543-542d-4391-946c-abb648a09158",
"virtualCluster": true
},
"node002": {
"baseType": "SysInfoCollector",
"biosDate": "04/01/2014",
...
}

4.2.7 Device Information Using /v1/device

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/device?indent=1"
[
{
"cluster": "basecm10",
"hostname": "basecm10",
"ip": "10.141.255.254",
"mac": "FA:16:3E:EF:71:05",
"network": "internalnet",
"roles": [
"backup",
"storage",
"firewall",
"headnode",
"monitoring",
"provisioning",
"boot"
],
"type": "HeadNode"
},
{
"category": "default",
"cluster": "basecm10",
"hostname": "node001",
"ip": "10.141.0.1",
"mac": "FA:16:3E:2B:A4:31",
"network": "internalnet",
"type": "PhysicalNode"
},
{
"category": "default",
"cluster": "basecm10",
"hostname": "node002",
"ip": "10.141.0.2",
"mac": "FA:16:3E:D4:C8:5A",
"network": "internalnet",
"type": "PhysicalNode"
}
]

4.2.8 WLM Information Using /v1/workload

The workkoad path has the following endpoints:

  • jobs

  • drain

These return information related to the endpoints.

The workload jobs path returns running jobs:

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/workload/jobs?indent=1"
[
{
"account": "projecty",
"group": "alice",
"job_id": "2301",
"job_name": "iozone",
"nodes": [
"node001"
],
"queue": "defq",
"run_time": "4m 39s",
"start_time": "2023/06/08 14:24:53",
"state": "RUNNING",
"submit_time": "2023/06/08 14:24:53",
"user": "alice"
},
{
"account": "projectx",
"group": "charlie",
"job_id": "2306",
"job_name": "sleep",
"nodes": [
"node001"
],
"queue": "defq",
"run_time": "3m 34s",
"start_time": "2023/06/08 14:25:58",
"state": "RUNNING",
"submit_time": "2023/06/08 14:25:57",
"user": "charlie"
},
{
"account": "projecty",
"group": "alice",
"job_id": "2307",
"job_name": "iozone",
...

The workload drain path returns node drain status information:

Example

[alice@basecm10 ~]$ <curlauth>/rest/v1/workload/drain?indent=1"
[{
"hostname": "node001",
"queue": "defq",
"reason": "",
"status": "UNDRAINED"
},
{
"hostname": "node002",
"queue": "defq",
"reason": "",
"status": "UNDRAINED"
},
...

4.2.9 Event Generation Using /v1/event

An event (section 12.10 of the Administrator Manual) can be generated in CMDaemon from a JSON format input used with the event endpoint.

An input can be as in the following rest.in file:

Example

[alice@basecm10 ~]$ cat /tmp/rest.in
{
"message": "hello world",
"details": "send via rest",
"severity": "notice"
}

The usual curl authentication string used so far, <curlauth> (section 4.1) is slightly modified from its value of:

curl --cert ~/.cm/cert.pem --key ~/.cm/cert.key -k "https://master:8081

to

curl --cert ~/.cm/cert.pem --key ~/.cm/cert.key -k --data "@/tmp/rest.in" "https://master:8081

This modified version allows POST data to be entered. The modified version can be called <curlauthpost>, and can be used as follows, returning true:

Example

[alice@basecm10 ~]$ <curlauthpost>/rest/v1/event"
true

The event is logged in the event logger, by default at /var/spool/cmd/events.log, as:

Example

Tues Jun 4 11:09:21 2024 [notice] basecm10: hello world
send via rest

With the default settings of cmsh, a window running cmsh shows:

Example

[root@basecm10 ~]# cmsh
[basecm10]%
Tue Jun 4 11:09:21 2024 [notice] basecm10: hello world
For details type: events details 1

and, if as suggested, events details 1 is typed, the value of details from the input is seen:

[basecm10]% events details 1
send via rest