Node Execution Filters And Execution Multiplexers
Node execution filters and execution multiplexers define where data producers are executed on the nodes of a cluster, and what nodes are targeted to obtain the data.
This appendix explains how node execution filters and execution multiplexers work with the help of some explicit basic examples. The aim is to have the cluster administrator understand how they work and how to use them.
The reference cluster in this section is a 5-node cluster, made up of a head node (basecm11) and 4 regular nodes (node001..node004). The commands run in this appendix are carried out during a cmsh session that continues on from the point that it left off earlier.
The terms “node execution filters” and “execution multiplexers” are commonly abbreviated to filters and multiplexers in this appendix.
A simple custom data producer script is created and used to explain some of the more-involved concepts of filters and multiplexers more clearly. The custom script is:
It should be made executable, for example, with chmod a+x. When run, the script outputs a random number, it outputs the host it is being run on ($(hostname)), and also the host the metric is targeting ($CMD_HOSTNAME). The hosts that it is run on can be defined by filters, while the hosts that are targeted by the metric can be defined by multiplexers.
The term multiplex: The word “multiplex” can be confusing to system administrators. In electronics, the term multiplex implies that signals are being gathered from various inputs, and going into a main input.
Here the idea is applicable to the signals (samples) from the execution multiplexers (nodes where the samples are). The samples are multiplexed (gathered) from those nodes, to the node (or nodes) where the data producer is executing.
-
The execution of the data producer is on the node (or nodes) defined by
nodeexecutionfilter. The data producer execution nodes are the ones listed using thenodescommand ofcmsh. -
The nodes where the samples are obtained from are defined by the
executionmultiplexersetting. Those muliplexer nodes can have their samples displayed as output using thesamplenowcommand (section 10.6.2.1, here) ofcmsh.
L.1 Data Producers: Default Configuration For Running And Sampling
If there is no configuration defined for the data producer in the filters or multiplexers for that data producer, then each node runs a data producer on itself, and that data producer targets the node that it is running on.
For example, the existing dmesg data producer comprises the dmesg health check (section G.2.1) and by default has no filter or multiplexer defined for it. If an attempt is made to list any filter or multiplexer for dmesg, then by default there is no content under the table headings:
Another way of seeing that no such filters or multiplexers have been defined for dmesg could be by seeing that none are defined in its submodes:
Most existing data producers have filters and multiplexers defined. The number of filters and multiplexers set per data producer can conveniently be viewed via list formatting:
Example
L.1.1 Nodes That Data Producers Are Running On By Default—The nodes Command
The nodes command shows which nodes the data producer runs on. By default, the data producer runs on all nodes, when nothing has been set explicitly, because each node runs the data producer for itself:
L.1.2 Nodes That Data Producers Target By Default—The samplenow Command
Nodes where samples are being obtained at can be seen using the samplenow command for the specified nodes.
Again, by default, each node is a target, because the target is same node that the dmesg data producer runs on:
In the outputs to samplenow displayed in this appendix, some columns are omitted for the sake of clarity.
The -t node option to samplenow expands to -n node001..node004,basecm11 for this reference cluster.
L.2 Data Producers: Configuration For Running And Targeting
Filters and multiplexers define which nodes run the data producers, and which nodes are the targets for measurables.
The fm.sh script introduced here can be used to define several custom metrics according to what nodes run the script and what nodes are targeted by the script.
L.2.1 Custom Metrics From The fm.sh Custom Script
Custom data producers of type metric are created in this section. These data producers comprise custom metrics, which are now set up with varying filtering and multiplexing definitions, to illustrate how the definitions work.
L.2.1.1 The Metric all_for_self
The metric from the data producer all_for_self can be set up with no filtering or multiplexing defined, as follows:
The value for class is mandatory but arbitrary. It is an arbitrary grouping mechanism, which can be useful in Base View for grouping folders in trees.
Sampling results for the metric all_for_self:
With no filtering or multiplexing, the metric just runs everywhere by default, with the target of the running metric being itself too. The Info field output from samplenow shows this behavior in the script output:
L.2.1.2 The Metric some_for_self
The metric some_for_self can be set up with filtering set up for some nodes, and no multiplexing set up, as follows:
Sampling results for the metric some_for_self: Only some filtering defined, and no multiplexing defined at all, means that the metric just runs on the filtered nodes, and targets only the nodes defined in filter too:
L.2.1.3 The Metric from head_for_some_others
The metric from_head_for_some_others can be set up with filtering defined for the head node, and multiplexing defined for some other regular nodes (other than node001 and node002 here), as follows:
The filter is given a resource, Active, which is a way to set the filter for the active node only. Available resources for a node can be seen by running the command monitoringresources for a device:
Example
Sampling results for the metric from head_for_some_others: With filtering defined for the active head node, and multiplexing defined for those other nodes, it means that the metric targets those other nodes, and the metric runs on the head node. That is, the other nodes are targeted by the head node that is running the metric:
The nodes command confirms that the head node, basecm11 is the filtered node, that is, the only node(s) running the metric:
L.3 Replacing A Resource With An Explicit Node Specification
Within the filter Active head node, associated with the metric from_head_for_some_others, the resource object Active can be replaced with a node object instead, if the node is defined as being basecm11. Doing this on a high-availability cluster where there is one active and one passive head node, would be unwise. However, doing this on the reference cluster for teaching purposes is of course absolutely fine because it helps make things a bit more concrete for the reader. The replacement can be carried out for the session as follows:
The sample results are the same kind of output, and the filter node used is the same. The target sample outputs are the same kind of output as before the replacement. The filter node on which the metric runs is also seen to be the same:
L.4 Excessive Sampling
If another node is appended to the node in the filter, then a warning comes up.
The warning is there because the node execution filters are doing the same thing from different nodes for an execution multiplexer target, and duplicating monitoring execution is typically a waste of resources, and thus typically a mistake.
However, the warnings are merely warnings, and not errors. So BCM just goes ahead with setting up the filter/muliplex system according to what the administrator has specified. The nodes and samplingnow commands now show:
The node node003 is now doing what the head node is, sampling the same targets, which is typically a bad idea. However the behavior is indeed as expected for this particular configuration.
Whether running a particular configuration is actually wise, is up to the administrator—but in any case the filter/multiplex system allows plenty of abuse of this kind.
L.5 Not Just For Nodes
Nodes are what node execution filters and execution multiplexers run on. However, sometimes it is more convenient to execute based on other types.
The possible types can be listed with tab-completion suggestions when adding a node execution filter or an execution multiplier:
L.6 Lua Node Execution Filters
Lua (https://www.lua.org/) is a lightweight scripting language embedded into CMDaemon. It allows more advanced node execution filters to be written using a Lua filter file.
Example
In the preceding example session, the name lua-filter is an arbitrary name, that is added to the object that is associated with the Lua filter file <Lua filter file name>.
The self Lua table is passed by CMDaemon, and contains the entity for which the filter is evaluated. Only devices have a self table that is not nil. All other entities have a self table that is nil.
For example, a filter can be created based on a regex match of the hostname:
Example
The Lua filter is evaluated for all nodes:
Example
Development of Lua filters is best done outside of CMDaemon. Doing so requires the cluster administrator to create node environments to be evaluated by hand:
Example
This is in addition to the original filter script:
Example
Both nodes can then be run through the filter using the lua interpreter:
Example
The Lua self exported by CMDaemon contains the following:
-
self: The main table object,nilfor non-devices. -
self.hostname: The hostname of the device. -
self.partition: The name of the partition to which the device belongs. -
self.category: The name of the category to which the compute node belongs. -
self.nodegroups: An array with the name of node groups the node belongs to. -
self.mac: The MAC address of the device. -
self.ip: The IP of the device, only available for non-node devices. -
self.network: The IP address of the device, only available for non-node devices. -
self.interfaces: The array of interface of a node. -
self.interfaces[1].name: The name of the first interface. -
self.interfaces[1].ip: The IP address of the first interface. -
self.interfaces[1].network: The name of the network of the first interface. -
self.roles: The array of roles of a node. -
self.roles[1].name: The name of the first role. -
self.roles[1].type: The type of the first role. -
self.status.status: The status of the device. -
self.status.user_message: The user message set for the device. -
self.status.info_message: The information message set for the device. -
self.status.closed: A boolean marking the node as closed. -
self.status.restart_required: A boolean marking the node needs to be rebooted. -
self.status.healthcheck_failed: A boolean marking at least one health check has returnedFAIL. -
self.status.healthcheck_unknown: A boolean marking at least one health check has returnedUNKNOWN. -
self.status.state_flapping: A boolean marking the node status transitioning often in a short time span -
self.system.name: The name of the system. -
self.system.manufacturer: The manufacturer of the system. -
self.system.motherboard.name: The name of the motherboard. -
self.system.motherboard.manufacturer: The manufacturer of the motherboard. -
self.system.bios.version: The BIOS version. -
self.system.bios.vendor: The BIOS vendor name. -
self.system.bios.date: The BIOS date. -
self.system.os.name: The OS name. -
self.system.os.version: The OS version. -
self.system.os.flavor: The OS flavor.
Using the self environment, complex filters can easily be created. For example, the following filter can be built to include only the nodes on the IB network, which also have a Slurm client role:
Example
It is also possible to use external sources, like the file system, to determine the filter for a node.
Example
It is important to understand that this Lua script is evaluated for all nodes, on the active head node. The Lua script should therefore be fast, and return within a few milliseconds.