> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Node Execution Filters And Execution Multiplexers Node execution filters and execution multiplexers define where data producers are executed on the nodes of a cluster, and what nodes are targeted to obtain the data. This appendix explains how node execution filters and execution multiplexers work with the help of some explicit basic examples. The aim is to have the cluster administrator understand how they work and how to use them. The reference cluster in this section is a 5-node cluster, made up of a head node (`basecm11`) and 4 regular nodes (`node001`..`node004`). The commands run in this appendix are carried out during a `cmsh` session that continues on from the point that it left off earlier. The terms “node execution filters” and “execution multiplexers” are commonly abbreviated to filters and multiplexers in this appendix. A simple custom data producer script is created and used to explain some of the more-involved concepts of filters and multiplexers more clearly. The custom script is: ```bash [root@basecm11 ~]# cat /cm/shared/fm.sh #!/bin/bash echo $((RANDOM%100)) echo "Sampled on $(hostname) for $CMD_HOSTNAME" >&3 ``` It should be made executable, for example, with `chmod a+x`. When run, the script outputs a random number, it outputs the host it is being run on (`$(hostname)`), and also the host the metric is targeting (`$CMD_HOSTNAME`). The hosts that it is run on can be defined by filters, while the hosts that are targeted by the metric can be defined by multiplexers. **The term multiplex:** The word “multiplex” can be confusing to system administrators. In electronics, the term multiplex implies that signals are being gathered from various inputs, and going into a main input. Here the idea is applicable to the signals (samples) from the execution multiplexers (nodes where the samples are). The samples are multiplexed (gathered) from those nodes, to the node (or nodes) where the data producer is executing. * The execution of the data producer is on the node (or nodes) defined by `nodeexecutionfilter`. The data producer execution nodes are the ones listed using the `nodes` command of `cmsh`. * The nodes where the samples are obtained from are defined by the `executionmultiplexer` setting. Those muliplexer nodes can have their samples displayed as output using the `samplenow` command (section [10.6.2.1](/admin/monitoring#10621-the-samplenow-command-for-on-demand-measurable-samples), [here](/admin/monitoring#10621-the-samplenow-command-for-on-demand-measurable-samples)) of `cmsh`. ## L.1 Data Producers: Default Configuration For Running And Sampling If there is no configuration defined for the data producer in the filters or multiplexers for that data producer, then each node runs a data producer on itself, and that data producer targets the node that it is running on. For example, the existing `dmesg` data producer comprises the `dmesg` health check (section [G.2.1](/admin/metricshealthchecksactions#g21-regular-health-checks)) and by default has no filter or multiplexer defined for it. If an attempt is made to list any filter or multiplexer for `dmesg`, then by default there is no content under the table headings: ```bash [root@basecm11 ~]# cmsh [basecm11]% monitoring setup [basecm11->monitoring->setup]% nodeexecutionfilters dmesg; list; ..;.. Type Name (key) Filter Filter operation ------------ ------------------------ ------------------------ ---------------- [basecm11->monitoring->setup]% executionmultiplexers dmesg; list; ..;.. Name (key) ------------------------ ``` Another way of seeing that no such filters or multiplexers have been defined for `dmesg` could be by seeing that none are defined in its submodes: ```bash [basecm11->monitoring->setup]% show dmesg | grep submode Execution multiplexer <0 in submode> Node execution filters <0 in submode> ``` Most existing data producers have filters and multiplexers defined. The number of filters and multiplexers set per data producer can conveniently be viewed via list formatting: **Example** ```bash [basecm11->monitoring->setup]% list -f name,nodeexecutionfilters,executionmultiplexer | more name (key) nodeexecutionfilters executionmultiplexer -------------------- -------------------- -------------------- AggregateNode <1 in submode> <1 in submode> AggregatePDU <1 in submode> <1 in submode> AlertLevel <1 in submode> <1 in submode> CMDaemonState <0 in submode> <0 in submode> Cassandra <1 in submode> <0 in submode> ClusterTotal <1 in submode> <0 in submode> DeviceState <1 in submode> <0 in submode> ... ``` ### L.1.1 Nodes That Data Producers Are Running On By Default—The `nodes` Command The `nodes` command shows which nodes the data producer runs on. By default, the data producer runs on all nodes, when nothing has been set explicitly, because each node runs the data producer for itself: ```bash [basecm11->monitoring->setup]% nodes dmesg node001..node004,basecm11 ``` ### L.1.2 Nodes That Data Producers Target By Default—The `samplenow` Command Nodes where samples are being obtained at can be seen using the `samplenow` command for the specified nodes. Again, by default, each node is a target, because the target is same node that the `dmesg` data producer runs on: ```bash [basecm11->monitoring->setup]% device samplenow -t node dmesg Entity Measurable Type Value Age Info ------------ ------------ ------------ ---------- ---------- ---------- node001 dmesg OS PASS 0.093s node002 dmesg OS PASS 0.087s node003 dmesg OS PASS 0.088s node004 dmesg OS PASS 0.09s basecm11 dmesg OS PASS 0.179s ``` In the outputs to `samplenow` displayed in this appendix, some columns are omitted for the sake of clarity. The `-t node` option to `samplenow` expands to `-n node001..node004,basecm11` for this reference cluster. ## L.2 Data Producers: Configuration For Running And Targeting Filters and multiplexers define which nodes run the data producers, and which nodes are the targets for measurables. The `fm.sh` script introduced here can be used to define several custom metrics according to what nodes run the script and what nodes are targeted by the script. ### L.2.1 Custom Metrics From The `fm.sh` Custom Script Custom data producers of type `metric` are created in this section. These data producers comprise custom metrics, which are now set up with varying filtering and multiplexing definitions, to illustrate how the definitions work. #### L.2.1.1 The Metric `all_for_self` The metric from the data producer `all_for_self` can be set up with no filtering or multiplexing defined, as follows: ```bash [basecm11->monitoring->setup]% add metric all_for_self [basecm11->monitoring->setup*[all_for_self*]]% set consolidator none [basecm11->monitoring->setup*[all_for_self*]]% set script /cm/shared/fm.sh [basecm11->monitoring->setup*[all_for_self*]]% set class Test [basecm11->monitoring->setup*[all_for_self*]]% commit ``` The value for `class` is mandatory but arbitrary. It is an arbitrary grouping mechanism, which can be useful in Base View for grouping folders in trees. **Sampling results for the metric `all_for_self`:** With no filtering or multiplexing, the metric just runs everywhere by default, with the target of the running metric being itself too. The `Info` field output from `samplenow` shows this behavior in the script output: ```bash [basecm11->monitoring->setup[all_for_self]]% exit [basecm11->monitoring->setup]% device samplenow -t node all_for_self Entity Measurable Type Value Age Info ------------ ------------- ----- ------ ------ ----------------------------------- node001 all_for_self Test 2 0.08s Sampled on node001 for node001 node002 all_for_self Test 13 0.084s Sampled on node002 for node002 node003 all_for_self Test 97 0.08s Sampled on node003 for node003 node004 all_for_self Test 18 0.084s Sampled on node004 for node004 basecm11 all_for_self Test 63 0.144s Sampled on basecm11 for basecm11 ``` #### L.2.1.2 The Metric `some_for_self` The metric `some_for_self` can be set up with filtering set up for some nodes, and no multiplexing set up, as follows: ```bash # on node001,node002 for itself [basecm11->monitoring->setup]% add metric some_for_self [basecm11->monitoring->setup*[some_for_self*]]% set consolidator none [basecm11->monitoring->setup*[some_for_self*]]% set class Test [basecm11->monitoring->setup*[some_for_self*]]% set script /cm/shared/fm.sh [basecm11->...*[some_for_self*]]% nodeexecutionfilters [basecm11->...*[some_for_self*]->nodeexecutionfilters]% add node some_nodes [basecm11->...->nodeexecutionfilters*[some_nodes*]]% set nodes node001 node002 [basecm11->...->nodeexecutionfilters*[some_nodes*]]% show Parameter Value -------------------------------- ------------------------------------------------ Filter operation Include Name some_nodes Nodes node001,node002 Revision Type Node [basecm11->...->nodeexecutionfilters*[some_nodes*]]% commit ``` **Sampling results for the metric `some_for_self`:** Only some filtering defined, and no multiplexing defined at all, means that the metric just runs on the filtered nodes, and targets only the nodes defined in filter too: ```bash [basecm11->...->nodeexecutionfilters[some_nodes]]% ..; ..; .. [basecm11->monitoring->setup]% device samplenow -t node some_for_self Entity Measurable Type Value Age Info ------------ -------------- ----- ------ ------ ------------------------------- node001 some_for_self Test 88 0.094s Sampled on node001 for node001 node002 some_for_self Test 98 0.075s Sampled on node002 for node002 ``` #### L.2.1.3 The Metric `from head_for_some_others` The metric `from_head_for_some_others` can be set up with filtering defined for the head node, and multiplexing defined for some other regular nodes (other than `node001` and `node002` here), as follows: ```bash # on active head for node003,node004 [basecm11->monitoring->setup]% add metric from_head_for_some_others [basecm11->monitoring->setup*[from_head_for_some_others*]]% set consolidator none [basecm11->monitoring->setup*[from_head_for_some_others*]]% set script /cm/shared/fm.sh [basecm11->monitoring->setup*[from_head_for_some_others*]]% set class Test [basecm11->...*[from_head_for_some_others*]]% nodeexecutionfilters [basecm11->...*[from_head_for_some_others*]->nodeexecutionfilters]% active Added active resource filter [basecm11->...*[from_head_for_some_others*]->nodeexecutionfilters]% show active head node Parameter Value -------------------------------- ------------------------------------------------ Filter operation Include Name Active head node Operator OR Resources Active Revision Type Resource [basecm11->monitoring->setup*[from_head_for_some_others*]->nodeexecutionfilters]% .. [basecm11->...*[from_head_for_some_others*]]% executionmultiplexers [basecm11->...*[from_head_for_some_others*]->executionmultiplexers]% add node other_nodes [basecm11->..._for_some_others*]->executionmultiplexers*[other_nodes*]]% set nodes node003 node004 [basecm11->...*[from_head_for_some_others*]->executionmultiplexers*[other_nodes*]]% show Parameter Value -------------------------------- ------------------------------------------------ Filter operation Include Name other_nodes Nodes node003,node004 Revision Type Node [basecm11->..._for_some_others*]->executionmultiplexers*[other_nodes*]]% commit; ..; ..;.. [basecm11->monitoring->setup]% ``` The filter is given a resource, `Active`, which is a way to set the filter for the active node only. Available resources for a node can be seen by running the command `monitoringresources` for a device: **Example** ```bash [basecm11->monitoring->setup]% device monitoringresources basecm11 Active Ethernet RDO backup boot ... ``` **Sampling results for the metric `from head_for_some_others`:** With filtering defined for the active head node, and multiplexing defined for those other nodes, it means that the metric targets those other nodes, and the metric runs on the head node. That is, the other nodes are targeted by the head node that is running the metric: ```bash [basecm11->monitoring->setup]% device samplenow -t node from_head_for_some_others Entity Measurable Type Value Age Info ------------ ------------------------- ----- ----- ----- --------------------------------- node003 from_head_for_some_others Test 20 0.084s Sampled on basecm11 for node003 node004 from_head_for_some_others Test 44 0.067s Sampled on basecm11 for node004 ``` The `nodes` command confirms that the head node, `basecm11` is the filtered node, that is, the only node(s) running the metric: ```bash [basecm11->monitoring->setup]% nodes from_head_for_some_others basecm11 ``` ## L.3 Replacing A Resource With An Explicit Node Specification Within the filter `Active head node`, associated with the metric `from_head_for_some_others`, the resource object `Active` can be replaced with a `node` object instead, if the node is defined as being `basecm11`. Doing this on a high-availability cluster where there is one active and one passive head node, would be unwise. However, doing this on the reference cluster for teaching purposes is of course absolutely fine because it helps make things a bit more concrete for the reader. The replacement can be carried out for the session as follows: ```bash [basecm11->monitoring->setup]% nodeexecutionfilters from_head_for_some_others [basecm11->...[from_head_for_some_others]->nodeexecutionfilters]% list Type Name (key) Filter Filter operation ------------ ------------------------ ------------------------ ---------------- Resource Active head node Active Include [basecm11->...[from_head_for_some_others]->nodeexecutionfilters]% remove active head node [basecm11->...*[from_head_for_some_others*]->nodeexecutionfilters*]% add node head [basecm11->...*[from_head_for_some_others*]->nodeexecutionfilters*[head*]]% set nodes basecm11 [basecm11->...*[from_head_for_some_others*]->nodeexecutionfilters*[head*]]% show Parameter Value -------------------------------- ------------------------------------------------ Filter operation Include Name head Nodes basecm11 Revision Type Node [basecm11->...*[from_head_for_some_others*]->nodeexecutionfilters*[head*]]% commit [basecm11->...[from_head_for_some_others]->nodeexecutionfilters*[head]]% ..;..;.. ``` The sample results are the same kind of output, and the filter node used is the same. The target sample outputs are the same kind of output as before the replacement. The filter node on which the metric runs is also seen to be the same: ```bash [basecm11->monitoring->setup]% device samplenow -t node from_head_for_some_others Entity Measurable Type Value Age Info ------------ -------------------------- ----- ----- ------ ----------------------------------- node003 from_head_for_some_others Test 11 0.081s Sampled on basecm11 for node003 node004 from_head_for_some_others Test 12 0.07s Sampled on basecm11 for node004 [basecm11->monitoring->setup]% nodes basecm11 ``` ## L.4 Excessive Sampling If another node is appended to the node in the filter, then a warning comes up. ```bash [basecm11->monitoring->setup]% nodeexecutionfilters from_head_for_some_others [basecm11->...[from_head_for_some_others]->nodeexecutionfilters]% append head nodes node003 [basecm11->...*[from_head_for_some_others*]->nodeexecutionfilters*]% commit ========================== from_head_for_some_others =========================== Field Message ------------------------ ------------------------------------------------------- executionMultiplexers Warning: Execution filters/multiplexers are set to run on multiple nodes for the same target. This likely means they are badly configured [basecm11->...[from_head_for_some_others]->nodeexecutionfilters]% show head Parameter Value ------------------------ ------------------------------------------------------- Filter operation Include Name head Nodes node003,basecm11 Revision Type Node ``` The warning is there because the node execution filters are doing the same thing from different nodes for an execution multiplexer target, and duplicating monitoring execution is typically a waste of resources, and thus typically a mistake. However, the warnings are merely warnings, and not errors. So BCM just goes ahead with setting up the filter/muliplex system according to what the administrator has specified. The `nodes` and `samplingnow` commands now show: ```bash [basecm11->...[from_head_for_some_others]->nodeexecutionfilters]% ..;.. [basecm11->monitoring->setup]% device samplenow -t node from_head_for_some_others Entity Measurable Type Value Age Info ------------ -------------------------- ----- ----- ------ --------------------------------- node003 from_head_for_some_others Test 3 0.151s Sampled on basecm11 for node003 node003 from_head_for_some_others Test 76 0.08s Sampled on node003 for node003 node004 from_head_for_some_others Test 25 0.08s Sampled on node003 for node004 node004 from_head_for_some_others Test 88 0.151s Sampled on basecm11 for node004 [basecm11->monitoring->setup]% nodes from_head_for_some_others node003,basecm11 ``` The node `node003` is now doing what the head node is, sampling the same targets, which is typically a bad idea. However the behavior is indeed as expected for this particular configuration. Whether running a particular configuration is actually wise, is up to the administrator—but in any case the filter/multiplex system allows plenty of abuse of this kind. ## L.5 Not Just For Nodes Nodes are what node execution filters and execution multiplexers run on. However, sometimes it is more convenient to execute based on other types. The possible types can be listed with tab-completion suggestions when adding a node execution filter or an execution multiplier: ```bash [basecm11->monitoring->setup[dmesg]->nodeexecutionfilters]% add category lua node overlay resource type ``` ## L.6 Lua Node Execution Filters Lua ([https://www.lua.org/](https://www.lua.org/)) is a lightweight scripting language embedded into CMDaemon. It allows more advanced node execution filters to be written using a Lua filter file. **Example** ```bash [basecm11->monitoring->setup[]->nodeexecutionfilters]% add lua lua-filter [basecm11->...nodeexecutionfilters*[lua-filter*]]% set code ``` In the preceding example session, the name `lua-filter` is an arbitrary name, that is added to the object that is associated with the Lua filter file \<*Lua filter file name*>. The `self` Lua table is passed by CMDaemon, and contains the entity for which the filter is evaluated. Only devices have a `self` table that is not `nil`. All other entities have a `self` table that is `nil`. For example, a filter can be created based on a regex match of the hostname: **Example** ```bash if self == nil then return false else return self.hostname:match("^node[0-9]+") ~= nil) end ``` The Lua filter is evaluated for all nodes: **Example** ```bash [basecm11->monitoring->setup[dmesg]]% nodes node001..node004 ``` Development of Lua filters is best done outside of CMDaemon. Doing so requires the cluster administrator to create node environments to be evaluated by hand: **Example** ```bash [root@basecm11 ~]# cat node001.lua self = {} self.hostname = "node001" self.category = "default" [root@basecm11 ~]# cat basecm11.lua self = {} self.hostname = "basecm11" ``` This is in addition to the original filter script: **Example** ```bash [root@basecm11 ~]# cat filter.lua if self == nil then return false else return self.hostname:match("^node[0-9]+") ~= nil end ``` Both nodes can then be run through the filter using the `lua` interpreter: **Example** ```bash [root@basecm11 ~]# lua Lua 5.1.4 Copyright (C) 1994-2015 Lua.org, PUC-Rio > dofile('basecm11.lua') > print(dofile('filter.lua')) false > dofile('node001.lua') > print(dofile('filter.lua')) true ``` The Lua `self` exported by CMDaemon contains the following: * `self`: The main table object, `nil` for non-devices. * `self.hostname`: The hostname of the device. * `self.partition`: The name of the partition to which the device belongs. * `self.category`: The name of the category to which the compute node belongs. * `self.nodegroups`: An array with the name of node groups the node belongs to. * `self.mac`: The MAC address of the device. * `self.ip`: The IP of the device, only available for non-node devices. * `self.network`: The IP address of the device, only available for non-node devices. * `self.interfaces`: The array of interface of a node. * `self.interfaces[1].name`: The name of the first interface. * `self.interfaces[1].ip`: The IP address of the first interface. * `self.interfaces[1].network`: The name of the network of the first interface. * `self.roles`: The array of roles of a node. * `self.roles[1].name`: The name of the first role. * `self.roles[1].type`: The type of the first role. * `self.status.status`: The status of the device. * `self.status.user_message`: The user message set for the device. * `self.status.info_message`: The information message set for the device. * `self.status.closed`: A boolean marking the node as closed. * `self.status.restart_required`: A boolean marking the node needs to be rebooted. * `self.status.healthcheck_failed`: A boolean marking at least one health check has returned `FAIL`. * `self.status.healthcheck_unknown`: A boolean marking at least one health check has returned `UNKNOWN`. * `self.status.state_flapping`: A boolean marking the node status transitioning often in a short time span * `self.system.name`: The name of the system. * `self.system.manufacturer`: The manufacturer of the system. * `self.system.motherboard.name`: The name of the motherboard. * `self.system.motherboard.manufacturer`: The manufacturer of the motherboard. * `self.system.bios.version`: The BIOS version. * `self.system.bios.vendor`: The BIOS vendor name. * `self.system.bios.date`: The BIOS date. * `self.system.os.name`: The OS name. * `self.system.os.version`: The OS version. * `self.system.os.flavor`: The OS flavor. Using the `self` environment, complex filters can easily be created. For example, the following filter can be built to include only the nodes on the IB network, which also have a Slurm client role: **Example** ```bash [root@basecm11 ~]# cat ib-slurm-filter.lua if self == nil then return false end on_ib_network = false for index, interface in ipairs(self.interfaces) do on_ib_network = on_ib_network or (interface.network == "ibnet") done slurm_client = false for index, role in ipair(self.roles) do slurm_client = slurm_client or (role.name == "SlurmClient") done return on_ib_network and slurm_client ``` It is also possible to use external sources, like the file system, to determine the filter for a node. **Example** ```bash [root@basecm11 ~]# cat file-check-filter.lua if self == nil then return false end function file_exists(name) local f = io.open(name, "r") if f ~= nil then io.close(f) return true else return false end end return file_exists(string.format('/opt/filter/%s', self.hostname)) ``` It is important to understand that this Lua script is evaluated for all nodes, on the active head node. The Lua script should therefore be fast, and return within a few milliseconds.