CMDaemon Configuration File Directives
This appendix lists all configuration file directives that may be used in the cluster management daemon configuration file. If a change is needed, then the directives are normally changed on the head node, or on both head nodes in the high availability configuration, in:
The directives can also be set in some cases for the regular nodes, via the software image in /cm/images/default-image/cm/local/apps/cmd/etc/cmd.conf on the head node. Changing the defaults already there is however not usually needed, and is not recommended.
Only one directive is valid per cmd.conf file.
To activate changes in a cmd.conf configuration file, the cmd service associated with it must be restarted.
-
For the head node this is normally done with the command:
-
For regular nodes,
cmdrunning on the nodes is restarted. Often, the image should be updated beforecmdis restarted. How to carry out these procedures for a directive is described with an example where the FrozenFile directive is activated on a regular node here.
C.1 Master directive
Syntax: Master = hostname
Default: Master = master
The cluster management daemon treats the host specified in the Master directive as the head node. A cluster management daemon running on a node specified as the head node starts in head mode. On a regular node, it starts in node mode.
C.2 Port directive
Syntax: Port = number
Default: Port = 8080
The number used in the syntax above is a number between 0 and 65535. The default value is 8080.
The Port directive sets the value of the port of the cluster management daemon to listen for non-SSL HTTP calls. By default, this happens only during init. All other communication with the cluster management daemon is carried out over the SSL port. Pre-init port adjustment can be carried out in the node-installer.conf configuration. The head node firewall may need to be modified to allow traffic through for a changed port.
C.3 SSLPort directive
Syntax: SSLPort = number
Default: SSLPort = 8081
The number used in the syntax above is a number between 0 and 65535. The default value is 8081.
The SSLPort directive sets the value of the SSL port of the cluster management daemon to listen for SSL HTTP calls. By default, it is used for all communication of CMDaemon with Base View and cmsh, except for when CMDaemon is started up from init.
This directive does not change the firewall port settings to match the value of the SSL port for CMDaemon communication.
C.4 SSLPortOnly directive
Syntax: SSLPortOnly = yes|no
Default: SSLPortOnly = no
The SSLPortOnly directive allows the non-SSL port to be disabled. By default, during normal running, both SSL and non-SSL ports are listening, but only the SSL port is used. Also by default, the non-SSL port is only used during CMDaemon start up.
If bootloaderprotocol (section 5.1.6) is set to HTTP, then SSLPortOnly must be set to no. The HTTPS protocol is unsupported by most bootloaders.
C.4.0.1 Changing CMDaemon Ports
The cm-cmd-ports utility changes the HTTP and HTTPS ports on which CMDaemon listens. The default ports are 8080 (HTTP) and 8081 (HTTPS), as set by the Port (here) and SSLPort directives in cmd.conf (here).
The current port configuration can be listed with:
To change the HTTPS port, for example to port 8082:
The firewall configuration is not automatically updated by cm-cmd-ports. The firewall rules file must be updated manually as part of the procedure. This procedure should be performed during a maintenance window.
Example
An example session is as follows:
On the active head node, the port is changed with:
The software images are updated on the provisioners with the updateprovisioners command (here), and an update is triggered for all non-image fsparts with the trigger command (here):
The provisioning status can be checked with:
An empty output indicates that provisioning is complete, and that the procedure may continue. CMDaemon can then be restarted on the active head node:
In a high availability configuration, CMDaemon on the passive head node is restarted as well, but only after the CMDaemon on the active head node has successfully restarted. This order keeps the passive in sync with the active. Updating the passive first can cause issues later on.
The updated cmd.conf is then copied from the software images to the compute nodes:
The command is run for each node image in use on the cluster.
CMDaemon on the compute nodes is then restarted:
The firewall rules on the head nodes are updated by editing /etc/shorewall/rules to replace the old port number with the new port number, after which Shorewall is reloaded:
The port change can be verified with:
From BCM 12 onwards, Shorewall is superseded by nftables.
C.5 CertificateFile directive
Syntax: CertificateFile = filename
Default: CertificateFile = "/cm/local/apps/cmd/etc/cert.pem"
The CertificateFile directive specifies the PEM-format certificate which is to be used for authentication purposes. On the head node, the certificate used also serves as a software license.
C.6 PrivateKeyFile directive
Syntax: PrivateKeyFile = filename
Default: PrivateKeyFile = "/cm/local/apps/cmd/etc/cert.key"
The PrivateKeyFile directive specifies the PEM-format private key which corresponds to the certificate that is being used.
C.7 CACertificateFile directive
Syntax: CACertificateFile = filename
Default: CACertificateFile = "/cm/local/apps/cmd/etc/cacert.pem"
The CACertificateFile directive specifies the path to the BCM PEM-format root certificate. It is normally not necessary to change the root certificate.
C.8 ClusterCertificateFile directive
Syntax: ClusterCertificateFile = filename
Default: ClusterCertificateFile = "/cm/local/apps/cmd/etc/cluster.pem"
The ClusterCertificateFile directive specifies the path to the BCM PEM-format cluster certificate file used as a software license, and to sign all client certificates
C.9 ClusterPrivateKeyFile directive
Syntax: ClusterPrivateKeyFile = filename
Default: ClusterPrivateKeyFile = "/cm/local/apps/cmd/etc/cluster.key"
The ClusterPrivateKeyFile directive specifies the path to the BCM PEM-format private key which corresponds to the cluster certificate file.
C.10 RandomSeedFile directive
Syntax: RandomSeedFile = filename
Default: RandomSeedFile = "/dev/urandom"
The RandomSeedFile directive specifies the path to a source of randomness for a random seed.
C.11 RandomSeedFileSize directive
Syntax: RandomSeedFileSize = number
Default: RandomSeedFileSize = 8192
The RandomSeedFileSize directive specifies the size of the random seed.
C.12 DHParamFile directive
Syntax: DHParamFile = filename
Default: DHParamFile = "/cm/local/apps/cmd/etc/dh1024.pem"
The DHParamFile directive specifies the path to the Diffie-Hellman parameters.
C.13 SSLHandshakeTimeout directive
Syntax: SSLHandshakeTimeout = number
Default: SSLHandshakeTimeout = 10
The SSLHandShakeTimeout directive controls the time-out period (in seconds) for SSL handshakes.
C.14 SSLSessionCacheExpirationTime directive
Syntax: SSLSessionCacheExpirationTime = number
Default: SSLSessionCacheExpirationTime = 300
The SSLSessionCacheExpirationTime directive controls the period (in seconds) for which SSL sessions are cached. Specifying the value 0 can be used to disable SSL session caching.
C.15 DBHost directive
Syntax: DBHost = hostname
Default: DBHost = "localhost"
The DBHost directive specifies the hostname of the MySQL database server.
C.16 DBPort directive
Syntax: DBPort = number
Default: DBHost = 3306
The DBPort directive specifies the TCP port of the MySQL database server.
C.17 DBUser directive
Syntax: DBUser = username
Default: DBUser = cmdaemon
The DBUser directive specifies the username used to connect to the MySQL database server.
C.18 DBPass directive
Syntax: DBPass = password
Default: DBPass = "<random string set during installation>”
The DBPass directive specifies the password used to connect to the MySQL database server.
C.19 DBName directive
Syntax: DBName = database
Default: DBName = "cmdaemon"
The DBName directive specifies the database used on the MySQL database server to store CMDaemon related configuration and status information.
C.20 DBUnixSocket directive
Syntax: DBUnixSocket = filename
Default: DBUnixSocket = "/var/lib/mysql/mysql.sock"
The DBUnixSocket directive specifies the named pipe used to connect to the MySQL database server if it is running on the same machine.
C.21 DBUpdateFile directive
Syntax: DBUpdateFile = filename
Default: DBUpdateFile = "/cm/local/apps/cmd/etc/cmdaemon_upgrade.sql"
The DBUpdateFile directive specifies the path to the file that contains information on how to upgrade the database from one revision to another.
C.22 EventBucket directive
Syntax: EventBucket = filename
Default: EventBucket = "/var/spool/cmd/eventbucket"
The EventBucket directive (section 10.10.3) specifies the path to the named pipe that is created to listen for incoming events from a user.
C.23 EventBucketFilter directive
Syntax: EventBucketFilter = filename
Default: EventBucketFilter = "/cm/local/apps/cmd/etc/eventbucket.filter"
The EventBucketFilter directive (section 10.10.3) specifies the path to the file that contains regular expressions used to filter out incoming messages on the event-bucket.
C.24 LDAPHost directive
Syntax: LDAPHost = hostname
Default: LDAPHost = "localhost"
The LDAPHost directive specifies the hostname of the LDAP server to connect to for user management.
C.25 LDAPUser directive
Syntax: LDAPUser = username
Default: LDAPUser = "root"
The LDAPUser directive specifies the username used when connecting to the LDAP server.
C.26 LDAPPass directive
Syntax: LDAPPass = password
Default: LDAPPass = "<random string set during installation>"
The LDAPPass directive specifies the password used when connecting to the LDAP server. It can be changed following the procedure described in Appendix I.
C.27 LDAPReadOnlyUser directive
Syntax: LDAPReadOnlyUser = username
Default: LDAPReadOnlyUser = "readonlyroot"
The LDAPReadOnlyUser directive specifies the username that will be used when connecting to the LDAP server during LDAP replication. The user is a member of the “rogroup” group, whose members have a read-only access to the whole LDAP directory.
C.28 LDAPReadOnlyPass directive
Syntax: LDAPReadOnlyPass = password
Default: LDAPReadOnlyPass = "<random string set during installation>"
The LDAPReadOnlyPass directive specifies the password that will be used when connecting to the LDAP server during LDAP replication.
C.29 LDAPSearchDN directive
Syntax: LDAPSearchDN = dn
Default: LDAPSearchDN = "dc=cm,dc=cluster"
The LDAPSearchDN directive specifies the Distinguished Name (DN) used when querying the LDAP server.
C.30 LDAPProtocol directive
Syntax: LDAPProtocol = ldap|ldaps
Default: LDAPProtocol = "ldaps"
The LDAPProtocol directive specifies the LDAP protocol to be used when querying the LDAP server.
C.31 LDAPPort directive
Syntax: LDAPPort = number
Default: LDAPPort = 636
The LDAPPort directive specifies the port to be used when querying the LDAP server.
C.32 LDAPCACertificate directive
Syntax: LDAPCACertificate = filename
Default: LDAPCACertificate = "/cm/local/apps/openldap/etc/certs/ca.pem"
The LDAPCACertificate directive specifies the CA certificate to be used when querying the LDAP server.
C.33 LDAPCertificate directive
Syntax: LDAPCertificate = filename
Default: LDAPCertificate = "/cm/local/apps/openldap/etc/certs/ldap.pem"
The LDAPCertificate directive specifies the LDAP certificate to be used when querying the LDAP server.
C.34 LDAPPrivateKey directive
Syntax: LDAPPrivateKey= filename
Default: LDAPPrivateKey= "/cm/local/apps/openldap/etc/certs/ldap.key"
The LDAPPrivateKey directive specifies the LDAP key to be used when querying the LDAP server.
C.35 HomeRoot directive
Syntax: HomeRoot = path
Default: HomeRoot = "/home"
The HomeRoot directive specifies the default user home directory used by CMDaemon. It is used for automatic mounts, exports, and when creating new users.
C.36 DocumentRoot directive
Syntax: DocumentRoot = path
Default: DocumentRoot = "/cm/local/apps/cmd/etc/htdocs"
The DocumentRoot directive specifies the directory mapped to the web-root of the CMDaemon. The CMDaemon acts as a HTTP-server, and can therefore in principle also be accessed by web-browsers.
C.37 SpoolDir directive
Syntax: SpoolDir = path
Default: SpoolDir = "/var/spool/cmd"
The SpoolDir directive specifies the directory which is used by the CMDaemon to store temporary and semi-temporary files.
C.38 EnableShellService directive
Syntax: EnableShellService = true|false
Default: EnableShellService = true
The EnableShellService directive allows shells to be started from Base View.
The connection runs over CMDaemon, which is running over SSL, which means that between Base View and the device, the connection is encrypted.
The directive does not affect cmsh’s rshell, rconsole telnet, and ssh commands.
C.39 DisableRemoteShell directive
Syntax: AdvancedConfig = {"DisableRemoteShell=0|1", ...}
Default: DisableRemoteShell=0
DisableRemoteShell is a parameter of the AdvancedConfig (here) directive.
By default CMDaemon provides access to devices via rshell, rconsole (SOL), telnet, and ssh. Setting the directive to 1 disables all RPC shell access.
A more fine-grained disabling is possible with the following AdvancedConfig directives, which follow the same syntax:
-
DisableRemoteRShell directive: disables a remote shell to the head node.
-
DisableRemoteRShellImage directive: disables a remote shell to the chrooted software image.
-
DisableRemoteRShellNode directive: disables a remote shell to the regular nodes.
-
DisableRemoteRShellTelnet directive: disables a telnet shell to a device.
-
DisableRemoteRShellSSH directive: disables SSH to a device.
-
DisableRemoteSOL directive: disables a console via SOL to a device.
C.40 EnableWebSocketService directive
Syntax: EnableWebSocketService = true|false
Default: EnableWebSocketService = true
The EnableWebSocketService directive allows the use of CMDaemon Lite (section 2.6.7).
C.41 EnablePrometheusMetricService directive
Syntax: EnablePrometheusMetricService = true|false
Default: EnablePrometheusMetricService = true
If true, the EnablePrometheusMetricService directive creates an HTTP endpoint for Prometheus-style exporters that do not have their own HTTP endpoint.
C.42 PrometheusMetricServicePath directive
Syntax: PrometheusMetricServicePath = path
Default: PrometheusMetricServicePath = SCRIPTS_DIR"metrics/prometheus"
The PrometheusMetricServicePath directive is the path from which CMDaemon can serve Prometheus metrics. SCRIPTS_DIR is the stem path /cm/local/apps/cmd/scripts by default.
C.43 EnablePrometheusExporterService directive
Syntax: EnablePrometheusExporterService = true|false
Default: EnablePrometheusExporterService = false
If true, the EnablePrometheusExporterService directive allows a Prometheus server to pull data from CMDaemon.
To collect all data, the external Prometheus YAML configuration (https://prometheus.io/docs/prometheus/latest/configuration/configuration/) should have all nodes with a monitoring role added to it:
Example
Configuring CMDaemon to sample metrics from a Prometheus exporter instead is covered in section 3.18.
C.44 PrometheusExporterRequireCertificate directive
Syntax: AdvancedConfig = {"PrometheusExporterRequireCertificate=0|1", ...}
Default: PrometheusExporterRequireCertificate = 1
PrometheusExporterRequireCertificate is a parameter of the AdvancedConfig (here) directive.
-
If set to
0, then all users can use the Prometheus service to pull data using CMDaemon. -
If set to
1, then only users using certificate-based access can use the Prometheus service to pull data using CMDaemon.
Certificate-based access can be configured as follows:
A custom profile with the name prometheus can be created for BCM users with profile mode (section 6.5) as follows:
A 2048-bit SSL certificate called mypromcert can be created for a non-privileged user prometheus with 3650 days validity using the createcertificate command (section 6.5.2) as follows:
The PEM certificate is the public part of the key pair. The private key must be owned and be readable by the Prometheus instance that is scraping from the node exporter.
To configure SSL authentication between Prometheus server and the Prometheus exporter node:
-
On the Prometheus server:
-
A directive value of:
-
0means that the session key is a temporary key that is auto-generated for any user -
1means that the user (the Prometheus server) must provide a certificate
-
-
CMDaemon in a default cluster manager installation uses a self-signed certificate. A copy of certificate can be picked up on the head node with:
Example
The Prometheus server should be configured to specify this CA certificate.
If the certificate used by CMDaemon is not a self-signed certificate—that is to say, it is signed by a recognized CA—then the certificate issued by CMDaemon is legitimate, and the Prometheus server does not need to have that CA certificate explicitly specified.
-
-
The Prometheus exporter always uses TLS, which is automatically managed by CMDaemon.
C.45 EnablePrometheusPushService directive
Syntax: EnablePrometheusPushService = true|false
Default: EnablePrometheusPushService = false
If true, the EnablePrometheusPushService directive allows a Prometheus server to push data to CMDaemon:
Example
C.46 CMDaemonAudit directive
Syntax: CMDaemonAudit = true|false
Default: CMDaemonAudit = true
When the CMDaemonAudit directive is set to true, and a value is set for the CMDaemon auditor file with the CMDaemonAuditorFile directive, then CMDaemon actions are time-stamped and logged in the CMDaemon auditor file.
C.47 CMDaemonAuditorFile directive
Syntax: CMDaemonAuditorFile = path to audit.log file
or
Syntax: CMDaemonAuditorFile = path to audit.json file
or
Syntax: CMDaemonAuditorFile = http(s) path to JSON server
Default: CMDaemonAuditorFile = "/var/spool/cmd/audit.log"
The CMDaemonAuditorFile directive sets where the audit logs for CMDaemon actions are logged. The log format for a .log file in a standard directory path is:
(time stamp) profile [IP-address] action (unique key)
Example
The directive can be set in the following kinds of formats:
-
CMDaemonAuditorFile = "/var/spool/cmd/audit.log" -
CMDaemonAuditorFile = "/var/spool/cmd/audit.json" -
CMDaemonAuditorFile = "http://<IP address>:<port>/some/path" -
CMDaemonAuditorFile = "https://<IP address>:<port>/some/path"
A simple POST web service can be faked using netcat:
Example
The JSON file always contains a valid array. An RPC call looks like this:
Example
C.48 DisableAuditorForProfiles directive
Syntax: DisableAuditorForProfiles = {profile [, profile] ...}
Default: DisableAuditorForProfiles = {NODE}
The DisableAuditorForProfiles directive sets the profile for which an audit log for CMDaemon actions is disabled. A profile (section 2.3.4) is what allows or denies what CMDaemon may do for that profile user. More than one profile can be set as a comma-separated list. New profiles can also be created via the profile mode of cmsh or via the navigation path Identity Management > Profiles > of Base View, thus making it possible to disable auditing for arbitrary groups of CMDaemon services.
C.49 EventLogger directive
Syntax: EventLogger = true|false
Default: EventLogger = true
The EventLogger directive sets whether to log events. If active, then by default it logs events to /var/spool/cmd/events.log on the active head. If a failover takes place, then the event logs on both heads should be checked and merged for a complete list of events.
The location of the event log on the filesystem can be changed using the EventLoggerFile directive (here).
Whether events are logged in files or not, events are cached and accessible using cmsh or Base View. The number of events cached by CMDaemon is determined by the parameter MaxEventHistory (here).
C.50 EventLoggerFile directive
Syntax: EventLoggerFile = filename
Default: EventLogger = "/var/spool/cmd/events.log"
The EventLogger directive sets where the events seen in the event viewer (section 10.10) are logged.
C.51 MaxEventHistory directive
Syntax: AdvancedConfig = {"MaxEventHistory=number", ...}
Default: MaxEventHistory=8192
MaxEventHistory is a parameter of the AdvancedConfig (here) directive.
By default, when not explicitly set, the maximum number of events that is retained by CMDaemon is 8192. Older events are discarded.
The parameter can take a value from 0 to 1000000. However, CMDaemon is less responsive with larger values, so in that case, setting the EventLogger directive (here) to true, to activate logging to a file, is advised instead.
C.52 TimingOverview directive
Syntax: TimingOverview = filename
Default: TimingOverview = true|false
If set to true, the TimingOverview directive records timing information for CMDaemon.
C.53 TimingOverviewFile directive
Syntax: TimingOverviewFile = filename
Default: TimingOverviewFile = "/var/spool/cmd/timing.overview.log"
The TimingOverviewFile directive sets the file where the timing data values for CMDaemon go.
C.54 PublicDNS directive
Syntax: PublicDNS = true|false
Default: PublicDNS = false
By default, internal hosts are resolved only if requests are from the internal network. Setting PublicDNS to true allows the head node name server to resolve internal network hosts for any network, including networks that are on other interfaces on the head node. Separate from this directive, port 53/UDP must also be opened up in Shorewall (section 7.2 of the Installation Manual) if DNS is to be provided for queries from an external network.
C.55 MaximalSearchDomains directive
Syntax: GlobalConfig = {"MaximalSearchDomains = number", ...}
Default: 64
The MaximalSearchDomains directive is a parameter of the GlobalConfig (here) directive. It sets the maximum number of search domains that BCM configures for the cluster, and can be set to any value from 1 to 1024. The default was 6 before NVIDIA Base Command Manager version 10.23.09, 32 from version 10.23.09 onward, and 64 from point release 11.35.0 onward.
On older versions of glibc (before 2.26), only the first 6 configured search domains are used by the operating system, regardless of this directive’s setting. For example, to raise the limit to 100 domains on a cluster running glibc 2.26 or later, the directive setting would be: GlobalConfig = { "MaximalSearchDomains=100" }
C.56 LockDownDhcpd directive
Syntax: LockDownDhcpd = true|false
Default: LockDownDhcpd = false
LockDownDhcpd is a deprecated legacy directive. If set to true, a global DHCP “deny unknown-clients” option is set. This means no new DHCP leases are granted to unknown clients for all networks. Unknown clients are nodes for which BCM has no MAC addresses associated with the node. The directive LockDownDhcpd is deprecated because its globality affects clients on all networks managed by BCM, which is contrary to the general principle of segregating the network activity of networks.
The recommended way now to deny letting new nodes boot up is to set the option for specific networks by using cmsh or Base View (section 3.2.1: figure 3.5 and table 3.1). Setting the cmd.conf LockDownDhcpd directive overrides lockdowndhcpd values set by cmsh or Base View.
C.57 MaxNumberOfProvisioningThreads directive
Syntax: MaxNumberOfProvisioningThreads = number
Default: MaxNumberOfProvisioningThreads = 10000
The MaxNumberOfProvisioningThreads directive specifies the cluster-wide total number of nodes that can be provisioned simultaneously. Individual provisioning servers typically define a much lower bound on the number of nodes that may be provisioned simultaneously.
C.58 SetupBMC directive
Syntax: SetupBMC = true|false
Default: SetupBMC = true
Automatically configure the username and password for the BMC interface of the head node. This may also be valid for regular nodes. The SetupBMC directive should not be confused with the setupBmc field of the node-installer configuration file, described in section 5.8.8.
The node-installer normally takes care of BMC interface configuration on regular nodes by acting on the node-installer configuration file field setupBmc. The CMDaemon directive SetupBMC can only work on regular nodes if the node-installer is not configuring the regular nodes. If the boolean parameters installbootrecord and allownetworkingrestart for a regular node are set to yes, then the SetupBMC directive is able to work for that regular node. Setting the boolean parameters at category level also makes the directive work for the associated nodes.
C.59 BMCSessionTimeout directive
Syntax: BMCSessionTimeout = number
Default: BMCSessionTimeout = 2000
The BMCSessionTimeout specifies the time-out for BMC calls in milliseconds.
C.60 BMCIdentifyScript directive
Syntax: AdvancedConfig = {"BMCIdentify=filename", ...}
Default: unset
BMCIdentifyScript is a parameter of the AdvancedConfig (here) directive.
The parameter takes a full file path to a script that can be used for identification with a BMC (section 3.7.5).
C.61 BMCIdentifyScriptTimeout directive
Syntax: AdvancedConfig = {"BMCIdentifyScriptTimeout=number from 1 to 360", ...}
Default: 60
BMCIdentifyScriptTimeout is a parameter of the AdvancedConfig (here) directive.
CMDaemon waits at the most BMCIdentifyScriptTimeout seconds for the script used by the BMCIdentify directive to complete.
C.62 BMCIdentifyCache directive
Syntax: AdvancedConfig = {"BMCIdentifyCache=0|1", ...}
Default: 1
BMCIdentifyCache is a parameter of the AdvancedConfig (here) directive.
If set to 1, then CMDaemon remembers the last value of the output of the script used by the BMCIdentify directive.
C.63 SnmpSessionTimeout directive
Syntax: SnmpSessionTimeout = number
Default: SnmpSessionTimeout = 500000
The SnmpSessionTimeout specifies the time-out for SNMP calls in microseconds.
C.64 PowerOffPDUOutlet directive
Syntax: PowerOffPDUOutlet = true|false
Default: PowerOffPDUOutlet = false
Enabling the PowerOffPDUOutlet directive allows PDU ports to be powered off for clusters that have both PDU and IPMI power control. Section 4.1.3 has more on this.
C.65 PowerThreadPoolSize directive
Syntax: AdvancedConfig = {"PowerThreadPoolSize= <integer>", ...}
Default: PowerThreadPoolSize=32
PowerThreadPoolSize is a parameter of the AdvancedConfig (here) directive.
The parameter can take positive integer values. Increasing its value increases the number of threads that are used to power up the nodes in a cluster (section 4.2.3), so that the cluster is fully operational quicker. The administrator should however take into account the power surge due to increasing the number of threads (number of subprocesses ) before increasing the value beyond its default.
C.66 DisableBootLogo directive
Syntax: DisableBootLogo = true|false
Default: DisableBootLogo = false
When DisableBootLogo is set to true, the BCM logo is not displayed on the first boot menu when nodes PXE boot.
C.67 StoreBIOSTimeInUTC directive
Syntax: StoreBIOSTimeInUTC = true|false
Default: StoreBIOSTimeInUTC = false
When StoreBIOSTimeInUTC is set to true, the system relies on the time being stored in BIOS as being UTC rather than local time.
C.68 FreezeChangesTo<wlm>Config directives: FreezeChangesToSlurmConfig directive FreezeChangesToLSFConfig directive
Syntax: FreezeChangesTo <wlm>Config= true|false
Default: FreezeChangesTo <wlm>Config = false
When FreezeChangesTo <wlm>Config is set to true, the CMDaemon running on that node does not make any modifications to the workload manager configuration for that node. Workload managers for which this value can be set are:
-
Slurm
-
LSF
Monitoring of jobs, and workload accounting and reporting continues for frozen workload managers.
Upgrades to newer workload manager versions may still require some manual adjustments of the configuration file, typically if a newer version of the workload manager configuration changes the syntax of one of the options in the file.
C.69 FrozenFile directive
Syntax: FrozenFile = { filename[, filename]... }
Example: FrozenFile = { "/etc/dhcpd.conf", "/etc/postfix/main.cf" }
The FrozenFile directive is used to prevent the CMDaemon-maintained sections of configuration files from being automatically generated. This is useful when site-specific modifications to configuration files have to be made.
The filewriteinfo command (section A.2) can be run from device mode to see which files are frozen. The output in the Frozen column then shows which files are protected from CMDaemon-managed changes.
To avoid problems, the file that is frozen should not be a symlink, but should be the ultimate destination file. The readlink -f <symlinkname> command returns the ultimate destination file of a symlink called <symlinkname>. This is also the case for an ultimate destination file that is reached via several chained symlinks.
C.69.1 FrozenFile directive for regular nodes
C.69.1.1 FrozenFile directive for regular nodes for CMDaemon
The FrozenFile directive can be used within the cmd.conf file of the regular node.
Example
To freeze the file /etc/named.conf on the regular nodes running with the image default-image, the file:
/cm/images/default-image/cm/local/apps/cmd/etc/cmd.conf
can have the following directive set in it:
FrozenFile = { "/etc/named.conf" }
The path of the file that is to be frozen on the regular node must be specified relative to the root of the regular node.
The running node should then have its image updated. This can be done with the imageupdate command in cmsh (section 5.6.2), or the Update node button in Base View (section 5.6.3). After the update, CMDaemon should be restarted within that category of nodes:
Example
C.69.1.2 FrozenFile regex specification
The FrozenFile directive allows regexes to be used for a path, if the path begins with the | character:
Example
In the preceding entry, all image directories under /cm/images/ are matched for the path etc/postfix/main.cf.
C.69.1.3 FrozenFile directive for regular nodes for the node-installer
CMDaemon directives only affect files on a regular node after CMDaemon starts up on the node during the init stage. So files frozen by the CMDaemon directive stay unchanged by CMDaemon after this stage, but they may still be changed before this stage.
Freezing files so that they also stay unchanged during the pre-init stage—that is during the node-installer stage—is possible with node-installer directives.
Node-installer freezing is independent of CMDaemon freezing, which means that if a file freeze is needed for the entire startup process as well as beyond, then both a node-installer as well as a CMDaemon freeze are sometimes needed.
Node-installer freezes can be done with the node-installer directives in /cm/node-installer/scripts/node-installer.conf, introduced in section 5.4:
-
frozenFilesPerNode -
frozenFilesPerCategory
For the node-installer.conf file in multidistro and multiarch (section 9.7) configurations, the directory path /cm/node-installer takes the form:
/cm/node-installer-<distribution>-<architecture>
The values for <distribution> and <architecture> can take the values outlined here.
Example
Per node:
Here, the * wildcard means that no restriction is set. Setting node003 means that only node003 is frozen.
Example
Per category:
Here, the nodes in the category mycategory are prevented from being changed by the node-installer.
C.69.1.4 Is A FrozenFile Directive Always Needed?
A FrozenFile directive is not always needed because in some cases a workaround is possible.
In a configuration file after a node is fully up, the effect of a statement earlier on can often be overridden by a statement later in the file. So, the following useful behavior is independent of whether FrozenFile is being used for a configuration file or not: A configuration file, for example /etc/postfix/main.cf, with a configuration statement in an earlier CMDaemon-maintained part of the file, for example:
can often be overridden by a statement later on outside the CMDaemon-maintained part of the file:
Using FrozenFile in CMDaemon or the node-installer can thus sometimes be avoided by the use of such overriding statements later on.
Whether overriding later on is possible depends on the software being configured. It is true for Postfix configuration files, for example, but it may not be so for the configuration files of other applications.
C.70 EaseNetworkValidation directive
Syntax: EaseNetworkValidation = 0|1|2
Default: EaseNetworkValidation = 0
CMDaemon enforces certain requirements on network interfaces and management/node-booting networks by default. In heavily customized setups, such as is common in Type 3 networks (section 3.3.9 of the Installation Manual), the user may wish to disable these requirements.
-
0enforces all requirements. -
1allows violation of the requirements, with validation warnings. This value should never be set except under instructions from BCM support. -
2allows violation of the requirements, without validation warnings. This value should never be set except under instructions from BCM support.
C.71 CustomUpdateConfigFileScript directive
Syntax: CustomUpdateConfigFileScript = filename
Default: commented out in the default cmd.conf file
Whenever one or more entities have changed, the custom script at filename, specified by a full path, is called 30s later. Python bindings can be used to get information on the current setup.
C.72 ConfigDumpPath directive
Syntax: ConfigDumpPath = filename
Default: ConfigDumpPath = /var/spool/cmd/cmdaemon.config.dump
The ConfigDumpPath directive sets a dump file for dumping the configuration used by the power control script /cm/local/apps/cmd/scripts/pctl/pctl. The pctl script is a fallback script to allow power operations if CMDaemon is not running.
-
If no directive is set (
ConfigDumpPath = ""), then no dump is done. -
If a directive is set, then the administrator must match the variable
cmdconfigfilein the powercontrol configuration file/cm/local/apps/cmd/scripts/pctl/config.pyto the value ofConfigDumpPath. By default, the value ofcmdconfigfileis set to/var/spool/cmd/cmdaemon.config.dump.
C.73 SyslogHost directive
Syntax: SyslogHost = hostname
Default: SyslogHost = "localhost"
The SyslogHost directive specifies the hostname of the syslog host.
C.74 SyslogFacility directive
Syntax: SyslogFacility = facility
Default: SyslogFacility = "LOG_LOCAL6"
The default value of LOG_LOCAL6 is set in:
-
/etc/rsyslog.confin RHEL, Ubuntu. -
/etc/syslog-ng/syslog-ng.confin SLES versions
These are the configuration files for the default syslog daemons syslog, rsyslog, and syslog-ng, respectively, that come with the distribution. BCM redirects messages from CMDaemon to /var/log/cmdaemon only for the default syslog daemon that the distribution provides. So, if another syslog daemon other than the default is used, then the administrator has to configure the non-default syslog daemon facilities manually.
The value of facility must be one of: LOG_KERN, LOG_USER, LOG_MAIL, LOG_DAEMON, LOG_AUTH, LOG_SYSLOG or LOG_LOCAL0..7
C.75 NameServerLocalhostLocation directive
Syntax: AdvancedConfig = {"NameServerLocalhostLocation=0|1", ...}
Default: NameServerLocalhostLocation=0
NameServerLocalhostLocation is a parameter of the AdvancedConfig (here) directive.
When set to 1, the location of the localhost as specified by the nameserver directive in /etc/resolv.conf is moved to the bottom of the list of nameserver entries. The default value of 0 places it at the top of those entries.
C.76 ResolveToExternalName directive
Syntax: ResolveToExternalName = true|false
Default: ResolveToExternalName = false
The value of the ResolveToExternalName directive determines under which domain name the primary and secondary head node hostnames are visible from within the head nodes, and to which IP addresses their hostnames are resolved. Enabling this directive resolves the head nodes’ hostnames to the IP addresses of their external interfaces.
Thus, on head nodes and regular nodes in both single-head and failover clusters
-
with
ResolveToExternalNamedisabled, themasterhostname and the actual hostname of the head node (e.g.head1,head2) by default always resolve to the internal IP address of the head node. -
with
ResolveToExternalNameenabled, themasterhostname and the actual hostname of the head node (e.g.head1,head2) by default always resolve to the external IP address of the head node.
The resolution behavior can be summarized by the following table:
The system configuration files on the head nodes that get affected by this directive include /etc/hosts and, on SLES systems, also the /etc/HOSTNAME. Also, the DNS zone configuration files get affected.
Additionally, in both the single-head and failover clusters, using the “hostname -f” command on a head node while ResolveToExternalName is enabled results in the host’s Fully Qualified Domain Name (FQDN) being returned with the host’s external domain name. That is, the domain name of the network that is specified as the “External network” in the base partition in cmsh (the output of “cmsh -c "partition use base; get externalnetwork"”).
Modifying the value of the ResolveToExternalName directive and restarting the CMDaemon while important system services (e.g. Slurm) are running should not be done. Doing so is likely to cause problems with accessing such services due to them then running with a different domain name than the one with which they originally started.
On a tangential note that is closely, but not directly related to the ResolveToExternalName directive: the cluster can be configured so that the “hostname -f” command executed on a regular node returns the FQDN of that node, and so that the FQDN in turn resolves to an external IP for that regular node. The details on how to do this are in the Knowledge Articles of the NVIDIA Enterprise Support Portal at https://enterprise-support.nvidia.com/. A search query for FQDN leads to the relevant entry.
C.77 ResolveMasterToExternalName directive
Syntax: AdvancedConfig = {"ResolveMasterToExternalName=0|1", ...}
Default: ResolveMasterToExternalName = 1
ResolveMasterToExternalName is a parameter of the AdvancedConfig (here) directive.
ResolveMasterToExternalName can only be used if ResolveToExternalName (here) is active.
If set to 1 (the default), then the master head node, as specified by the name master, resolves to the IP address as set by ResolveToExternalName.
If set to 0 then the master head node resolves to the internal shared IP address.
C.78 ResolveMasterToExternalDomainName directive
Syntax: AdvancedConfig = {"ResolveMasterToExternalDomainName=0|1", ...}
Default: ResolveMasterToExternalDomainName = 1
ResolveMasterToExternalDomainName is a parameter of the AdvancedConfig (here) directive.
ResolveMasterToExternalDomainName can only be used if ResolveToExternalName (here) is active.
The external domain name as defined for the external network, in the form master.<external domain>, can be used in /etc/hosts during name resolution.
If set to 1 (the default), then the external domain name is used in /etc/hosts.
If set to 0, then the external domain name is not used in /etc/hosts.
Resolution occurs as shown in the following table:
C.79 DisableLua directive
Syntax: DisableLua = true|false
Default: DisableLua = false
The value of the DisableLua directive determines if Lua code (section L.6) used in monitoring expressions can be executed.
C.80 AdvancedConfig directive
Syntax: AdvancedConfig = { "<key1>=<value1>", "<key2>=<value2>", ... }
Default: Commented out in the default cmd.conf file
The AdvancedConfig directive is not part of the standard directives. It takes a set of key/value pairs as parameters, with each key/value pair allowing a particular functionality, and is quite normal in that respect. However, the functionality of a parameter to this directive is often applicable only under restricted conditions, or for non-standard configurations. The AdvancedConfig parameters are therefore generally not recommended for use by the administrator, nor are they generally documented.
Like for the other directives, only one AdvancedConfig directive line is used. This means that whatever functionality is to be enabled by this directive, its corresponding parameters must be added to that one line. These key/value pairs are therefore added by appending them to any existing AdvancedConfig key/value pairs, which means that the directive line can be a long list of key/value pairs to do with a variety of configurations.
C.80.0.1 Managing Key/Value Pairs With The cm-manipulate-advanced-config.py Utility
The cm-manipulate-advanced-config.py utility can be used to make it easier to manage AdvancedConfig key/value pairs.
For example, to add a key/value pair key8=value8:
To show the current state of the AdvancedConfig, the -s|--show option can be used:
A key/value pair can be removed by specifying its key with the -r|--remove option:
The utility can be used on cmd.conf in node images too, using the -i|--image option.
The -q option causes the utility to exit with code 1 if cmd.conf has changed.
Further options can be seen with the -h|--help option.
C.81 GlobalConfig directive
Syntax: GlobalConfig = { "<key1>=<value1>", "<key2>=<value2>", ... }
Default: not in the default cmd.conf file
The GlobalConfig directive is not part of the standard directives. It takes a set of key/value pairs as parameters, with each key/value pair allowing a particular functionality, and is quite normal in that respect. However, the parameter to this directive only needs to be specified on the head node. The non-head node CMDaemons take this value upon connection, which means that the cmd.conf file on the non-head nodes do not need to have this specified.
This allows nodes to set up, for example, their search domains using the MaximalSearchDomains GlobalConfig directive (here).
Like for the other directives, only one GlobalConfig directive line is used. This means that whatever functionality is to be enabled by this directive, its corresponding parameters must be added to that one line. These key/value pairs are therefore added by appending them to any existing GlobalConfig key/value pairs, which means that the directive line can be a long list of key/value pairs to do with a variety of configurations.
C.82 ScriptEnvironment directive
Syntax: ScriptEnvironment = { "CMD_ENV1=<value1>", "CMD_ENV2=<value2>", ... }
Default: Commented out in the default cmd.conf file
The ScriptEnvironment directive sets extra environment variables for CMDaemon and child processes.
For example, if CMDaemon is running behind a web proxy, then the environment variable http_proxy may need to be set for it. If, for example, the proxy is the host brawndo, and it is accessed via port 8080 using a username/password pair of joe/electrolytes, then the directive becomes:
C.83 BurnSpoolDir directive
Syntax: BurnSpoolDir = path
Default: BurnSpoolDir = "/var/spool/burn/"
The BurnSpoolDir directive specifies the directory under which node burn log files are placed (Chapter 11 of the Installation Manual). The log files are logged under a directory named after the booting MAC address of the NIC of the node. For example, for a MAC address of 00:0c:29:92:55:5e the directory is /var/spool/burn/00-0c-29-92-55-5e.
C.84 IdleThreshold directive
Syntax: IdleThreshold = number
Default: IdleThreshold = 1.0
The IdleThreshold directive sets a threshold value for loadone. If loadone exceeds this value, then data producers that have Only when idle (here) set to true (enabled), will not run. If the data producer is sampled on a regular node rather than on the head node, then cmd.conf on the regular node should be modified and its CMDaemon restarted.
C.85 MonitoringPath directive
Syntax: AdvancedConfig = {"MonitoringPath=path", ...}
Default: Implicit value: "MonitoringPath=/var/spool/cmd/monitoring/"
MonitoringPath is a parameter of the AdvancedConfig (here) directive.
Its value determines the path of the directory in which monitoring data is saved (section 14.8).
C.86 MonitoringEntityActionHistoryDisabled directive
Syntax: AdvancedConfig = {"MonitoringEntityActionHistoryDisabled =0|1"}
Default: 0
MonitoringEntityActionHistoryDisabled is a parameter of the AdvancedConfig (here) directive.
If it is set to a value of 1 then the action history (section 10.6.6) is disabled.
C.87 MaxServiceFailureCount directive
Syntax: AdvancedConfig = {"MaxServiceFailureCount=number", ...}
Default: Implicit value: "MaxServiceFailureCount=10"
MaxServiceFailureCount is a parameter of the AdvancedConfig (here) directive.
Its value determines the number of times a service failure event is logged (here). Restart attempts on the service still continue when this number is exceeded.
C.88 InitdScriptTimeout directive
Syntax: AdvancedConfig = {"InitdScriptTimeout[.service]=timeout", ...}
Default: Implicit value: "InitdScriptTimeout=30"
InitdScriptTimeout is a parameter of the AdvancedConfig (here) directive. It can be set globally or locally:
-
Global (all services)
InitdScriptTimeoutcan be set as a global value for init scripts, by assigning timeout as a period in seconds. If an init script fails to start up its service within this period, then CMDaemon kills the service and attempts to restart it.-
If
InitdScriptTimeouthas a value for timeout set, then all init scripts have a default timeout of timeout seconds. -
If
InitdScriptTimeouthas no timeout value set, then all init scripts have a default timeout of 30 seconds.
-
-
Local (for a specific service)
IfInitdScriptTimeout.service is assigned a timeout value, then the init script for that service times out in timeout seconds. This timeout overrides, for that service only, any existing global default timeout.
When a timeout happens for an init script attempting to start a service, the event is logged. If the number of restart attempts exceeds the value determined by the MaxServiceFailureCount directive (here), then the event is no longer logged, but the restart attempts continue.
Example
An fhgfs startup takes a bit longer than 30 seconds, and therefore times out with the default timeout value of 30s. This results in the following logs in /var/log/cmdaemon:
Here, service is fhgfs-client, so setting the parameter can be done with:
This allows a more generous timeout of 60 seconds instead.
Restarting CMDaemon then should allow the fhgs startup to complete
A more refined approach that avoids a complete CMDaemon restart would be to execute a reset (here) on the fhgfs-client from within CMDaemon, as follows:
C.89 CMDaemonListenOnInterfaces directive
Syntax: AdvancedConfig = {"CMDaemonListenOnInterfaces= <interfaces>", ...}
Default: all interfaces listening to port 8081
CMDaemonListenOnInterfaces is a parameter of the AdvancedConfig (here) directive.
When set explicitly, CMDaemon listens only to the interfaces listed in <interfaces>. The form of <interfaces> is a comma-separated list of interface device names:
Example
If the interface list item lo is omitted from the list of names, it will still be listened to. This is because CMDaemon must always be able to talk to itself on the loopback interface.
C.90 DisableInotifyInterface directive
Syntax: AdvancedConfig = {"DisableInotifyInterface =0|1"}
Default: 0
DisableInotifyInterface is a parameter of the AdvancedConfig (here) directive.
When set to 1 CMDaemon ignores inotify events from all interfaces.
When set to 0 CMDaemon does not ignore inotify events from all interfaces.
C.91 IgnoreInotifyInterface directive
Syntax: AdvancedConfig = {"IgnoreInotifyInterface = <interfaces>", ...}
Syntax: GlobalConfig = {"IgnoreInotifyInterface = <interfaces>", ...}
Default: veth*,lxc*,fg-*,qg-*,qr-*,sg-*,tbr-*,qbr*,qvb*,qvo*,cali*,flannel*,ecu_*,chassis_*, enx*,az*,tap*,rdma*,cni0,/[a-f0-9]12``_[hc]/
In the preceding lines showing the value for Default, the line ending with chassis_*, must be concatenated to the line starting with enx*,
IgnoreInotifyInterface is a parameter of the AdvancedConfig (here) directive, as well as the GlobalConfig (here) directive.
IgnoreInotifyInterface is active if DisableInotifyInterface is 0.
By default, the directive has CMDaemon ignore the normally harmless messages that are typically generated by Kubernetes as Calico creates and removes interfaces.
CMDaemon ignores events from the interfaces listed in <interfaces>. The form of <interfaces> is a comma-separated list of interface device names, with some globbing allowed.
C.92 CookieCooldownTime directive
Syntax: AdvancedConfig = {"CookieCooldownTime=number from 60 to 86400", ...}
Default: 900
CookieCooldownTime is a parameter of the AdvancedConfig (here) directive.
It defines the number of seconds until the Base View connection to CMDaemon times out, if there is no user activity at the Base View client side.
C.93 DHCPMaxleaseTime directive
Syntax: AdvancedConfig = {"DHCPMaxleaseTime=number", ...}
Default: client default
DHCPMaxleaseTime is a parameter of the AdvancedConfig (here) directive.
DHCPMaxleaseTime sets max-lease-time in DHCPOFFER. This is the maximum lease time, in seconds, that the DHCP server on the head node allows to the DHCP client on the node.
C.94 SlurmStraightExtraTopology directive
Syntax: AdvancedConfig = {"SlurmStraightExtraTopology=0|1", ...}
Default: 1
SlurmStraightExtraTopology is a parameter of the AdvancedConfig (here) directive. If set to 0, then the order of switches is reversed when setting extra values for SlurmTopology (here).
C.95 SlurmConcatTopologySwitchName directive
Syntax: AdvancedConfig = {"SlurmConcatTopologySwitchName=0|1", ...}
Default: 0
SlurmConcatTopologySwitchName is a parameter of the AdvancedConfig (here) directive. If set to 1, then it allows concatenation of the switch names in the topology defined via the SlurmTopology extra values (here).
C.96 JobsSamplingMetricsInterval directive
Syntax: AdvancedConfig = {"JobsSamplingMetricsInterval= <number>", ...}
Default: 60
JobsSamplingMetricsInterval is a parameter of the AdvancedConfig (here) directive. Its value is a time period, in seconds, and it applies only to metrics associated with job queues. Such metric sampling is carried out with this time period if job queues are added, or if job queues are re-created after disappearing.
C.97 MembershipQueryInterval directive
Syntax: AdvancedConfig = {"MembershipQueryInterval= <number>", ...}
Default: 4
MembershipQueryInterval is a parameter of the AdvancedConfig (here) directive. Its value is a time period, in seconds. This time period value elapses between the checks that CMDaemon makes to determine the node states (section 5.5) in a cluster. If the network is very congested, then a larger value can be used to reduce the network load caused by these checks.
C.98 AddUserPasswordReset directive
Syntax: AdvancedConfig = {"AddUserPasswordReset=0|1", ...}
Default: 0
AddUserPasswordReset is a parameter of the AdvancedConfig (here) directive. If this parameter is set to 1, then an LDAP user added after the change is in effect is forced to change their password on first login:
Example
With the AdvancedConfig configuration in effect, an already-existing user can also be forced to change password on the next login, after the extra variable pwdReset is set to true for that already-existing user. For example, for a user fred, executing the following:
Example
means that fred is forced to change password on the next login.
C.99 AddUserScript directive
Syntax: AdvancedConfig = {"AddUserScript= <path>", ...}
Default: none
AddUserScript is a parameter of the AdvancedConfig (here) directive. If this parameter is set to a path leading to a script, and if a new user is added using cmsh or Base View, then the script is automatically run by CMDaemon, with the username of the new user automatically passed as the first argument to the script. The script has a default timeout of 5 seconds.
C.100 AddUserScriptPasswordInEnvironment directive
Syntax: AdvancedConfig = {"AddUserScriptPasswordInEnvironment=0|1", ...}
Default: 0
AddUserScriptPasswordInEnvironment is a parameter of the AdvancedConfig (here) directive. If this parameter is set to 1, then CMDaemon passes the CMD_USER_PASSWORD environment variable to the script defined by the AddUserScript directive.
C.101 RemoveUserScript directive
Syntax: AdvancedConfig = {"RemoveUserScript= <path>", ...}
Default: none
RemoveUserScript is a parameter of the AdvancedConfig (here) directive. If this parameter is set to a path leading to a script, and if an existing user is removed using cmsh or Base View, then the script is automatically run by CMDaemon. The script has a default timeout of 5 seconds.
C.102 AddUserScriptTimeout directive
Syntax: AdvancedConfig = {"AddUserScriptTimeout= <number>", ...}
Default: 5
AddUserScriptTimeout is a parameter of the AdvancedConfig (here) directive. It sets the timeout value in seconds, for the script set by AddUserScript.
C.103 RemoveUserScriptTimeout directive
Syntax: AdvancedConfig = {"RemoveUserScriptTimeout= <number>", ...}
Default: 5
RemoveUserScriptTimeout is a parameter of the AdvancedConfig (here) directive. It sets the timeout value in seconds, for the script set by RemoveUserScript.
C.104 AutomaticMountAll directive
Syntax: AdvancedConfig = {"AutomaticMountAll=0|1", ...}
Default: 1
The AutomaticMountAll directive is a parameter of the AdvancedConfig (here) directive. If set to the default of 1, then a mount -a operation is carried out when a mount change is carried out by CMDaemon.
The mount -a operation has to do with attempting to mount devices listed in /etc/fstab. It should not be confused with auto-mounting of filesystems, which has to do with mounting an arbitrary device to a filesystem automatically.
If the AutomaticMountAll directive is set to 0, then /etc/fstab is written, but the mount -a command is not run by CMDaemon. However, the administrator should be aware that since mount -a is run by the distribution during booting, a node reboot implements the mount change.
The directive is only set in the cmd.conf file of the compute nodes, and not the head node(s).
C.105 AllowImageUpdateWithAutoMount directive
Syntax: AdvancedConfig = {"AllowImageUpdateWithAutoMount=0|1|2|3", ...}
Default: 0
The AllowImageUpdateWithAutoMount directive is a parameter of the AdvancedConfig (here) directive. The values it takes decide how an auto-mounted filesystem should be dealt with during image updates (section 5.6.2) or grab to image (syncing node-to-image, section 9.5.2). It must be set in cmd.conf per software image or per node where the automount is running, and the changes activated as usual, as described here.
How an auto-mounted filesystem can be configured using the autofs service in BCM is discussed in section 3.14. The need for adding the automounted filesystem to excludelistupdate is discussed here.
C.106 DNS::options_allow-query directive
Syntax: AdvancedConfig = {"DNS::options_allow-query= <subnet1>,<subnet2>,…”,…}
Default: unset
The DNS::options_allow-query directive is a parameter of the AdvancedConfig (here) directive. If a subnet value is specified, in CIDR notation, then that subnet can query the DNS running on the head node. Setting a subnet places an entry within the allow-query section of /etc/named.conf. Extra subnets must specified with comma-separation, without whitespace.
The standard directive PublicDNS (here) simply adds the entry 0.0.0.0/0 to the allow-query section and can be used if no specific subnet needs to be added.
C.107 CipherList directive
Syntax: AdvancedConfig = {"CipherList= <ciphers>", ...}
Default: CipherList=ALL:!aNULL:!eNULL
The CipherList directive is a parameter of the AdvancedConfig (here) directive. It sets the cipher list of the OpenSSL suite that CMDaemon negotiates with clients. Ciphers in the cipher list can be viewed with:
Example
The columns are: the cipher name, SSL protocol, key exchange used, authentication mechanism, encryption used, and MAC digest algorithm.
Ciphers can be specified for the CipherList directive according to the specification described in man ciphers.1. For example, as:
Example
An analogous non-CMDaemon directive is found in LDAP client-server negotiation, where TLSCipherSuite can be set in the slapd.conf file, described in man slapd.conf.5.
C.108 SSLServerMethod directive (with TLS masking)
Syntax: AdvancedConfig = {"SSLServerMethod=TLS <versionnumber>", ...}
Syntax: GlobalConfig = {"SSLServerMethod=TLS <versionnumber>", ...}
Masking syntax: AdvancedConfig = {"TLS <versionnumber>=<boolean>”, …}
Masking syntax: GlobalConfig = {"TLS <versionnumber>=<boolean>”, …}
Default: GlobalConfig = { "TLS 1.2=1", "TLS 1.3=1", ... }
The SSLServerMethod directive is a parameter of the AdvancedConfig (here) directive, as well as the GlobalConfig (here) directive. It sets the SSL server method for the OpenSSL suite that CMDaemon negotiates with clients. Possible values for <versionnumber> are:
-
1 -
1.1 -
1.2 -
1.3
By default, with no SSLServerMethod directive, TLS versions 1.2 and 1.3 are enabled.
This is because there are still some SSL clients that require a TLS 1.2 cipher and have no TLS 1.3 cipher negotiation ability.
If the SSLServerMethod directive is specified, then only that TLS version is negotiated. For example, to allow only TLS 1.2 negotiation:
Example
The SSLServerMethod directive is implied by the TLS masking syntax. The TLS masking syntax allows TLS negotiation methods to be set concurrently. For example, TLS 1 and TLS1.1 could be disabled, and TLS 1.2 and TLS 1.3 could be enabled, with:
Example
The GlobalConfig value is the directive that the cluster administrator should set in most cases. It only needs to be set on cmd.conf on the head node (or head nodes in a high availability configuration), in order to configure the same TLS negotiation settings for the regular nodes.
The GlobalConfig value is overridden by the AdvancedConfig value. The AdvancedConfig value needs to be set on cmd.conf on the head node(s), but also needs to be set on the non-head nodes that are to be disabled. Setting the AdvancedConfig directive allows custom configuration of the TLS negotiations to be carried out per node, and if the negotiations do not match with the other end, then other settings are tried. This does mean that sometimes nodes with the extra settings end up running extra processes for no reason. Running the directive as an AdvancedConfig is therefore suboptimal for most reasonable use cases.
Reverting to the TLS 1.1 fallback cipher availability of earlier versions of CMDaemon is possible by setting:
AdvancedConfig = { "TLS 1.1=1", "TLS 1.2=1", "TLS 1.3=1" }
Setting this brings with it the risk of allowing cryptographic downgrade attacks. It is therefore not recommended, and is also why it is disabled by default.
C.109 JobInformationDisabled directive
Syntax: AdvancedConfig = {"JobInformationDisabled=0|1", ...}
Syntax: GlobalConfig = {"JobInformationDisabled=0|1", ...}
Default: JobInformationDisabled=0
The JobInformationDisabled directive is a parameter of the AdvancedConfig (here) directive, as well as the GlobalConfig (here) directive. If set to 1 it disables job-based monitoring (Chapter 11). The details of how this is done are discussed shortly.
For a cluster that is running millions of jobs at a time, job-based monitoring can typically consume significant resources. The monitoring of data for so many small jobs is typically not useful and not of interest. For such a case, setting this directive improves cluster performance by not having to deal with the information on the current jobs running on the cluster.
The GlobalConfig value is the directive that the cluster administrator should set in most cases. It only needs to be set on cmd.conf on the head node (or head nodes in a high availability configuration). Setting it disables the collection of job information by the nodes running the jobs, and no collection of the job information from those nodes is done by the head (monitoring) node.
The GlobalConfig value is overridden by the AdvancedConfig value. The AdvancedConfig value needs to be set on cmd.conf on the head node(s), but also needs to be set on the non-head nodes that are to be disabled. Setting the AdvancedConfig directive still allows job information to be collected by the nodes running the jobs, but the information is not collected from the nodes by the head (monitoring) node. This means that nodes are running extra processes for probably no reason. Running the directive as an AdvancedConfig is therefore suboptimal for most use cases.
C.110 JobInformationKeepDuration directive
Syntax: AdvancedConfig = {"JobInformationKeepDuration= <number>", ...}
Default: JobInformationKeepDuration=2419200
The JobInformationKeepDuration directive is a parameter of the AdvancedConfig (here) directive. It takes on a value in seconds. If a job has finished more than that many seconds ago, then it will be removed along with all its monitoring data from the database. By default it is set to 28 days (24×3600×28 seconds).
If persistent is set to yes (cmsh:wlm > jobs), then job information is not removed even after JobInformationKeepDuration has been exceeded.
C.111 JobInformationChargeBackKeepDuration directive
Syntax: AdvancedConfig = {"JobInformationChargeBackKeepDuration= <number>", ...}
Default: JobInformationChargeBackKeepDuration=158112000
The JobInformationChargeBackKeepDuration directive is a parameter of the AdvancedConfig (here) directive. It takes on a value in seconds. If a job has finished more than that many seconds ago, then it will be removed along with all its monitoring data from the database. By default it is set to a little more than 5 years (24×3600×366×5 seconds).
C.112 JobInformationKeepCount directive
Syntax: AdvancedConfig = {"JobInformationKeepCount= <number>", ...}
Default: JobInformationKeepCount=8192
The JobInformationKeepCount directive is a parameter of the AdvancedConfig (here) directive. If the total number of jobs is greater than (JobInformationKeepCount + 10%), then the job record and its data content are discarded, starting from the oldest job first, until the total number of jobs remaining becomes JobInformationKeepCount. If it is set to 0, then none of the job records and content are removed.
The maximum value for this directive is 65536. A larger value is treated as 65536.
C.113 JobInformationChargeBackKeepCount directive
Syntax: AdvancedConfig = {"JobInformationChargeBackKeepCount= <number>", ...}
Default: JobInformationChargeBackKeepCount=1048576
The JobInformationChargeBackKeepCount directive is a parameter of the AdvancedConfig (here) directive. If the total number of jobs is greater than (JobInformationChargeBackKeepCount + 10%), then the job record and its data content are discarded, starting from the oldest job first, until the total number of jobs remaining becomes JobInformationChargeBackKeepCount. If it is set to 0, then none of the job records and content are removed.
The default value of about 1 million corresponds to about 0.5 GB of storage.
The maximum value for this directive is about 1 billion (1,073,741,824), which corresponds to about 500 GB of storage.
C.114 JobInformationMinimalJobDuration directive
Syntax: AdvancedConfig = {"JobInformationMinimalJobDuration= <number>", ...}
Default: JobInformationMinimalJobDuration=0
The JobInformationMinimalJobDuration directive is a parameter of the AdvancedConfig (here) directive. If set, then jobs that run for less than this number of seconds are not stored in the cache. Its default value of 0 seconds means that all jobs are handled.
C.115 JobInformationFlushInterval directive
Syntax: AdvancedConfig = {"JobInformationFlushInterval= <number>", ...}
Default: JobInformationFlushInterval=600
The JobInformationFlushInterval directive is a parameter of the AdvancedConfig (here) directive. If this interval, in seconds, is set, then the cache is flushed to the database with that interval. Its default value is 10 minutes (10×60 seconds). Values of around 30 seconds or less will conflict with the default CMDaemon maintenance timeout value of 30 seconds, and will mostly simply add load.
C.116 JobInformationChargeBackRemoveInterval directive
Syntax: AdvancedConfig = {"JobInformationChargeBackRemoveInterval= <number>", ...}
Default: JobInformationChargeBackRemoveInterval=600
The JobInformationChargeBackRemoveInterval directive is a parameter of the AdvancedConfig (here) directive. If this interval, in seconds, is set, then the cache is flushed to the database with that interval. Its default value is 10 minutes (10×60 seconds). Values of around 30 seconds or less will conflict with the default CMDaemon maintenance timeout value of 30 seconds, and will mostly simply add load.
The maximum remove interval is 1 day (86400 seconds).
C.117 ActionLoggerFile directive
Syntax: AdvancedConfig = {"ActionLoggerFile=filename", ...}
Default: /var/spool/cmd/actions.log
The directive is a parameter of the AdvancedConfig (here) directive. Its value overrides the default path.
The directive needs to be implemented per node or image.
C.118 ActionLoggerOnSuccess directive
Syntax: AdvancedConfig = {"ActionLoggerOnSuccess=0|1", ...}
Default: ActionLoggerOnSuccess=0
The directive is a parameter of the AdvancedConfig (here) directive.
By default, only failed actions are logged. Successful actions are also logged if setting ActionLoggerOnSuccess=1
The directive needs to be implemented per node or image.
The log file shows timestamped output with one line per run for an action script, with the script response.
Example
The success line only appears if ActionLoggerOnSuccess=1.
C.119 FailoverPowerRetries directive
Syntax: AdvancedConfig = {"FailoverPowerRetries= <number>", ...}
Default: FailoverPowerRetries=5
The FailoverPowerRetries directive is a parameter of the AdvancedConfig (here) directive.
After a decision to carry out the failover has been made, CMDaemon sends a power off command to the BMC of the head node that is meant to be powered off. If the power off command fails, then on getting the fail response, CMDaemon waits for 1 second. After that second, it sends out a power off command again. The value of FailoverPowerRetries is the number of times that CMDaemon retries sending the power off command to the BMC of the head node that is intended to be powered off during the failover, if the response from the power off command remains a fail response.
The power down attempts cease, either when the BMC reports that the head node is OFF, or when the number of attempts reaches the FailoverPowerRetries value.
FailoverPowerRetries takes a maximum value of 120.
A value of 0 means that 1 attempt to power off is carried out during failover, but no retry is attempted after the first attempt.
Because CMDaemon waits for a period of 1s before checking for an OFF reponse, it means that the number of retries is about the same as the number of seconds before CMDaemon decides that powering off has failed, unless the OFF response also takes some time to get to CMDaemon.
Increasing the value for this directive can be useful for some BMC cards that take longer than about 5s to report their power status, because the power off attempt may otherwise time out.
C.120 AddUserDefaultGroupID directive
Syntax: AdvancedConfig = { "AddUserDefaultGroupID = <number>", ...}
Default: none
The AddUserDefaultGroupID directive is a parameter of the AdvancedConfig (here) directive.
If the AddUserDefaultGroupID is unset, then the default group ID of a new user is the same as the User ID.
If the AddUserDefaultGroupID is set, then the set value becomes the default GUID of new users, when new users are created via the cmsh or Base View front ends to CMDaemon.
The directive is not intended to set a non-default group occasionally during user creation. In that case, a non-default group ID can be set from the cmsh or Base View front ends, by setting the groupid value for the user.
To set a default group for a directory, while retaining the default group for the user, it may be possible to use the setgid bit for a directory.
C.121 LowUserId directive
Syntax: AdvancedConfig = {"LowUserId= <number>", ...}
Default: 1000 (500 on SLES)
The LowUserId directive is a parameter of the AdvancedConfig (here) directive. From BCM 11.34 onwards, it controls the starting point for auto-allocated UIDs and GIDs. The default is 1000 (500 on SLES), so behavior is unchanged for clusters that have not set this parameter explicitly.
Administrators who need UIDs and GIDs to start at a different value can set LowUserId in cmd.conf:
Example
C.122 MaxMeasurablesPerProducer directive
Syntax: AdvancedConfig = { "MaxMeasurablesPerProducer = <number>", ...}
Default: 500
The MaxMeasurablesPerProducer directive is a parameter of the AdvancedConfig (here) directive.
By default there is a software limit of 500 measurables per data producer. If this limit is exceeded, then the CMDaemon monitoring info logs show complaints about “too many measurables”.
In that case, if the cluster hardware is not too slow, and if the measurables produced are not some kind of hardware garbage data values, then increasing the value of this directive should allow more measurables to be dealt with.
C.123 HeadNodeCertificateNameAlternatives directive
Syntax: AdvancedConfig = {"HeadNodeCertificateNameAlternatives=<name,…>",...}
Default: none
The HeadNodeCertificateNameAlternatives directive is a key of the AdvancedConfig (here) directive. It takes on a comma-separated list of alternative DNS names as values to add to the SSL certificate that it generates for CMDaemon. Extra names must specified with comma-separation, without whitespace.
The old certificates must be removed before the new ones are used.
Example