Generated Files

This appendix contains lists of system configuration files that are managed by CMDaemon, and system configuration files that are managed by node-installer. These are files created or modified on the head nodes (section A.1), and on the regular nodes (sections A.2.3 and A.2.3). These files should not be confused with configuration files that are merely installed (section A.3).

Section 2.6.5 describes how system configuration files on all nodes are written out using the Cluster Management Daemon (CMDaemon). CMDaemon is introduced in section 2.6.5 and its configuration directives are listed in Appendix C.

All of these configuration files may be listed as Frozen Files in the CMDaemon configuration file to prevent them from being modified further by CMDaemon. The files can be frozen for the head node by setting the directive at /cm/local/apps/cmd/etc/cmd.conf. They can also be frozen on the regular nodes by setting the directive in the software image, by default at /cm/images/default-image/cm/local/apps/cmd/etc/cmd.conf.

A list of CMDaemon- and node-installer-managed files for nodes can be seen by running the command filewriteinfo in device mode. The command has options to run it for node groupings, as well useful path and sort options.

A.1 System Configuration Files Created Or Modified By CMDaemon On Head Nodes

If the filewriteinfo command is run for the head node, then it displays the files that have been written by CMDaemon on the head node. Running the filewriteinfo command for the head node might display the following on a fresh installation:

[basecm10->device]% filewriteinfo basecm10
Hostname   Path                                                               Timestamp   Actor Frozen
--------  ------------------------------------------------------------------  ----------  ----- ------
basecm10  /cm/images/default-image/etc/mkinitrd_cm.conf                       Mon Oct...  cmd   no
basecm10  /cm/images/default-image/etc/modprobe.d/bright-cmdaemon.conf        Mon Oct...  cmd   no
basecm10  /cm/images/default-image/etc/securetty                              Mon Oct...  cmd   no
basecm10  /etc/chrony.conf                                                    Mon Oct...  cmd   no
basecm10  /etc/dhcp/dhclient.conf                                             Mon Oct...  cmd   no
basecm10  /etc/dhcpd.conf                                                     Mon Oct...  cmd   no
basecm10  /etc/exports                                                        Mon Oct...  cmd   no
basecm10  /etc/genders                                                        Mon Oct...  cmd   no
basecm10  /etc/named.conf                                                     Mon Oct...  cmd   no
basecm10  /etc/postfix/canonical                                              Mon Oct...  cmd   no
basecm10  /etc/postfix/generic                                                Mon Oct...  cmd   no
basecm10  /etc/postfix/main.cf                                                Mon Oct...  cmd   no
basecm10  /etc/resolv.conf                                                    Mon Oct...  cmd   no
basecm10  /etc/security/pam_bright.d/cm-check-alloc.conf                      Mon Oct...  cmd   no
basecm10  /etc/shorewall.staging/interfaces                                   Mon Oct...  cmd   no
basecm10  /etc/shorewall.staging/netmap                                       Mon Oct...  cmd   no
basecm10  /etc/shorewall.staging/policy                                       Mon Oct...  cmd   no
basecm10  /etc/shorewall.staging/snat                                         Mon Oct...  cmd   no
basecm10  /etc/shorewall.staging/zones                                        Mon Oct...  cmd   no
basecm10  /etc/sysconfig/network-scripts/ifcfg-eth1                           Mon Oct...  cmd   no
basecm10  /tftpboot/images/default-image//boot                                Mon Oct...  cmd   no
basecm10  /tftpboot/images/default-image/initrd                               Mon Oct...  cmd   no
basecm10  /tftpboot/images/default-image/vmlinuz                              Mon Oct...  cmd   no
basecm10  /tftpboot/mtu.conf                                                  Mon Oct...  cmd   no
basecm10  /tftpboot/pxelinux.cfg/category.default                             Mon Oct...  cmd   no
basecm10  /var/run/cmd.url                                                    Mon Oct...  cmd   no
basecm10  /var/spool/cmd/my.cmd.url                                           Mon Oct...  cmd   no
basecm10  /var/spool/cmd/my.master.cmd.url                                    Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-chrony.conf                   Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-dhcp-dhclient.conf            Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-dhcpd.conf                    Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-exports                       Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-named.conf                    Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-postfix-canonical             Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-postfix-generic               Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-postfix-main.cf               Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-resolv.conf                   Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-shorewall.staging-interfaces  Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-shorewall.staging-netmap      Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-shorewall.staging-policy      Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-shorewall.staging-snat        Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-shorewall.staging-zones       Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/etc-sysconfig-network-scripts-
                                                                 ifcfg-eth1   Mon Oct...  cmd   no
basecm10  /var/spool/cmd/saved-config-files/tftpboot-mtu.conf                 Mon Oct...  cmd   no

The filewriteinfo command is covered in more detail in section A.2.

The more important head node files that are managed by CMDaemon are listed here for a plain installation on the various distributions.

Some of the more important files managed automatically on the head node by CMDaemon In RHEL and derivatives, Ubuntu 20.04, 22.04, 24.04, SLES15

¹ User-added zone files ending in *.zone that are placed for a corresponding zone statement in the

include file /etc/[bind/]named.conf.include are wiped by CMDaemon activity. Another pattern,

eg: *.myzone, must therefore be used instead

² For Ubuntu, the zone files are under /etc/bind/

A.2 System Configuration Files Created Or Modified Directly On The Node

Files that are created or modified by CMDaemon or the node-installer directly on the node by CMDaemon can be seen with the filewriteinfo command.

For example, for a particular node, in this case node002 (some output elided):

Example

[basecm10->device]% filewriteinfo node002
Hostname    Path                                    Timestamp                 Actor           Frozen
----------- --------------------------------------- ------------------------- --------------- ------
node002     /certificates/fa-16-3e-54-bb-a1/cert    Thu Jul 14 12:16:00 2022  node-installer  no
...
node002     /etc/chrony.conf                        Thu Jul 14 12:17:08 2022  node-installer  no
...
node002     /etc/exports                            Thu Jul 14 12:17:58 2022  cmd             no
...

In the preceding, /etc/exports is seen to have been changed by cmd, which is listed under the Actor column.

A.2.1 Options To filewriteinfo

Options to filewriteinfo can be viewed with the help filewwriteinfo command.

Some of the options are:

• -n|--node: This can be used to specify a node list (here).

• --field: The --field option can be used with, for example, the mac field, as follows (some output elided):

  Example

  [basecm10->device]% get node002 mac
  FA:16:3E:54:BB:A1
  [basecm10->device]% filewriteinfo --field mac=FA:16:3E:54:BB:A1
  Hostname    Path                                    Timestamp                 Actor           Frozen
  ----------- --------------------------------------- ------------------------- --------------- ------
  node002     /certificates/fa-16-3e-54-bb-a1/cert    Thu Jul 14 12:16:00 2022  node-installer  no
  ...
  node002     /etc/chrony.conf                        Thu Jul 14 12:17:08 2022  node-installer  no
  ...
  node002     /etc/exports                            Thu Jul 14 12:17:58 2022  cmd             no
  ...

  The preceding example has the same output as for the session with filewriteinfo node002 here.

  • A pitfall to avoid is the following: The columns headers of the output of the filewriteinfo command, such as Actor, are not keys for the --field option.

    Example

    [basecm10->device]% filewriteinfo node002 --field Actor=cmd         #this is incorrect

    Instead, as is the norm for a device mode command, it is the fields of the mode that are the key=value pairs (for keys that are valid for the device concerned). The fields of the device mode are displayed when the format command (here) is run.

• --sort: This sorts the filewriteinfo output according to column headers (Actor, Timestamp and so on).

• --path: This can be used to specify a file.

  Example

  [basecm10->device]% filewriteinfo  --path /etc/chrony.conf --sort actor
  Hostname         Path              Timestamp                 Actor           Frozen
  ---------------- ----------------- ------------------------- --------------- ------------
  basecm10         /etc/chrony.conf  Thu Jul 14 12:14:25 2022  cmd             no
  node001          /etc/chrony.conf  Thu Jul 14 12:17:08 2022  node-installer  no
  node002          /etc/chrony.conf  Thu Jul 14 12:17:08 2022  node-installer  no

A.2.2 Files Created On Regular Nodes By CMDaemon

Files on a regular node that are modified by CMDaemon can be seen in the output of filewriteinfo. The filewriteinfo command displays a list of the last files written by the cluster manager.

For a default installation, the list is as shown in the following table:

System configuration files created or modified on a default regular nodes image by CMDaemon

A.2.3 Files Created On Regular Nodes By The Node-Installer

The list of files on a regular node that are modified by the node-installer during the last installation session can be viewed in the logs at /var/log/modified-by-node-installer.log on the node, and in the output of filewriteinfo. For a default installation, these are as shown in the following table:

System configuration files created or modified on regular nodes by the node-installer in RHEL8, RHEL9 and derivatives

A.3 Files Not Generated, But Installed In RHEL And Derivatives

This appendix (Appendix A) is mainly about generated configuration files. This section (A.3) of the appendix discusses a class of files that is not generated, but may still be confused with generated files. The discussion in this section clarifies the issue, and explains how to check if non-generated installed files differ from the standard distribution installation.

A design goal of BCM is that of minimal interference. That is, to stay out of the way of the distributions that it works with as much as is reasonable. Still, there are inevitably cluster manager configuration files that are not generated, but installed from a cluster manager package. A cluster manager configuration file of this kind overwrites the distribution configuration file with its own special settings to get the cluster running, and the file is then not maintained by the node-installer or CMDaemon. Such files are therefore not listed on any of the tables in this chapter.

Sometimes the cluster file version may differ unexpectedly from the distribution version. To look into this, the following steps may be followed:

Is the configuration file a BCM version or a distribution version? A convenient way to check if a particular file is a cluster file version is to grep for it in the packages list for the cluster packages. For example, for nsswitch.conf:

[root@basecm10 ~]# repoquery -l $(repoquery -a | grep -F _cm10.0) | grep nsswitch.conf$

The inner repoquery displays a list of all the packages. By grepping for the cluster manager version string, for example _cm10.0 for NVIDIA Base Command Manager 10, the list of cluster manager packages is found. The outer repoquery displays the list of files within each package in the list of cluster manager packages. By grepping for nsswitch.conf$, any file paths ending in nsswitch.conf in the cluster manager packages are displayed. The output is:

/cm/conf/etc/nsswitch.conf

Files under /cm/conf are placed by BCM packages when updating the head node. From there they are copied over during the post-install section of the RPM to where the distribution version configuration files are located by the cluster manager, but only during the initial installation of the cluster. The distribution version file is overwritten in this way to prevent RPM dependency conflicts of the BCM version with the distribution version. The configuration files are not copied over from /cm/conf during subsequent reboots after the initial installation. The cm/conf files are however updated when BCM packages are updated. During such a BCM update, a notification is displayed that new configuration files are available.

Inverting the cluster manager version string match displays the files not provided by BCM. These are normally the files provided by the distribution:

[root@basecm10 ~]# repoquery -l $(repoquery -a | grep -F -v _cm10.0) | grep nsswitch.conf$
...
/usr/share/factory/etc/nsswitch.conf
/usr/share/factory/etc/nsswitch.conf
/etc/authselect/nsswitch.conf
/etc/authselect/user-nsswitch.conf
/usr/share/authselect/default/minimal/nsswitch.conf
/usr/share/authselect/default/sssd/nsswitch.conf
/usr/share/authselect/default/winbind/nsswitch.conf
/var/lib/authselect/nsswitch.conf
/etc/authselect/nsswitch.conf
/etc/authselect/user-nsswitch.conf
/usr/share/authselect/default/minimal/nsswitch.conf
/usr/share/authselect/default/sssd/nsswitch.conf
/usr/share/authselect/default/winbind/nsswitch.conf
/var/lib/authselect/nsswitch.conf
/etc/nsswitch.conf
/etc/nsswitch.conf
/usr/share/authselect/vendor/libnss-mysql/nsswitch.conf
/usr/share/rear/skel/default/etc/nsswitch.conf

Which package provides the file in BCM and in the distribution? The packages that provide these files can be found by running the “yum whatprovides *” command on the paths given by the preceding output, for example:

~# yum whatprovides */cm/conf/etc/nsswitch.conf
...
cm-config-ldap-client<various types and versions are seen>

This reveals that some BCM LDAP packages can provide an nsswitch.conf file. The file is a plain file provided by the unpacking and placement that takes place when the package is installed. The file is not generated or maintained periodically after placement, which is the reason why this file is not seen in the tables of sections A.1 and A.2.3 of this appendix.

Similarly, looking through the output for the less specific case:

~# yum whatprovides */etc/nsswitch.conf
...
cm-config-ldap-client*
glibc*
rear*
systemd*

shows that glibc provides a distribution version of the nsswitch.conf file, that the rear package from the distribution provides a version of it, and that there is also a systemd version of this file available from the distribution packages. The glob * in the output in this manual represents a variety of types and versions. The actual display that is seen on the screen is an expansion of the glob.

Similar Ubuntu queries: For Ubuntu, a query of the form dpkg -S <filename> shows the packages that provide a file with the pattern <filename>.

Example

root@head:~# dpkg -S "nsswitch.conf"
cm-config-ldap-client-master: /cm/conf/etc/nsswitch.conf
manpages: /usr/share/man/man5/nsswitch.conf.5.gz
libc-bin: /usr/share/libc-bin/nsswitch.conf

To work out which Ubuntu package is a distribution package and which is a BCM package, queries similar to the following can be run:

Example

root@head:~# dpkg-query -W -f='${binary:Package} ${Version}\n' $(dpkg -S "nsswitch.conf" | cut -f1 -d:) |\
 grep cm10.0
cm-config-ldap-client-master 10.0-155-cm10.0
root@head:~# dpkg-query -W -f='${binary:Package} ${Version}\n' $(dpkg -S "nsswitch.conf" | cut -f1 -d:) |\
 grep -v cm10.0
libc-bin 2.35-0ubuntu3.8
manpages 5.10-1ubuntu1

What are the differences between the BCM version and the distribution versions of the file? Sometimes it is helpful to compare a distribution version and cluster version of nsswitch.conf to show the differences in configuration. The versions of the RPM packages containing the nsswitch.conf can be downloaded, their contents extracted, and their differences compared as follows:

~# mkdir yumextracted ; cd yumextracted
~# yumdownloader glibc-2.34
~# rpm2cpio glibc-2.34-100.el9_4.4.x86_64.rpm | cpio -idmv
~# yumdownloader cm-config-ldap-client-master
~# rpm2cpio cm-config-ldap-client-master-10.0-155_cm10.0.noarch.rpm | cpio -idmv
~# diff etc/nsswitch.conf cm/conf/etc/nsswitch.conf
...

What are the configuration files in an RPM package? An RPM package allows files within it to be marked as configuration files. Files marked as configuration files can be listed with rpm -qc <package>. Optionally, piping the list through “sort -u” filters out duplicates.

Example

~# rpm -qc glibc | sort -u
/etc/gai.conf
/etc/ld.so.cache
/etc/ld.so.conf
/etc/nsswitch.conf
/etc/rpc
/usr/lib64/gconv/gconv-modules
/usr/lib/gconv/gconv-modules
/var/cache/ldconfig/aux-cache

How does an RPM installation deal with local configuration changes? Are there configuration files or critical files that BCM misses? Whenever an RPM installation detects a file with local changes, it can treat the local system file as if:

1. the local system file is frozen. The installation does not interfere with the local file, but places the updated file as an .rpmnew file in the same directory.

2. the local system file is not frozen. The installation changes the local file. It copies the local file to an .rpmsave file in the same directory, and installs a new file from the RPM package.

NOTE: This package manager freezing should not be confused with CMDaemon freezing. With CMDaemon freezing, which is done with the FrozenFile directive (Appendix C.69), a file or section of a file that would normally be maintained by CMDaemon before freezing it, is no longer maintained by CMDaemon after freezing it.

When building BCM packages, the package builders can specify which of these two methods apply. When dealing with the built package, the system administrator can use an rpm query method to determine which of the two methods applies for a particular file in the package. For example, for glibc, the following query can be used and grepped:

rpm -q --queryformat '[%{FILENAMES}\t%{FILEFLAGS:fflags}\n]' glibc | egrep '[[:space:]].*(c|n).*$' | sort -u
/etc/gai.conf   cmng
/etc/ld.so.cache        cmng
/etc/ld.so.conf cn
/etc/nsswitch.conf      cn
/etc/rpc        cn
/usr/lib64/gconv/gconv-modules  cn
/usr/lib/gconv/gconv-modules    cn
/var/cache/ldconfig/aux-cache   cmng

Here, the second column of the output displayed shows which of the files in the package have a configuration (c) flag or a noreplace (n) flag. The c flag without the n flag indicates that an .rpmsave file will be created, while a c flag together with an n flag indicates that an .rpmnew file will be created.

In any case, files that are not marked as configuration files are overwritten during installation.

So:

• If a file is not marked as a configuration file, and it has been customized by the system administrator, and this file is provided by an RPM package, and the RPM package is updated on the system, then the file is overwritten silently.

• If a file is marked as a configuration file, and it has been customized by the system administrator, and this file is provided by an RPM package, and the RPM package is updated on the system, then it is good practice to look for .rpmsave and .rpmnew versions of that file, and run a comparison on detection.

BCM should however mark all critical files as configuration files in BCM packages.

Sometimes, RPM updates can overwrite a particular file that the administrator has changed locally and then would like to keep frozen.

To confirm that this is the problem, the following should be checked:

• The --queryformat option should be used to check that file can indeed be overwritten by updates. If the file has an n flag (regardless of whether it is a configuration file or not) then overwriting due to RPM updates does not happen, and the local file remains frozen. If the file has no n flag, then replacement occurs during RPM updates.

For files with no n flag, but where the administrator would still like to freeze the file during updates, the following can be considered:

• The file text content should be checked to see if it is a CMDaemon-maintained file (section 2.6.5), or checked against the list of generated files (Appendix A). This is just to make sure to avoid confusion about how changes are occurring in such a file.

  • If it is a CMDaemon-maintained file, then configuration changes put in by the administrator will also not persist in the maintained section of the file unless the FrozenFile directive (section C.69) is used to freeze the change.

  • If it is only a section that CMDaemon maintains, then configuration changes can be placed outside of the maintained section.

  Wherever the changes are placed in such a file, these changes are in any case by default overwritten on RPM updates if the file has no n flag.

• Some regular node updates can effectively be maintained in a desired state with the help of a finalize script (Appendix E).

• Updates can be excluded from YUM/zypper (section 11.3.2), thereby avoiding the overwriting of that file by the excluded package.

A request to change the package build flag may be sent to the BCM support team if the preceding suggested options are unworkable.