> This page is for version 11 (default).
> For other versions, use one of these documentation indexes:
> - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt
> - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server.

# Burning Nodes

The *burn framework* is a component of NVIDIA Base Command Manager 11 that can automatically run test scripts on specified nodes within a cluster. The framework is designed to stress test newly built machines and to detect components that may fail under load. Nodes undergoing a burn session with the default burn configuration, lose their filesystem and partition data for all attached drives, and revert to their software image on provisioning after a reboot.

## 11.1  Test Scripts Deployment

The framework requires power management to be running and working properly so that the node can be power cycled by the scripts used. In modern clusters power management is typically achieved by enabling a baseboard management controller such as IPMI, iLO, or Redfish. Details on power management are given in [Chapter 4 of the *Administrator Manual*](/admin/power).

The framework can run any executable script. The default test scripts are mostly `bash` shell scripts and Perl scripts. Each test script has a directory in `/cm/shared/apps/cmburn` containing the script. The directory and test script must have the same name. For example: `/cm/shared/apps/cmburn/disktest/disktest` is the default script used for testing a disk. More on the contents of a test script is given in section [11.3.2](#1132-writing-a-test-script).

## 11.2  Burn Configurations

### 11.2.1  Burn Configuration Simple XML Example

A *burn configuration* is an XML file stored in the CMDaemon database that specifies the burn tests and the order in which they run. Within the burn configuration the tests are normally grouped into sequences, and several sequences typically make up a phase. Phases in turn are grouped in either a pre-install section or post-install section. A simple example of such a burn configuration could therefore look like:

**Example**

```bash
<?xml version="1.0"?>
<burnconfig>

  <mail>
    <address>root@master</address>
    <address>some@other.address</address>
  </mail>

  <pre-install>

    <phase name="01-hwinfo">
      <test name="hwinfo"/>
      <test name="hwdiff"/>
      <test name="sleep" args="10"/>
    </phase>

    <phase name="02-disks">
      <test name="disktest" args="30"/>
      <test name="mce_check" endless="1"/>
    </phase>

  </pre-install>

  <post-install>

    <phase name="03-hpl">
      <test name="hpl"/>
      <test name="mce_check" endless="1"/>
    </phase>

    <phase name="04-compile">
      <test name="compile" args="6"/>
      <test name="mce_check" endless="1"/>
    </phase>

  </post-install>

</burnconfig>
```

#### 11.2.1.1  Mail Tag

The optional `<mail>` tag pair can add a sequence of e-mail addresses, with each address enclosed in an `<address>` tag pair. These addresses receive burn failure and warning messages, as well as a notice when the burn run has completed.

#### 11.2.1.2  Pre-install And Post-install

The pre-install part of a burn configuration is configured with the tag pair, and run from inside a node-installer environment. This environment is a limited Linux environment and allows some simpler tests to run before loading up the full Linux node environment.

Similarly, the post-install part of a burn configuration uses the tag pair to run from inside the full Linux node environment. This environment allows more complex tests to run.

#### 11.2.1.3  Post-burn Install Mode

The optional tag pair allows the administrator to specify [the install mode (section 5.4.4 of the *Administrator Manual*)](/admin/provisioning#544-determining-install-mode-type-and-execution-mode) after burn. The tag pair can enclose a setting of AUTO, FULL, MAIN, or NOSYNC. The default setting is the install mode that was set before burn started.

#### 11.2.1.4  Phases

The phases sections must exist. If there is no content for the phases, the phases tags must still be in place (“must exist”). Each phase must have a unique name and must be written in the burn configuration file in alphanumerical order. By default, numbers are used as prefixes. The phases are executed in sequence.

#### 11.2.1.5  Tests

Each phase consists of one or more test tags. The tests can optionally be passed arguments using the args property of the burn configuration file (section [11.2](#112-burn-configurations)). If multiple arguments are required, they should be a space separated list, with the (single) list being the args property.

Tests in the same phase are run simultaneously.

Most tests test something and then end. For example, the disk test tests the performance of all drives and then quits.

Tests which are designed to end automatically are known as *non-endless* tests.

Tests designed to monitor continuously are known as *endless tests*. Endless tests are not really endless. They end once all the non-endless tests in the same phase are ended, thus bringing an end to the phase. Endless tests typically test for errors caused by the load induced by the non-endless tests. For example the `mce_check` test continuously keeps an eye out for Machine Check Exceptions while the non-endless tests in the same phase are run.

A special test is the final test, `memtest86`, which is part of the default burn run, as configured in the XML configuration `default-destructive`. It does run endlessly if left to run. To end it, the administrator can deal with its output at the node console or can power reset the node. It is usually convenient to remove `memtest86` from the default XML configuration in larger clusters, and to rely on the `HPL` and `memtester` tests instead, for uncovering memory hardware errors.

### 11.2.2  Burn Configuration XML Schema Definition

In BCM, burn configuration setups such as in section [11.2.1](#1121-burn-configuration-simple-xml-example) have their global structure defined using an XML schema definition, which is installed on the head node in `/cm/local/apps/cmd/etc/htdocs/xsd/burnconfig.xsd`.

The XML schema definition can also be viewed with:

**Example**

```bash
[root@basecm11 ]# cmsh -c "main xsdschema burnconfig" | less
```

## 11.3  Running A Burn Configuration

Burn configurations can be viewed and executed from `cmsh`.

### 11.3.1  Burn Configuration And Execution In `cmsh`

#### 11.3.1.1  Burn Configuration File Settings

From `cmsh`, the burn configurations can be accessed from `partition` mode as follows:

**Example**

```bash
[basecm11]% partition use base
[basecm11->partition[base]]% burnconfigs
[basecm11->partition[base]->burnconfigs]% list
Name (key)           Description              XML
-------------------  ------------------------ -------------
default-destructive  Standard burn test.      <2614 bytes>
long-hpl             Run HPL test for a long+ <829 bytes>
```

The values of a particular burn configuration (`default-destructive` in the following example) can be viewed as follows:

**Example**

```bash
[basecm11->partition[base]->burnconfigs]% use default-destructive
[basecm11->partition[base]->burnconfigs[default-destructive]]% show
Parameter              Value
---------------------- --------------------------------------------------------
Description            Standard destructive burn test. Beware, wipes the disks!
Name                   default-destructive
Revision
XML                    <2614 bytes>
```

The `set` command can be used to modify existing values of the burn configuration, that is: `Description`, `Name`, and `XML`. `XML` is the burn configuration file itself. The `get xml` command can be used to view the file, while using `set xml` opens up the default text editor, thus allowing the burn configuration to be modified.

A new burn configuration can also be added with the `add` command. The new burn configuration can be created from scratch with the `set` command. However, an XML file can also be imported to the new burn configuration by specifying the full path of the XML file to be imported:

**Example**

```bash
[basecm11->partition[base]->burnconfigs]% add boxburn
[basecm11->partition[base]->burnconfigs*[boxburn*]]% set xml /tmp/im.xml
```

The burn configuration can also be edited when carrying out burn execution with the `burn` command.

#### 11.3.1.2  Executing A Burn

A burn as specified by the burn configuration file can be executed in `cmsh` using the `burn` command of device mode.

**Burn commands:** The burn commands can modify these properties, as well as execute other burn-related operations.

The burn commands are executed within device mode, and are:

* `burn start`

* `burn stop`

* `burn status`

* `burn log`

The burn help text that follows lists the detailed options. Next, operations with the burn commands illustrate how the options may be used along with some features.

```bash
[head1->device[node005]]% burn
Name:     burn - Node burn control

Usage:    burn [OPTIONS] status
          burn [OPTIONS] start
          burn [OPTIONS] stop
          burn [OPTIONS] log

Options:  -n, --nodes <node>
               List of nodes, e.g. node001..node015,node020..node028,node030
               or ^/some/file/containing/hostnames

          -g, --group <group>
               Include all nodes that belong to the node group, e.g. testnodes or test01,test03

          -c, --category <category>
               Include all nodes that belong to the category, e.g. default or default,gpu

          -r, --rack <rack>
               Include all nodes that are located in the given rack, e.g rack01
               or rack01..rack04

          -h, --chassis <chassis>
               Include all nodes that are located in the given chassis, e.g chassis01
               or chassis03..chassis05

          -e, --overlay <overlay>
               Include all nodes that are part of the given overlay, e.g overlay1
               or overlayA,overlayC

          -m, --image <image>
               Include all nodes that have the given image, e.g default-image or
               default-image,gpu-image

          -t, --type <type>
               Type of devices, e.g node or virtualnode,cloudnode

          -i, --intersection
               Calculate the intersection of the above selections

          -u, --union
               Calculate the union of the above selections

          -l, --role role
               Filter all nodes that have the given role

          -s, --status <status>
               Only run command on nodes with specified status, e.g. UP, "CLOSED|DOWN",
               "INST.*"

          --config <name>
               Burn with the specified burn configuration. See in partition burn configurations
               for a list of valid names

          --file <path>
               Burn with the specified file instead of burn configuration

          --later
               Do not reboot nodes now, wait until manual reboot

          --edit
               Open editor for last minute changes

          --no-drain
               Do not drain the node from WLM before starting to burn

          --no-undrain
               Do not undrain the node from WLM after burn is complete

          -p, --path
               Show path to the burn log files. Of the form: /var/spool/burn/<mac>.

          -v, --verbose
               Show verbose output (only for burn status)

          --sort <field1>[,<field2>,...]
               Override default sort order (only for burn status)

          -d, --delimiter <string>
               Set default row separator (only for burn status)

          -d, --delay <seconds>
               Wait <seconds> between executing two sequential power commands. This option is
               ignored for the status command

Examples:
          burn --config default-destructive start -n node001
```

**Burn command operations:** Burn commands allow the following operations, and have the following features:

* `start`, `stop`, `status`, `log`: The basic burn operations allow a burn to be started or stopped, and the status of a burn to be viewed and logged.

  * The “`burn start`” command always needs a configuration file name. In the following it is `boxburn`. The command also always needs to be given the nodes it operates on:

    ```bash
    [basecm11->device]% burn --config boxburn -n node007 start
    Power reset nodes
    [basecm11->device]%
    ipmi0 .................... [  RESET  ] node007
    Fri Nov  3 ... [notice] basecm11: node007 [ DOWN ]
    [basecm11->device]%
    Fri Nov  3 ... [notice] basecm11: node007 [ INSTALLING ] (node installer started)
    [basecm11->device]%
    Fri Nov  3 ... [notice] basecm11: node007 [ INSTALLING ] (running burn in tests)
    ...
    ```

  * The “`burn stop`” command only needs to be given the nodes it operates on, for example:

    ```bash
     [basecm11->device]% burn -n node007 stop
    ```

  * The “`burn status`” command:

    * may be given the nodes for which the status is to be found, for example:

      ```bash
        [basecm11->device]% burn status -n node005..node007
        Hostname Burn name Status                    New burn on PXE Phase    ...
        -------- --------- ------------------------- --------------- -------- ...
        node005            no burn results available no                       ...
        node006            currently not burning     no                       ...
        node007  boxburn   Burning                   yes             02-disks ...

        each line of output is quite long, so each line has been rendered truncated and ellipsized.
        The ellipsis marks in the 5 preceding output lines align with the lines that follow.
        That is, the lines that follow are the endings of the preceding 5 lines:

         ...Warnings  Tests
         ...--------- --------------------------------------------------------------
         ...0
         ...0
         ...0         /var/spool/burn/c8-1f-66-f2-61-c0/02-disks/disktest (S,171),\
                      /var/spool/burn/c8-1f-66-f2-61-c0/02-disks/kmon (S),\
                      /var/spool/bu+
      ```

  * The “`burn log`” command displays the burn log for specified node groupings. Each node with a boot MAC address of \<*mac*> has an associated burn log file.

* Advanced options include the following:

  * `-n|--nodes`, `-g|--group`, `-c|--category`, `-r|--rack`, `-h|--chassis`, `-e|--overlay`, `-t|--type`, `-i|--intersection`, `-u|--union`, `-r|--role`, `-s|--status`: These options allow burn commands to be executed over various node groupings.

  * `--config` \<*burn configuration*>, `--file` \<*path*>: The burn configuration file can be set as one of the XML burn configurations listed in `partition` mode, or it can be set as the path of the XML file from the file system

  * `-l|--later`: This option disables the immediate power reset that occurs on running the “`burn start`” or “`burn stop`” command on a node. This allows the administrator to power down manually, when preferred.

  * `-e|--edit`: The burn configuration file can be edited with the `-e` option for the “`burn start`” command. This is an alternative to editing the burn configuration file in `partition` mode.

  * `--no-drain`, `--no-undrain`: The node is not drained before burning, or not undrained after burning

  * `-p|--path`: This shows the burn log path, by default (section [11.4](#114-relocating-the-burn-logs)) on the head node under a directory named after the mac address `/var/spool/burn/` \<*`mac`*>.

    **Example**

    ```bash
    [basecm11->device]% burn -p -n node001..node003 log
    node001: /var/spool/burn/fa-16-3e-60-1d-c7
    node002: /var/spool/burn/fa-16-3e-8a-d8-0e
    node003: /var/spool/burn/fa-16-3e-af-38-cd
    ```

**Burn command output examples:**

The `burn status` command has a compact one-line output per node:

**Example**

```bash
[basecm11->device]% burn -n node001 status
node001 (00000000a000) - W(0) phase 02-disks 00:02:58 (D:H:M) FAILED, mce_check (SP),\
 disktest (SF,61), kmon (SP)
```

The fields in the preceding output example are:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>
        **Description**
      </th>

      <th>
        **Value**
      </th>

      <th>
        **Meaning Here**
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        The node name
      </td>

      <td>
        `node001`
      </td>

      <td />
    </tr>

    <tr>
      <td>
        The node tag
      </td>

      <td>
        `(00000000a000)`
      </td>

      <td />
    </tr>

    <tr>
      <td>
        Warnings since start of burn
      </td>

      <td>
        `(0)`
      </td>

      <td />
    </tr>

    <tr>
      <td rowspan="2">
        The current phase name
      </td>

      <td rowspan="2">
        `02-disks`
      </td>

      <td>
        Burn configuration phase being
      </td>
    </tr>

    <tr>
      <td>
        run is 

        `02-disks`
      </td>
    </tr>

    <tr>
      <td>
        Time since phase started
      </td>

      <td>
        `00:02:58 (D:H:M)`
      </td>

      <td>
        2 hours 58 minutes
      </td>
    </tr>

    <tr>
      <td>
        State of current phase
      </td>

      <td>
        `FAILED`
      </td>

      <td>
        Failed in 

        `02-disks`
      </td>
    </tr>

    <tr>
      <td>
        burn test for MCE
      </td>

      <td>
        `mce_check (SP)`
      </td>

      <td>
        Started and Passed
      </td>
    </tr>

    <tr>
      <td rowspan="3">
        burn test for disks
      </td>

      <td rowspan="3">
        `disktest (SF,61)`
      </td>

      <td>
        Started and Failed
      </td>
    </tr>

    <tr>
      <td>
        61 is the speed, and is custom
      </td>
    </tr>

    <tr>
      <td>
        information
      </td>
    </tr>

    <tr>
      <td>
        burn test kernel log monitor
      </td>

      <td>
        `kmon (SP)`
      </td>

      <td>
        Started and Passed
      </td>
    </tr>
  </tbody>
</table>

Each test in a phase uses these letters to display its status:

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>
        **Letter**
      </th>

      <th>
        **Meaning**
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        S
      </td>

      <td>
        started
      </td>
    </tr>

    <tr>
      <td>
        W
      </td>

      <td>
        warning
      </td>
    </tr>

    <tr>
      <td>
        F
      </td>

      <td>
        failed
      </td>
    </tr>

    <tr>
      <td>
        P
      </td>

      <td>
        passed
      </td>
    </tr>
  </tbody>
</table>

The “`burn log`” command output looks like the following (some output elided):

```bash
[basecm11->device]% burn -n node001 log
Thu ... 2012: node001 - burn-control: burn framework initializing
Thu ... 2012: node001 - burn-control: e-mail will be sent to: root@master
Thu ... 2012: node001 - burn-control: finding next pre-install phase
Thu ... 2012: node001 - burn-control: starting phase 01-hwinfo
Thu ... 2012: node001 - burn-control: starting test /cm/shared/apps/cmburn/hwinfo
Thu ... 2012: node001 - burn-control: starting test /cm/shared/apps/cmburn/sleep
Thu ... 2012: node001 - sleep: sleeping for 10 seconds
Thu ... 2012: node001 - hwinfo: hardware information
Thu ... 2012: node001 - hwinfo: CPU1: vendor_id = AuthenticAMD
...
Thu ... 2012: node001 - burn-control: test hwinfo has ended, test passed
Thu ... 2012: node001 - burn-control: test sleep has ended, test passed
Thu ... 2012: node001 - burn-control: all non-endless test are done, terminating endless tests
Thu ... 2012: node001 - burn-control: phase 01-hwinfo passed
Thu ... 2012: node001 - burn-control: finding next pre-install phase
Thu ... 2012: node001 - burn-control: starting phase 02-disks
Thu ... 2012: node001 - burn-control: starting test /cm/shared/apps/cmburn/disktest
Thu ... 2012: node001 - burn-control: starting test /cm/shared/apps/cmburn/mce_check
Thu ... 2012: node001 - burn-control: starting test /cm/shared/apps/cmburn/kmon
Thu ... 2012: node001 - disktest: starting, threshold = 30 MB/s
Thu ... 2012: node001 - mce_check: checking for MCE's every minute
Thu ... 2012: node001 - kmon: kernel log monitor started
Thu ... 2012: node001 - disktest: detected 1 drives: sda
...
Thu ... 2012: node001 - disktest: drive sda wrote 81920 MB in 1278.13
Thu ... 2012: node001 - disktest: speed for drive sda was 64 MB/s -> disk passed
Thu ... 2012: node001 - burn-control: test disktest has ended, test FAILED
Thu ... 2012: node001 - burn-control: all non-endless test are done, terminating endless tests
Thu ... 2012: node001 - burn-control: asking test /cm/shared/apps/cmburn/kmon/kmon to terminate
Thu ... 2012: node001 - kmon: kernel log monitor terminated
Thu ... 2012: node001 - burn-control: test kmon has ended, test passed
Thu ... 2012: node001 - burn-control: asking test /cm/shared/apps/cmburn/mce_check/mce_check\
 to terminate
Thu ... 2012: node001 - mce_check: terminating
Thu ... 2012: node001 - mce_check: waiting for mce_check to stop
Thu ... 2012: node001 - mce_check: no MCE's found
Thu ... 2012: node001 - mce_check: terminated
Thu ... 2012: node001 - burn-control: test mce_check has ended, test passed
Thu ... 2012: node001 - burn-control: phase 02-disks FAILED
Thu ... 2012: node001 - burn-control: burn will terminate
```

The output of the `burn log` command is actually the `messages` file in the burn directory, for the node associated with a MAC-address directory \<*`mac`*>. The burn directory is at `/var/spool/burn/` and the `messages` file is thus located at:

`/var/spool/burn/` \<*`mac`*>`/messages`

The tests have their log files in their own directories under the MAC-address directory, using their phase name. For example, the pre-install section has a phase named `01-hwinfo`. The output logs of this test are then stored under:

`/var/spool/burn/` \<*`mac`*>`/01-hwinfo/`

### 11.3.2  Writing A Test Script

This section describes a sample test script for use within the burn framework. The script is typically a shell or Perl script. The sample that follows is a Bash script, while the `hpl` script is an example in Perl.

Section [11.1](#111-test-scripts-deployment) describes how to deploy the script.

#### 11.3.2.1  Non-endless Tests

The following example test script is not a working test script, but can be used as a template for a non-endless test:

**Example**

```bash
#!/bin/bash

# We need to know our own test name, amongst other things for logging.
me=`basename $0`

# This first argument passed to a test script by the burn framework is a
# path to a spool directory. The directory is created by the framework.
# Inside the spool directory a sub-directory with the same name as the
# test is also created. This directory ($spooldir/$me) should be used
# for any output files etc. Note that the script should possibly remove
# any previous output files before starting.
spooldir=$1

# In case of success, the script should touch $passedfile before exiting.
passedfile=$spooldir/$me.passed

# In case of failure, the script should touch $failedfile before exiting.
# Note that the framework will create this file if a script exits without
# creating $passedfile. The file should contain a summary of the failure.
failedfile=$spooldir/$me.failed

# In case a test detects trouble but does not want the entire burn to be
# halted $warningfile _and_ $passedfile should be created. Any warnings
# should be written to this file.
warningfile=$spooldir/$me.warning

# Some short status info can be written to this file. For instance, the
# stresscpu test outputs something like 13/60 to this file to indicate
# time remaining.
# Keep the content on one line and as short as possible!
statusfile=$spooldir/$me.status

# A test script can be passed arguments from the burn configuration. It
# is recommended to supply default values and test if any values have
# been overridden from the config file. Set some defaults:
option1=40
option2=some_other_value

# Test if option1 and/or option2 was specified (note that $1 was to
# spooldir parameter):
if [ ! x$2 = "x" ]; then
  option1=$2
fi
if [ ! x$3 = "x" ]; then
  option2=$3
fi

# Some scripts may require some cleanup. For instance a test might fail
# and be
# restarted after hardware fixes.
rm -f $spooldir/$me/*.out &>/dev/null

# Send a message to the burn log file, syslog and the screen.
# Always prefix with $me!
blog "$me: starting, option1 = $option1 option2 = $option2"

# Run your test here:
run-my-test
if [ its_all_good ]; then
  blog "$me: wOOt, it's all good! my-test passed."
  touch $passedfile
  exit 0
elif [ was_a_problem ]; then
  blog "$me: WARNING, it did not make sense to run this test. You don't have special device X."
  echo "some warning" >> $warningfile  # note the append!
  touch $passedfile
  exit 0
else
  blog "$me: Aiii, we're all gonna die! my-test FAILED!"
  echo "Failure message." > $failedfile
  exit 0
fi
```

#### 11.3.2.2  Endless Tests

The following example test script is not a working test, but can be used as a template for an endless test.

**Example**

```bash
#!/bin/bash

# We need to know our own test name, amongst other things for logging.
me=`basename $0`

# This first argument passed to a test script by the burn framework is a
# path to a spool directory. The directory is created by the framework.
# Inside the spool directory a sub-directory with the same name as the
# test is also created. This directory ($spooldir/$me) should be used
# for any output files etc. Note that the script should possibly remove
# any previous output files before starting.
spooldir=$1

# In case of success, the script should touch $passedfile before exiting.
passedfile=$spooldir/$me.passed

# In case of failure, the script should touch $failedfile before exiting.
# Note that the framework will create this file if a script exits without
# creating $passedfile. The file should contain a summary of the failure.
failedfile=$spooldir/$me.failed

# In case a test detects trouble but does not want the entire burn to be
# halted $warningfile _and_ $passedfile should be created. Any warnings
# should be written to this file.
warningfile=$spooldir/$me.warning

# Some short status info can be written to this file. For instance, the
# stresscpu test outputs something like 13/60 to this file to indicate
# time remaining.
# Keep the content on one line and as short as possible!
statusfile=$spooldir/$me.status

# Since this in an endless test the framework needs a way of stopping it
# once all non-endless test in the same phase are done. It does this by
# calling the script once more and passing a "-terminate" argument.
if [ "$2" == "-terminate" ]; then
  blog "$me: terminating"

  # remove the lock file the main loop is checking for
  rm $spooldir/$me/running

  blog "$me: waiting for $me to stop"
  # wait for the main loop to die
  while [ -d /proc/`cat $spooldir/$me/pid` ]
  do
    sleep 1
  done
  blog "$me: terminated"

else
  blog "$me: starting test, checking every minute"

  # Some scripts may require some cleanup. For instance a test might fail
  # and be restarted after hardware fixes.
  rm -f $spooldir/$me/*.out &>/dev/null

  # create internal lock file, the script will remove this if it is
  # requested to end
  touch $spooldir/$me/running

  # save our process id
  echo $$ > "$spooldir/$me/pid"

  while [ -e "$spooldir/$me/running" ]
  do

    run-some-check
    if [ was_a_problem ]; then
      blog "$me: WARNING, something unexpected happened."
      echo "some warning" >> $warningfile  # note the append!
    elif [ failure ]; then
      blog "$me: Aiii, we're all gonna die! my-test FAILED!"
      echo "Failure message." > $failedfile
    fi
    sleep 60

  done

  # This part is only reached when the test is terminating.
  if [ ! -e "$failedfile" ]; then
    blog "$me: no problem detected"
    touch $passedfile
  else
    blog "$me: test ended with a failure"
  fi

fi
```

### 11.3.3  Burn Failures

Whenever the burn process fails, the output of the `burn log` command shows the phase that has failed and that the burn terminates.

**Example**

```bash
Thu ... 2012: node001 - burn-control: phase 02-disks FAILED
Thu ... 2012: node001 - burn-control: burn will terminate
```

Here, `burn-control`, which is the parent of the disk testing process, keeps track of the tests that pass and fail. On failure of a test, `burn-control` terminates all tests.

The node that has failed then requires intervention from the administrator in order to change state. The node does not restart by default. The administrator should be aware that the state reported by the node to CMDaemon remains `burning` at this point, even though it is not actually doing anything.

To change the state, the burn must be stopped with the `burn stop` command in `cmsh`. If the node is restarted without explicitly stopping the burn, then it simply retries the phase at which it failed.

Under the burn log directory, the log of the particular test that failed for a particular node can sometimes suggest a reason for the failure. For retries, old logs are not overwritten, but moved to a directory with the same name, and a number appended indicating the try number. Thus:

**Example**

First try, and failing at `02-disks` tests:

```bash
cd /var/spool/burn/48:5b:39:19:ff:b3
ls -ld 02-disks*/
drwxr-xr-x 6 root root 4096 Jan 10 16:26 02-disks
```

2nd try, after failing again:

```bash
ls -ld 02-disks*/
drwxr-xr-x 6 root root 4096 Jan 10 16:49 02-disks
drwxr-xr-x 6 root root 4096 Jan 10 16:26 02-disks.1
```

3rd try, after failing again:

```bash
ls -ld 02-disks*/
drwxr-xr-x 6 root root 4096 Jan 10 16:59 02-disks
drwxr-xr-x 6 root root 4096 Jan 10 16:49 02-disks.1
drwxr-xr-x 6 root root 4096 Jan 10 16:26 02-disks.2
```

## 11.4  Relocating The Burn Logs

A burn run can append substantial amounts of log data to the default burn spool at `/var/spool/burn`. To avoid filling up the head node with such logs, they can be appended elsewhere.

### 11.4.1  Configuring The Relocation

The 3-part procedure that can be followed is:

1. The `BurnSpoolDir` setting can be set in the CMDaemon configuration file on the head node, at `/cm/local/apps/cmd/etc/cmd.conf`. The `BurnSpoolDir` setting tells CMDaemon where to look for burn data when the burn status is requested through `cmsh`.

* ```bash
  BurnSpoolDir="/var/spool/burn"
  ```

CMDaemon should be restarted after the configuration has been set. This can be done with:

```bash
service cmd restart
```

2.

The `burnSpoolHost` setting, which matches the host, and `burnSpoolPath` setting, which matches the location, can be changed in the node-installer configuration file on the head node, at `/cm/node-installer/scripts/node-installer.conf` (for multiarch/multidistro configurations the path takes the form: `/cm/node-installer-`*\<distribution>-\<architecture>*`/scripts/node-installer.conf`). These have the following values by default:

* ```bash
  burnSpoolHost = master
  ```

* ```bash
  burnSpoolPath = /var/spool/burn
  ```

These values define the NFS-mounted spool directory.

The `burnSpoolHost` value should be set to the new DNS host name, or to an IP address. The `burnSpoolPath` value should be set to the new path for the data.

3. Part 3 of the procedure adds a new location to export the burn log. This is only relevant if the spool directory is being relocated within the head node. If the spool is on an external fileserver, the existing burn log export may as well be removed.

The new location can be added to the head node as a path value, from a writable filesystem export name. The writable filesystem export name can most easily be added using Base View, via the navigation path:

<code>
  Devices > All Devices > Head Node

   \> Settings > Filesystem exports

   \> + Add
</code>

Adding a new name like this is recommended, instead of just modifying the path value in an existing `Filesystem exports` name. This is because changing things back if the configuration is done incorrectly is then easy. By default, the existing `Filesystem exports` for the burn directory has the name:

* `/var/spool/burn@internalnet`

and has a path associated with it with a default value of:

* `/var/spool/burn`

When the new name is set in `Filesystem exports`, the associated path value can be set in agreement with the values set earlier in parts 1 and 2.

If using `cmsh` instead of Base View, then the change can be carried out from within the `fsexports` submode. [Section 3.14.1 of the *Administrator Manual*](/admin/configuring#3141-exporting-a-filesystem-using-base-view-and-cmsh) gives more detail on similar examples of how to add such filesystem exports.

### 11.4.2  Testing The Relocation

To test the changes, it is wise to first try a single node with a short burn configuration. This allows the administrator to check that install and post-install tests can access the spool directories. Otherwise there is a risk of waiting hours for the pre-install tests to complete, only to have the burn abort on the post-install tests. The following short burn configuration can be used:

**Example**

```bash
<burnconfig>
<pre-install>
<phase name="01-hwinfo">
<test name="hwinfo"/>
<test name="sleep" args="10"/>
</phase>
</pre-install>
<post-install>
<phase name="02-mprime">
<test name="mprime" args="2"/>
<test name="mce_check" endless="1"/>
<test name="kmon" endless="1"/>
</phase>
</post-install>
</burnconfig>
```

To burn a single node with this configuration, the following could be run from the device mode of `cmsh`:

**Example**

```bash
[basecm11->device]% burn start --config default-destructive --edit -n node001
```

This makes an editor pop up containing the default burn configuration. The content can be replaced with the short burn configuration. Saving and quitting the editor causes the node to power cycle and start its burn.

The example burn configuration typically completes in less than 10 minutes or so, depending mostly on how fast the node can be provisioned. It runs the `mprime` test for about two minutes.