Confirming AJR is Operational#

AJR functionality can be tested by running simple bash scripts that simulate a fault and confirm that AJR resumes the job. A script is provided for each of the two auto resume modes. Use requeue unless your workflow specifically requires the singleton dependency chain.

Requeue (Recommended)#

#!/bin/bash

#SBATCH -t 00:05:00
#SBATCH --comment='{"APS": {"auto_resume_mode": "requeue", "max_requeue_times": 1}}'

DATETIME=`date +'date_%y-%m-%d_time_%H-%M-%S'`
echo 'AJR light-weight fault simulation test (auto resume mode: requeue)'

# This srun command sets the output file for the job step. In the job step, it simply sleep for 30s, then print a
# log line that will be recognized by AJR as segment fault, then sleep for another 60s before exiting with 1.

srun --output="$(pwd)/%x_%j_$DATETIME.log" bash -c \
     "echo 'Start time: $DATETIME' && echo 'Sleeping 30 seconds' && \
     sleep 30 && echo 'Rank0: (1) segmentation fault: artificial segfault' && \
     echo 'Sleep another 60 seconds' && \
     sleep 60 && exit 1"

Copy the content of the preceding bash script and save it into a file in Slurm, e.g.: sbatch_test.sh.
Submit the job to Slurm with the sbatch script. Note that you have to specify the job name (with -J option) and the partition name (with -p option) since they are not encoded in the script. Submit it once only; AJR requeues the job itself when it detects the fault. Replace <your_partition_name> with a Slurm partition you have access to. On clusters set up with the BCM wizard this is usually defq, which is the default partition name the wizard creates, but sites often define their own. Run sinfo to list the partitions available to you.

sbatch -p <your_partition_name> -J AJR-verification-test sbatch_test.sh

Singleton Dependency (Legacy)#

#!/bin/bash

#SBATCH -t 00:05:00
#SBATCH --dependency=singleton
#SBATCH --comment='{"APS": {"auto_resume_mode": "singleton_dependency"}}'

DATETIME=`date +'date_%y-%m-%d_time_%H-%M-%S'`
echo 'AJR light-weight fault simulation test (auto resume mode: singleton_dependency)'

# This srun command sets the output file for the job step. In the job step, it simply sleep for 30s, then print a
# log line that will be recognized by AJR as segment fault, then sleep for another 60s before exiting with 1.

srun --output="$(pwd)/%x_%j_$DATETIME.log" bash -c \
     "echo 'Start time: $DATETIME' && echo 'Sleeping 30 seconds' && \
     sleep 30 && echo 'Rank0: (1) segmentation fault: artificial segfault' && \
     echo 'Sleep another 60 seconds' && \
     sleep 60 && exit 1"

Copy the content of the preceding bash script and save it into a file in Slurm, e.g.: sbatch_test_singleton.sh.
Submit two jobs to Slurm with the sbatch script. Note that you have to specify the job name (with -J option) and the partition name (with -p option) since they are not encoded in the script. Also, make sure the second command is run with the --hold option as well. Replace <your_partition_name> with a Slurm partition you have access to. On clusters set up with the BCM wizard this is usually defq, which is the default partition name the wizard creates, but sites often define their own. Run sinfo to list the partitions available to you.

sbatch -p <your_partition_name> -J AJR-verification-test sbatch_test_singleton.sh

sbatch --hold -p <your_partition_name> -J AJR-verification-test sbatch_test_singleton.sh

The --hold option on the second submission is required. It places the job in a user-held state so that it does not start when the first job ends, which gives AJR the opportunity to compute the isolated nodes and update the job with an exclusion list before releasing it. AJR releases the held job automatically once it has attributed the failure of the first job.

Because AJR does not submit jobs on a user’s behalf, the chain runs only as many times as jobs were submitted. If no held job remains in the queue, the workflow stops.