fq2bam (BWA-MEM + GATK)

Generate BAM/CRAM output given one or more pairs of FASTQ files. Can also optionally generate a BQSR report.

Refer to the fq2bam Reference section for a detailed listing of all available options.

What is BWA-MEM?

BWA-MEM is a fast, accurate algorithm for mapping DNA sequence reads to a reference genome, performing local alignment and producing alignment for different parts of the query sequence. It is the default algorithm in Burrows-Wheeler Aligner (BWA) for reads that are longer than 70bp and is designed for high-throughput sequencing technologies such as Illumina and Pacific Biosciences.

Why BWA-MEM?

BWA-MEM is capable of handling longer reads and is less sensitive to errors than other alignment algorithms. It is therefore used for a variety of applications, from routine analysis of sequencing data to more advanced applications such as de novo assembly and variant calling.

Some of the advantages of using BWA-MEM over similar tools include:

1. It is faster than many other alignment algorithms, making it the ideal choice for high-throughput sequencing.
2. It has a lower false positive rate than many other alignment algorithms, which means fewer false-positive variants are reported.
3. It is memory-efficient, allowing it to be used on limited resources.
4. It is highly accurate, with a reported accuracy of over 99% on Illumina data.

What is fq2bam?

BWA-MEM can be deployed within Parabricks, a software suite designed for accelerated secondary analysis in genomics, bringing industry standard tools and workflows from CPU to GPU, and delivering the same results at up to 60x faster runtimes. FQ2BAM is the Parabricks wrapper for BWA-MEM, which will sort the output and can mark duplicates and recalibrate base quality scores in line with GATK best practices. A 30x whole genome can be run through FQ2BAM in as little as 6 minutes on an NVIDIA DGX system, compared to 4-9 hours on a CPU instance (m5.24xlarge, 96 x vCPU). GPUs with at least 16 GB of memory are supported; see Useful Options for Performance for how GPU streams are sized automatically.

How Should I Use BWA-MEM in fq2bam?

fq2bam uses an accelerated version of BWA-MEM to generate BAM/CRAM output given one or more pairs of FASTQ files. The user can turn-off marking of duplicates by adding the —no-markdups option. The BQSR step is only performed if the —knownSites input and —out-recal-file output options are provided; doing so will also generate a BQSR report.

Quick Start

# This command assumes all the inputs are in the current working directory and all the outputs go to the same place.
docker run --rm --gpus all --volume $(pwd):/workdir --volume $(pwd):/outputdir \
    --workdir /workdir \
    nvcr.io/nvidia/clara/clara-parabricks:4.8.0-1 \
    pbrun fq2bam \
    --ref /workdir/${REFERENCE_FILE} \
    --in-fq /workdir/${INPUT_FASTQ_1} /workdir/${INPUT_FASTQ_2}  \
    --knownSites /workdir/${KNOWN_SITES_FILE} \
    --out-bam /outputdir/${OUTPUT_BAM} \
    --out-recal-file /outputdir/${OUTPUT_RECAL_FILE}

Useful Options for Performance

Suggested parameters to control host memory use:

• The parameter --bwa-normalized-queue-capacity controls the amount of batches that will be in memory to be processed. By default, the value normalized to the number of GPUs in the run is 10, corresponding to <number of GPUs> * <normalized capacity> for the total queue size. Multiple queues are used. Lowering this value would be a good strategy to reduce host memory use. However, a value that is too low may hamper performance.

By default, Parabricks automatically configures GPU streams and CPU threading. The option --bwa-nstreams auto selects the number of streams based on total GPU memory. This heuristic is tuned for a human reference genome; if a run exceeds GPU memory, users should lower --bwa-nstreams manually or use --low-memory. The option --bwa-primary-cpus auto selects 16 primary CPU threads per GPU, while --bwa-cpu-thread-pool defaults to 1 thread per primary CPU thread. These defaults are optimized for end-to-end runtime. Users may experiment further with the --bwa-nstreams, --bwa-primary-cpus, and --bwa-cpu-thread-pool parameters to potentially achieve better performance given a particular system configuration.

For advanced performance tuning, the options --bwa-primary-cpus and --bwa-cpu-thread-pool provide fine-grained control of CPU threading. A manually set --bwa-primary-cpus value is the total number of primary CPU threads for the whole run, not a per-GPU value; only auto scales with the number of GPUs. Each primary CPU thread drives P CPU thread pool threads as specified with the option --bwa-cpu-thread-pool. The total number of CPU threads processing the CPU stages of alignment is the product of the --bwa-primary-cpus and --bwa-cpu-thread-pool parameters. This allows the user to control the ratio of “primary” CPU threads, which act independently, to thread pool threads, which act in unison. Depending on the server configuration, input data, and available GPUs, performance may improve by adjusting the balance between primary CPU threads and thread pool threads. Changing the number of primary CPU threads may increase the CPU resources required.

• The parameter --gpuwrite uses the GPU to compress the final BAM or CRAM file for improved performance during the final stage.

  • --gpuwrite-deflate-algo can be used to control the compression ratio. See below for more details.

• The parameter --gpusort uses the GPU to sort the BAM or CRAM file.

• The option --cigar-on-gpu offloads CIGAR generation from the CPU (default) to the GPU. Consider enabling it on CPU-constrained systems, or if benchmarking with and without the option indicates the CPU is a bottleneck; it may improve overall runtime.

• The option --max-read-length-on-gpu can be set to 1024 bp to process longer reads on GPU. Keep the default value of 512 bp unless the input has many reads longer than 512 bp. Batches with reads longer than the selected value continue to use CPU recovery. Reads longer than --max-read-length are filtered from the input; to process longer reads, --max-read-length must also be increased. Refer to the Options section for the default value.

• On multi-GPU systems, the option --run-partition enables partition mode, which forks one worker process per partition to more efficiently split work across GPUs. We recommend trying this on systems with 4 or more GPUs. The --bwa-gpu-num-per-partition option controls how many GPUs each partition drives (default: 2).

Compatible CPU-based BWA-MEM, GATK4 Commands

The commands below are the bwa-0.7.17 and GATK4 counterpart of the Parabricks command above. The output from these commands will be equivalent to the output from the above command. Refer to the Output Comparison page for comparing the results.

# Run bwa-mem and pipe the output to create a sorted BAM.
$ bwa mem \
    -t 32 \
    -K 10000000 \
    -R '@RG\tID:sample_rg1\tLB:lib1\tPL:bar\tSM:sample\tPU:sample_rg1' \
    <INPUT_DIR>/${REFERENCE_FILE} <INPUT_DIR>/${INPUT_FASTQ_1} <INPUT_DIR>/${INPUT_FASTQ_2} | \
  gatk SortSam \
    --java-options -Xmx30g \
    --MAX_RECORDS_IN_RAM 5000000 \
    -I /dev/stdin \
    -O cpu.bam \
    --SORT_ORDER coordinate
# Mark duplicates.
$ gatk MarkDuplicates \
    --java-options -Xmx30g \
    -I cpu.bam \
    -O mark_dups_cpu.bam \
    -M metrics.txt
# Generate a BQSR report.
$ gatk BaseRecalibrator \
    --java-options -Xmx30g \
    --input mark_dups_cpu.bam \
    --output <OUTPUT_DIR>/${OUTPUT_RECAL_FILE} \
    --known-sites <INPUT_DIR>/${KNOWN_SITES_FILE} \
    --reference <INPUT_DIR>/${REFERENCE_FILE}

Source of Mismatches

While Parabricks fq2bam does not lose any accuracy in functionality when compared with BWA-mem and GATK, still there are several sources that can result in differences in output files.

• BWA-mem -K argument

In pair-ended mode, the chunk size specified by -K can cause small mismatches in the output BAM file. To get rid of the mismatches here, make sure to pass the same number to both baseline BWA-mem and Parabricks fq2bam, e.g. -K 10000000.

• PA aux tag

  • Parabricks fq2bam puts the PA tag last while BWA-mem puts it first.
  • BWA-mem rounds PA tag to 3 digits while Parabricks fq2bam does not. And aux tag can be filtered by samtools view -x <TAG>

• MC (mate-CIGAR) aux tag

  • As fq2bam is compatible with BWA-MEM 0.7.17, paired-end records now receive an MC mate-CIGAR tag. To recapitulate the behavior of BWA-MEM 0.7.15 and earlier by omitting the mate-CIGAR tag, use --no-mate-cigar.

• Unmapped reads

  • Parabricks fq2bam sorts unmapped reads slightly differently than baseline GATK SortSam. Unmapped reads can be filtered with samtools by running samtools view -F 4.

Options