fq2bam (FQ2BAM + BWA-MEM)
Generate BAM/CRAM output given one or more pairs of FASTQ files. Can also optionally generate a BQSR report.
BWA-MEM is a fast, accurate algorithm for mapping DNA sequence reads to a reference genome, performing local alignment and producing alignment for different parts of the query sequence. It is the default algorithm in Burrows-Wheeler Aligner (BWA) for reads that are longer than 70bp and is designed for high-throughput sequencing technologies such as Illumina and Pacific Biosciences.
BWA-MEM is capable of handling longer reads and is less sensitive to errors than other alignment algorithms. It is therefore used for a variety of applications, from routine analysis of sequencing data to more advanced applications such as de novo assembly and variant calling.
Some of the advantages of using BWA-MEM over similar tools include:
It is faster than many other alignment algorithms, making it the ideal choice for high-throughput sequencing.
It has a lower false positive rate than many other alignment algorithms, which means fewer false-positive variants are reported.
It is memory-efficient, allowing it to be used on limited resources.
It is highly accurate, with a reported accuracy of over 99% on Illumina data.
BWA-MEM can be deployed within NVIDIA Parabricks, a software suite designed for accelerated secondary analysis in genomics, bringing industry standard tools and workflows from CPU to GPU, and delivering the same results at up to 60x faster runtimes. FQ2BAM is the Parabricks wrapper for BWA-MEM, which will sort the output and can mark duplicates and recalibrate base quality scores in line with GATK best practices. A 30x whole genome can be run through FQ2BAM in as little as 17 minutes on an NVIDIA DGX system, compared to 4-9 hours on a CPU instance (m5.24xlarge, 96 x vCPU).
fq2bam uses an accelerated version of BWA-MEM to generate BAM/CRAM output given one or more pairs of FASTQ files. The user can turn-off marking of duplicates by adding the --no-markdups option. The BQSR step is only performed if the --knownSites input and --out-recal-file output options are provided; doing so will also generate a BQSR report.
# This command assumes all the inputs are in INPUT_DIR and all the outputs go to OUTPUT_DIR.
$ docker run --rm --gpus all --volume INPUT_DIR:/workdir --volume OUTPUT_DIR:/outputdir \
--workdir /workdir \
nvcr.io/nvidia/clara/clara-parabricks:4.1.0-1 \
pbrun fq2bam \
--ref /workdir/${REFERENCE_FILE} \
--in-fq /workdir/${INPUT_FASTQ_1} /workdir/${INPUT_FASTQ_2} \
--knownSites /workdir/${KNOWN_SITES_FILE} \
--out-bam /outputdir/${OUTPUT_BAM} \
--out-recal-file /outputdir/${OUTPUT_RECAL_FILE}
The commands below are the bwa-0.7.15 and GATK4 counterpart of the Parabricks command above. The output from these commands will be identical to the output from the above command. See the Output Comparison page for comparing the results.
# Run bwa-mem and pipe the output to create a sorted BAM.
$ bwa mem \
-t 32 \
-K 10000000 \
-R '@RG\tID:sample_rg1\tLB:lib1\tPL:bar\tSM:sample\tPU:sample_rg1' \
<INPUT_DIR>/${REFERENCE_FILE} <INPUT_DIR>/${INPUT_FASTQ_1} <INPUT_DIR>/${INPUT_FASTQ_2} | \
gatk SortSam \
--java-options -Xmx30g \
--MAX_RECORDS_IN_RAM 5000000 \
-I /dev/stdin \
-O cpu.bam \
--SORT_ORDER coordinate
# Mark duplicates.
$ gatk MarkDuplicates \
--java-options -Xmx30g \
-I cpu.bam \
-O mark_dups_cpu.bam \
-M metrics.txt
# Generate a BQSR report.
$ gatk BaseRecalibrator \
--java-options -Xmx30g \
--input mark_dups_cpu.bam \
--output <OUTPUT_DIR>/${OUTPUT_RECAL_FILE} \
--known-sites <INPUT_DIR>/${KNOWN_SITES_FILE} \
--reference <INPUT_DIR>/${REFERENCE_FILE}
fq2bam includes some options that can be used to tune the software for the specific hardware being used.
- --gpuwrite
- --gpusort
- --low-memory
Use one GPU to accelerate writing final BAM. (default: None)
Use GPUs to accelerate sorting and marking. (default: None)
This reduces the minimum required memory from 24 GB to 16 GB of GPU RAM, by trading slower speed for a smaller memory requirement.
Run GPU-bwa mem, co-ordinate sorting, marking duplicates, and Base Quality Score Recalibration to convert FASTQ to BAM/CRAM.
Input/Output file options
- --ref REF
- --in-fq [IN_FQ [IN_FQ ...]]
- --in-se-fq [IN_SE_FQ [IN_SE_FQ ...]]
- --in-fq-list IN_FQ_LIST
- --knownSites KNOWNSITES
- --interval-file INTERVAL_FILE
- --out-recal-file OUT_RECAL_FILE
- --out-bam OUT_BAM
- --out-duplicate-metrics OUT_DUPLICATE_METRICS
- --out-qc-metrics-dir OUT_QC_METRICS_DIR
Path to the reference file. (default: None)
Option is required.
Path to the pair-ended FASTQ files followed by optional read groups with quotes (Example: "@RGtID:footLB:lib1tPL:bartSM:sampletPU:foo"). The files must be in fastq or fastq.gz format. All sets of inputs should have a read group; otherwise, none should have a read group, and it will be automatically added by the pipeline. This option can be repeated multiple times. Example 1: --in-fq sampleX_1_1.fastq.gz sampleX_1_2.fastq.gz --in-fq sampleX_2_1.fastq.gz sampleX_2_2.fastq.gz. Example 2: --in-fq sampleX_1_1.fastq.gz sampleX_1_2.fastq.gz "@RGtID:footLB:lib1tPL:bartSM:sampletPU:unit1" --in-fq sampleX_2_1.fastq.gz sampleX_2_2.fastq.gz "@RGtID:foo2tLB:lib1tPL:bartSM:sampletPU:unit2". For the same sample, Read Groups should have the same sample name (SM) and a different ID and PU. (default: None)
Path to the single-ended FASTQ file followed by optional read group with quotes (Example: "@RGtID:footLB:lib1tPL:bartSM:sampletPU:foo"). The file must be in fastq or fastq.gz format. Either all sets of inputs have a read group, or none should have one, and it will be automatically added by the pipeline. This option can be repeated multiple times. Example 1: --in-se-fq sampleX_1.fastq.gz --in-se-fq sampleX_2.fastq.gz . Example 2: --in-se-fq sampleX_1.fastq.gz "@RGtID:footLB:lib1tPL:bartSM:sampletPU:unit1" --in-se-fq sampleX_2.fastq.gz "@RGtID:foo2tLB:lib1tPL:bartSM:sampletPU:unit2" . For the same sample, Read Groups should have the same sample name (SM) and a different ID and PU (default: None)
Path to a file that contains the locations of pair-ended FASTQ files. Each line must contain the location of two FASTQ files followed by a read group, each separated by a space. Each set of files (and associated read group) must be on a separate line. Files must be in fastq/fastq.gz format. Line syntax: <fastq_1> <fastq_2> <read group> (default: None)
Path to a known indels file. The file must be in vcf.gz format. This option can be used multiple times. (default: None)
Path to an interval file in one of these formats: Picard-style (.interval_list or .picard), GATK-style (.list or .intervals), or BED file (.bed). This option can be used multiple times. (default: None)
Path of a report file after Base Quality Score Recalibration. (default: None)
Path of a BAM/CRAM file. (default: None)
Option is required.
Path of duplicate metrics file after Marking Duplicates. (default: None)
Path of the directory where QC metrics will be generated. (default: None)
Tool Options:
- -L INTERVAL, --interval INTERVAL
- --bwa-options BWA_OPTIONS
- --no-warnings
- --gpuwrite
- --gpusort
- --low-memory
- --filter-flag FILTER_FLAG
- --skip-multiple-hits
- --min-read-length MIN_READ_LENGTH
- --align-only
- --no-markdups
- --fix-mate
- --markdups-assume-sortorder-queryname
- --markdups-picard-version-2182
- --optical-duplicate-pixel-distance OPTICAL_DUPLICATE_PIXEL_DISTANCE
- --read-group-sm READ_GROUP_SM
- --read-group-lb READ_GROUP_LB
- --read-group-pl READ_GROUP_PL
- --read-group-id-prefix READ_GROUP_ID_PREFIX
- -ip INTERVAL_PADDING, --interval-padding INTERVAL_PADDING
Interval within which to call bqsr from the input reads. All intervals will have a padding of 100 to get read records, and overlapping intervals will be combined. Interval files should be passed using the --interval-file option. This option can be used multiple times, e.g. "-L chr1 -L chr2:10000 -L chr3:20000+ -L chr4:10000-20000". (default: None)
Pass supported bwa mem options as one string. The current original bwa mem supported options are -M, -Y and -T, e.g. --bwa-options="-M -Y". (default: None)
Suppress warning messages about system thread and memory usage. (default: None)
Use one GPU to accelerate writing final BAM. (default: None)
Use GPUs to accelerate sorting and marking. (default: None)
Use low memory mode. (default: None)
Don't generate SAM entries in the output if the entry's flag's meet this criteria. Criteria: (flag & filter != 0) (default: 0)
Filter SAM entries whose length of SA is not 0. (default: None)
Skip reads below minimum read length. They will not be part of output. (default: None)
Generate output BAM after bwa-mem. The output will not be co-ordinate sorted or duplicates will not be marked. (default: None)
Do not perform the Mark Duplicates step. Return BAM after sorting. (default: None)
Add mate cigar (MC) and mate quality (MQ) tags to the output file. (default: None)
Assume the reads are sorted by queryname for Marking Duplicates. This will mark secondary, supplementary, and unmapped reads as duplicates as well. This flag will not impact variant calling while increasing processing times. (default: None)
Assume marking duplicates to be similar to Picard version 2.18.2. (default: None)
The maximum offset between two duplicate clusters in order to consider them optical duplicates. Ignored if --out-duplicate-metrics is not passed. (default: None)
SM tag for read groups in this run. (default: None)
LB tag for read groups in this run. (default: None)
PL tag for read groups in this run. (default: None)
Prefix for the ID and PU tags for read groups in this run. This prefix will be used for all pairs of fastq files in this run. The ID and PU tags will consist of this prefix and an identifier, that will be unique for a pair of fastq files. (default: None)
Amount of padding (in base pairs) to add to each interval you are including. (default: None)
Common options:
- --logfile LOGFILE
- --tmp-dir TMP_DIR
- --with-petagene-dir WITH_PETAGENE_DIR
- --keep-tmp
- --no-seccomp-override
- --version
Path to the log file. If not specified, messages will only be written to the standard error output. (default: None)
Full path to the directory where temporary files will be stored.
Full path to the PetaGene installation directory. By default, this should have been installed at /opt/petagene. Use of this option also requires that the PetaLink library has been preloaded by setting the LD_PRELOAD environment variable. Optionally set the PETASUITE_REFPATH and PGCLOUD_CREDPATH environment variables that are used for data and credentials (default: None)
Do not delete the directory storing temporary files after completion.
Do not override seccomp options for docker (default: None).
View compatible software versions.
GPU options:
- --num-gpus NUM_GPUS
Number of GPUs to use for a run. GPUs 0..(NUM_GPUS-1) will be used.
The --in-fq option takes the names of two FASTQ files, optionally followed by a quoted read group. The FASTQ filenames must not start with a hyphen.
When using the --in-fq-list option a read group is required on each line of the input file.