FASTQ AND BAM PROCESSING OVERVIEW

NVIDIA Clara Parabricks Pipelines tools that can process fastq files and refine bam files

Here are the articles in this section:

Generate BAM/CRAM output given one or more pairs of fastq files. Optionally generate BQSR report.

fq2bam performs the following steps. The user can decide to turn-off marking of duplicates. The BQSR step is only performed if the –knownSites input and –out-recal-file output options are provided.

QUICK START

            

            
$ pbrun fq2bam --ref Ref/Homo_sapiens_assembly38.fasta \
--in-fq Data/sample_1.fq.gz Data/sample_2.fq.gz  \
--knownSites Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \
--out-bam mark_dups_gpu.bam \
--out-recal-file recal_gpu.txt \
--tmp-dir /raid/myrun
        

COMPATIBLE CPU BASED BWA-MEM, GATK4 COMMANDS

The command below is the bwa-0.7.15 and GATK4 counterpart of the Parabricks command above. The output from these commands will generate the exact same results as the output from the above command. Please look at Output Comparison page on how you can compare the results.

            

            
# Run bwa-mem and pipe output to create sorted bam
$ bwa mem -t 32 -K 10000000 -R '@RG\tID:sample_rg1\tLB:lib1\tPL:bar\tSM:sample\tPU:sample_rg1' \
Ref/Homo_sapiens_assembly38.fasta Data/sample_1.fq.gz Data/sample_2.fq.gz | gatk \
SortSam --java-options -Xmx30g --MAX_RECORDS_IN_RAM=5000000 -I=/dev/stdin \
-O=cpu.bam --SORT_ORDER=coordinate --TMP_DIR=/raid/myrun

# Mark Duplicates
$ gatk MarkDuplicates --java-options -Xmx30g -I=cpu.bam -O=mark_dups_cpu.bam \
-M=metrics.txt --TMP_DIR=/raid/myrun

# Generate BQSR Report
$ gatk BaseRecalibrator --java-options -Xmx30g --input mark_dups_cpu.bam --output \
recal_cpu.txt --known-sites Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \
--reference Ref/Homo_sapiens_assembly38.fasta
        

OPTIONS

--ref

(required) The reference genome in fasta format. We assume that the indexing required to run bwa has been completed by the user.

--in-fq

(required) Pair ended fastq files. These can be in .fastq or .fastq.gz format. You can provide read group information as an optional third argument.

Example 1:

--in-fq sampleX_1_1.fastq.gz sampleX_1_2.fastq.gz

Example 2:

--in-fq sampleX_1_1.fastq.gz sampleX_1_2.fastq.gz "@RG\tID:foo\tLB:lib1\tPL:bar\tSM:sample\tPU:unit1"

This option can be repeated multiple times as well.

Example 1:

--in-fq sampleX_1_1.fastq.gz sampleX_1_2.fastq.gz --in-fq sampleX_2_1.fastq.gz sampleX_2_2.fastq.gz

Example 2:

--in-fq sampleX_1_1.fastq.gz sampleX_1_2.fastq.gz "@RG\tID:foo\tLB:lib1\tPL:bar\tSM:sample\tPU:unit1" \

--in-fq sampleX_2_1.fastq.gz sampleX_2_2.fastq.gz "@RG\tID:foo2\tLB:lib1\tPL:bar\tSM:sample\tPU:unit2"

Either all sets of inputs have read group or none should have it and will be automatically added by the pipeline. For same sample, Read Groups should have same sample name (SM) and different ID and PU.

--in-se-fq

(required) Single ended fastq files. These can be in .fastq or .fastq.gz format. You can provide read group information as an optional third argument.

Example 1:

--in-se-fq sampleX.fastq.gz

Example 2:

--in-se-fq sampleX.fastq.gz "@RG\tID:foo\tLB:lib1\tPL:bar\tSM:sample\tPU:unit1"

This option can be repeated multiple times as well.

Example 1:

--in-se-fq sampleX_1.fastq.gz --in-se-fq sampleX_2.fastq.gz

Example 2:

--in-se-fq sampleX_1.fastq.gz "@RG\tID:foo\tLB:lib1\tPL:bar\tSM:sample\tPU:unit1" \

--in-se-fq sampleX_2.fastq.gz "@RG\tID:foo2\tLB:lib1\tPL:bar\tSM:sample\tPU:unit2"

Either all sets of inputs have read group or none should have it and will be automatically added by the pipeline. For the same sample, Read Groups should have the same sample name (SM) and different ID and PU.

--in-fq-list

Path to a file that contains the locations of pair ended fastq files followed by optional read group, each separated by a space on the same line. Each set of pairs must be on a new line. Either all sets of inputs have read group or none should have it and will be automatically added by the pipeline. Files in the list can be in fastq or fastq.gz format . Line syntax: <fastq_1> <fastq_2> <optional read group> (default: None)

--out-bam

(required) Path to the file that will contain BAM or CRAM output.

--out-recal-file

Path of report file (.txt format) after Base Quality Score Recalibration. This option can be passed in, if –knownSites are provided.

--out-duplicate-metrics

Path of duplicate metrics file after Marking Duplicates.

--knownSites

Known indel files in .vcf.gz format. These should be compressed VCF files for known SNPs and indels. You can use this option multiple times. If you provide this option, then you must also provide an –out-recal-file.

--interval-file

Path to an interval file for BQSR step with possible formats:

Picard-style (.interval_list or .picard), GATK-style (.list or .intervals), or BED file (.bed). This option can be used multiple times (default: None)

--interval

(-L) Interval strings within which to call BQSR from the input reads. All intervals will have a padding of 100 to get read records and overlapping intervals will be combined. Interval files should be passed using the –interval-file option. This option can be used multiple times. e.g. “-L chr1 -L chr2:10000 -L chr3:20000+ -L chr4:10000-20000” (default: None)

--interval-padding

(-ip) Padding size (in base pairs) to add to each interval you are including (default: None)

--no-markdups

Defaults to False.

Do not mark duplicates, generate bam after co-ordinate sorting.

--bwa-options

Pass supported bwa mem options as one string. Current original bwa mem supported options: -M, -Y, -T. e.g. –bwa-options “-M -Y” (default: None)

--markdups-assume-sortorder-queryname

Defaults to False.

Assume the reads are sorted by queryname for Marking Duplicates. This will mark secondary, supplementary and unmapped reads as duplicates as well. This flag will not impact variant calling while increasing processing times.

--markdups-picard-version-2182

Assume marking duplicates to be similar to Picard version 2.18.2

--optical-duplicate-pixel-distance

The maximum offset between two duplicate clusters in order to consider them optical duplicates. This option is only used if you provide an –out-duplicate-metrics.

--out-qc-metrics-dir

Path of directory where QC metrics will be generated (default: None)

--read-group-sm

SM tag for read groups in this run.

--read-group-lb

LB tag for read groups in this run.

--read-group-pl

PL tag for read groups in this run.

--read-group-id-prefix

Prefix for ID and PU tag for read groups in this run. This prefix will be used for all pair of fastq files in this run. The ID and PU tag will consist of this prefix and an identifier which will be unique for a pair of fastq files.

--no-warnings

Suppress warning messages about system thread and memory usage.

--num-gpus NUM_GPUS

Number of GPUs to use for a run. GPUs 0..(NUM_GPUS-1) will be used. If you are using flexera, please include –gpu-devices too.

--gpu-devices GPU_DEVICES

Which GPU devices to use for a run. By default, all GPU devices will be used. To use specific GPU devices enter a comma-separated list of GPU device numbers. Possible device numbers can be found by examining the output of the nvidia-smi command. For example, using –gpu-devices 0,1 would only use the first two GPUs.

--tmp-dir TMP_DIR

Full path to the directory where temporary files will be stored.

--seccomp-override

Do not override seccomp options for docker

--with-petagene-dir WITH_PETAGENE_DIR

Full path to the PetaGene installation directory where bin/ and species/ folders are located.

--keep-tmp

Do not delete the directory storing temporary files after completion.

--license-file LICENSE_FILE

Path to license file license.bin if not in installation directory.

--version

View compatible software versions.

bqsr performs the Base Quality Score Recalibration (BQSR) in a stand alone fashion.

QUICK START

            

            
$ pbrun bqsr --ref Ref/Homo_sapiens_assembly38.fasta \
--in-bam mark_dups_gpu.bam \
--knownSites Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \
--out-recal-file recal_gpu.txt \
        

COMPATIBLE GATK4 COMMAND

The command below is the GATK4 counterpart of the Parabricks command above. The output from these commands will generate the exact same results as the output from the above command.

            

            
$ gatk BaseRecalibrator --java-options -Xmx30g --input mark_dups_gpu.bam --output \
recal_cpu.txt --known-sites Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \
--reference Ref/Homo_sapiens_assembly38.fasta
        

OPTIONS

--ref

(required) The reference genome in fasta format. We assume that the indexing required to run bwa has been completed by the user.

--in-bam

(required) Path to the input BAM/CRAM file.

--knownSites

(required) Path to a known indels file. Must be in vcf/vcf.gz format. This option can be used multiple times.

--interval-file

Path to an interval file for BQSR step with possible formats: Picard-style (.interval_list or .picard), GATK-style (.list or .intervals), or BED file (.bed). This option can be used multiple times (default: None)

--interval

(-L) Interval within which to call BQSR from the input reads. All intervals will have a padding of 100 to get read records and overlapping intervals will be combined. Interval files should be passed using the –interval-file option. This option can be used multiple times. e.g. -L chr1 -L chr2:10000 -L chr3:20000+ -L chr4:10000-20000 (default: None)

--interval-padding

(-ip) Padding size (in base pairs) to add to each interval you are including (default: None)

--out-recal-file

(required) Output BQSR Report File.

--num-gpus NUM_GPUS

Number of GPUs to use for a run. GPUs 0..(NUM_GPUS-1) will be used. If you are using flexera, please include –gpu-devices too.

--gpu-devices GPU_DEVICES

Which GPU devices to use for a run. By default, all GPU devices will be used. To use specific GPU devices enter a comma-separated list of GPU device numbers. Possible device numbers can be found by examining the output of the nvidia-smi command. For example, using –gpu-devices 0,1 would only use the first two GPUs.

--tmp-dir TMP_DIR

Full path to the directory where temporary files will be stored.

--seccomp-override

Do not override seccomp options for docker

--with-petagene-dir WITH_PETAGENE_DIR

Full path to the PetaGene installation directory where bin/ and species/ folders are located.

--keep-tmp

Do not delete the directory storing temporary files after completion.

--license-file LICENSE_FILE

Path to license file license.bin if not in installation directory.

--version

View compatible software versions.

applybqsr updates the Base Quality Scores using the BQSR report.

QUICK START-CLI

            

            
$ pbrun applybqsr --ref Ref/Homo_sapiens_assembly38.fasta \
--in-bam mark_dups_gpu.bam \
--in-recal-file recal_gpu.txt  \
--out-bam S1_updated.bam \
        

COMPATIBLE GATK4 COMMAND

The command below is the GATK4 counterpart of the Parabricks command above. The output from these commands will generate the exact same results as the output from the above command.

            

            
$ gatk ApplyBQSR --java-options -Xmx30g -R Ref/Homo_sapiens_assembly38.fasta \
-I=mark_dups_gpu.bam --bqsr-recal-file=recal_cpu.txt  -O=S1_updated.bam
        

OPTIONS

--ref

(required) The reference genome in fasta format. We assume that the indexing required to run bwa has been completed by the user.

--in-bam

(required) Path to the input BAM/CRAM file.

--in-recal-file

(required) Input BQSR Report File.

--out-bam

(required) Path to the output BAM/CRAM file.

--interval-file

Path to an interval file for BQSR step with possible formats: Picard-style (.interval_list or .picard), GATK-style (.list or .intervals), or BED file (.bed). This option can be used multiple times (default: None)

--interval

(-L) All intervals will have a padding of 100 to get read records and overlapping intervals will be combined. Interval files should be passed using the –interval-file option. This option can be used multiple times. e.g. -L chr1 -L chr2:10000 -L chr3:20000+ -L chr4:10000-20000 (default: None)

--interval-padding

(-ip) Padding size (in base pairs) to add to each interval you are including (default: None)

--num-threads

Defaults to 8. Number of threads for worker.

--num-gpus NUM_GPUS

Number of GPUs to use for a run. GPUs 0..(NUM_GPUS-1) will be used. If you are using flexera, please include –gpu-devices too.

--gpu-devices GPU_DEVICES

Which GPU devices to use for a run. By default, all GPU devices will be used. To use specific GPU devices enter a comma-separated list of GPU device numbers. Possible device numbers can be found by examining the output of the nvidia-smi command. For example, using –gpu-devices 0,1 would only use the first two GPUs.

--tmp-dir TMP_DIR

Full path to the directory where temporary files will be stored.

--seccomp-override

Do not override seccomp options for docker

--with-petagene-dir WITH_PETAGENE_DIR

Full path to the PetaGene installation directory where bin/ and species/ folders are located.

--keep-tmp

Do not delete the directory storing temporary files after completion.

--license-file LICENSE_FILE

Path to license file license.bin if not in installation directory.

--version

View compatible software versions.

Run bam2fq to convert bam/cram to fastq

QUICK START-CLI

            

            
$ pbrun bam2fq --ref Ref/Homo_sapiens_assembly38.fasta \
--in-bam sample.bam \
--out-fq1 sample_1.fastq.gz  \
--out-fq2 sample_2.fastq.gz
        

COMPATIBLE GATK4 COMMAND

The command below is the GATK4 counterpart of the Parabricks command above. The output from these commands will generate the exact same results as the output from the above command.

            

            
$ gatk SamToFastq -I sample.bam \
-F sample_1.fastq.gz \
-F2 sample_2.fastq.gz
        

OPTIONS

--ref

(required) The reference genome in fasta format. We assume that the indexing required to run bwa has been completed by the user.

--in-bam

(required) Path to the input BAM/CRAM file.

--out-fq1

(required) Path of first fastq output file. Must be in gz format (default: None)

--out-fq2

(required) Path of second fastq output file. Must be in gz format (default: None)

--num-threads

Defaults to 6. Number of threads for worker.

--tmp-dir TMP_DIR

Full path to the directory where temporary files will be stored.

--seccomp-override

Do not override seccomp options for docker

--with-petagene-dir WITH_PETAGENE_DIR

Full path to the PetaGene installation directory where bin/ and species/ folders are located.

--keep-tmp

Do not delete the directory storing temporary files after completion.

--license-file LICENSE_FILE

Path to license file license.bin if not in installation directory.

--version

View compatible software versions.