GETTING STARTED

Get NVIDIA Clara Parabricks Pipelines up and running on a server and start using it in 10 minutes.

There are two types of Parabricks installation licenses:

• Node Lock licenses are tied to a specific set of GPUs on a server.

• Flexera licenses allow for a set amount of GPUs to be used at once.

Follow any specific steps for the license type you are using.

STEP 1: Make sure installation requirements are met

PREREQUISITES

The following are required to install Parabricks:

• Access to the internet

• nvidia-driver that supports cuda-9.0 or higher

• nvidia-driver that supports cuda-10.0 or higher if you want to run deepvariant or cnnscorevariants

• nvidia-docker or singularity version 2.6.1 or higher

• Python 3

• curl (Most Linux systems will already have this installed)

The following are the hardware requirements:

Any GPU that supports CUDA architecture 60, 61, 70, 75 and has 12GB GPU RAM or more. It has been tested on NVIDIA P100, NVIDIA V100, NVIDIA A100, and NVIDIA T4 GPUs.

• 2 GPU server should have 100GB CPU RAM, at least 24 CPU threads

• 4 GPU server should have 196GB CPU RAM, at least 32 CPU threads

• 8 GPU server should have 392GB CPU RAM, at least 48 CPU threads

(Flexera License Only): Setting up a license server

Install the NVIDIA License Server to your system:

            

            
# Step 1: Update Installation Packages.
$ sudo apt-get update

# Step 2: Install Java Runtime Environment.
$ sudo apt install -y default-jre

# Step 3: Install and Run Apache Tomcat Server.
$ sudo apt install -y tomcat8
$ sudo systemctl enable tomcat8.service
$ sudo systemctl start tomcat8.service

# Step 4: Unzip the NVIDIA License Server.
$ sudo apt install -y unzip
$ unzip NVIDIA-ls-linux-xxxx.xx.x.xxxxxxxx.zip

# Step 5: Run the NVIDIA License Server Setup.
$ cd NVIDIA-ls-linux-xxxx.xx.x.xxxxxxxx
$ chmod +x setup.bin
$ sudo ./setup.bin
        

After installation, access the license server management interface in a web browser by using the format of the base URL:

scheme

The URL scheme, which is either http or https.

hostname

The hostname of the license server, which can be a fully-qualified domain name such as gridlicense1.example.com, or an IP address such as 10.31.20.45.

port

The port number through which the license server is accessed. By default, the http scheme has a port of 8080.

Once the license server management interface is opened, navigate to License Management in the License Server panel. From here, a flexera license file can be uploaded to the server.

A proper installation will display the message:

STEP 2: Downloading installation package

Request NVIDIA Parabricks access from https://developer.nvidia.com/clara-parabricks to get an installation package for your GPU server.

STEP 3: Install the Parabricks suite

Install the Parabricks package to your system:

            

            
# Step 1: Unzip the package.
$ tar -xzf parabricks.tar.gz

# Step 2 (Node Lock License): Run the installer.
$ sudo ./parabricks/installer.py

# Step 2 (Flexera License): Run the installer replacing [hostname] with the hostname of the license server.
$ sudo ./parabricks/installer.py --flexera-server [hostname]:7070

# Step 3: verify your installation.
# This should display the parabricks version number:
$ pbrun version
        

After installation the pbrun executable will start any tool in the Parabricks software suite. During installation you can choose to create a link at /usr/bin/pbrun to make it available for system wide access. Otherwise you can access pbrun from your local installation directory (default: /opt/parabricks/pbrun).

            

            
$ sudo ./parabricks/installer.py --ampere
        

INSTALLING WITH SINGULARITY 3.0 OR HIGHER

            

            
1. Log in to a machine with sudo access and singularity 3.0 or higher.
2. To download the software:
   $ wget -O parabricks.tar.gz " < DOWNLOAD_LINK > "
3. Unzip the package
   $ tar -xvzf parabricks.tar.gz
4. Do a local installation:$ mkdir localdir
   $ sudo ./parabricks/installer.py --install-location localdir --container singularity
5. Everything will be installed in localdir/parabricks folder. Tar this folder.
   $ cd localdir
   $ tar -cvzf parabricks_install.tar.gz parabricks
6. and copy it to the node which will be used for testing and has singularity v3.x.
7. Untar the tar.gz file in some <INSTALL_DIR>$ cd <INSTALL_DIR>
   $ tar -xvzf parabricks_install.tar.gz
        

STEP 4: Example run

# Run the fq2bam tool, which aligns, co-ordinate sorts and marks duplicates # in a pair-ended fastq file. Ref.fa is the bwa-indexed reference file

            

            
$ pbrun fq2bam --ref Ref.fa --in-fq sample_1.fq.gz sample_2.fq.gz --out-bam output.bam
        

You can download a sample dataset using the following command:

            

            
$ wget -O parabricks_sample.tar.gz \
"https://s3.amazonaws.com/parabricks.sample/parabricks_sample.tar.gz?Expires=1613069864&Signature=WxLeyitbvR%2B0rO4MX%2B0GohDw89g%3D&AWSAccessKeyId=AKIAJGDUNN2G2ZAH3Q3A"
        

To run the sample dataset:

            

            
$ tar -xvzf parabricks_sample.tar.gz
$ /parabricks/pbrun fq2bam --ref parabricks_sample/Ref/Homo_sapiens_assembly38.fasta --in-fq parabricks_sample/Data/sample_1.fq.gz parabricks_sample/Data/sample_2.fq.gz --out-bam output.bam
        

The above test should take under 250 seconds on a 4 V100 GPU system

The goal of Parabricks software is to get the highest performance for bioinformatics and genomic analysis. There are a few key system options that a user can tune to achieve maximum performance.

Use a fast local SSD for files

Parabricks software operates with two kinds of files:

• Input/output files specified by the user

• Temporary files created during execution and deleted at the end

THe best performance is achieved when both kinds of files are on a fast local SSD.

If this is not possible the input/output files can be placed on a fast network storage device and the temporary files placed on the local SSD by using the –tmp-dir option.

DGX Users

A DGX comes with a SSD, usually mounted on /raid. Please use that disk and use a directory on that disk as –tmp-dir. For initial testing you can even copy the Input files to this disk to eliminate variability in performance.

Specifying GPUs to use

To select specific GPUs to run on, please use gpu-devices and num-gpus options

            

            
$ pbrun fq2bam --ref Ref.fa --in-fq S1_1.fastq.gz --in-fq S1_2.fastq.gz --gpu-devices 0,1  --num-gpus 2
        

Parabricks software can be run as a single tool or as a full pipeline. This page will introduce how to run the software either way. The first option to pbrun is either the name of the tool or the full pipeline to run.

The details of the tools can be found at Tools Overview and the details of the pipelines can be found at Pipeline Overview. Performance optimizations techniques can be found on the Performance Tuning page.

Running a standalone tool

The example below shows how you can run the fq2bam tool.

            

            
# Run the fq2bam tool, which aligns, co-ordinate sorts and marks duplicates
# in a pair-ended fastq file. Ref.fa is the bwa-indexed reference file

$ pbrun fq2bam --ref Ref.fa --in-fq sample_1.fq.gz sample_2.fq.gz --out-bam output.bam
        

Running a pipeline

The example below shows how you can run the germline pipeline.

            

            
# Run the germline pipeline for sample_1.fq.gz sample_2.fq.gz to generate
# variant calls using Ref.fa which is the bwa-indexed reference file.

$ pbrun germline --ref Ref.fa --in-fq sample_1.fq.gz sample_2.fq.gz --out-bam output.bam \
--knownSites dbsnp_146.vcf.gz --out-recal-file recal.txt --out-variants result.vcf
        

Many users want to compare output generated by Parabricks software with other standard tools. We recommend the following way to compare output generated by Parabricks software and the counterpart non-accelerated software.

BAM COMPARISON

GATK4 sorts the SAM files based on QNAME, FLAG, RNAME, POS, MAPQ, MRNM/RNEXT, MPOS/PNEXT, and ISIZE. If all these fields are the same for two different SAMs they are considered equal for sorting purposes. Therefore, the way that we compare two sorted BAMs is by using the BamUtil diff tool to compare these fields, and there should be no difference reported.

            

            
$ bam diff --in1 mark_dups_gpu.bam --in2 mark_dups_cpu.bam --noCigar --isize --flag --mate --mapQual
        

The output of this comparison should result in no differences.

BQSR REPORT COMPARISON

The files generated by Parabricks and GATK4 should be exactly the same. There should be no output of the following command

            

            
$ diff -w recal_gpu.txt recal_cpu.txt
        

VCF COMPARISON

To compare VCF files we use the GATK Concordance tools to get sensitivity and specificity of SNPs and INDELs. When the following command is run, variant accuracy results will be stored in out.txt

            

            
$ gatk Concordance --evaluation result_gpu.vcf --truth result_cpu.vcf --summary out.txt