Deploy NVIDIA Parabricks

Get started by deploying Parabricks using a Docker image. Then you can run and customize Parabricks.

Getting the Parabricks Docker Image

Run the following command to obtain the image:

$ docker pull nvcr.io/nvidia/clara/clara-parabricks:4.8.0-1

To confirm which version of Parabricks the image contains, run pbrun version:

$ docker run --rm --gpus all nvcr.io/nvidia/clara/clara-parabricks:4.8.0-1 pbrun version

The last line of the output shows the version, in the form pbrun: <version>.

Running NVIDIA Parabricks

You can run Parabricks using either the command line or the base command platform.

Using the Command Line to Run Parabricks

After deploying Parabricks using a Docker image, you can begin customizing it. There are two parts to customizing a Parabricks run:

• Customizing Docker container specific options: These are the options that are passed to the docker command before the name of the container. For example, the user should mount their data directories within the Docker container by passing the -v option to Docker. Refer to the Tutorials for more detailed examples.

• Parabricks specific options: These options are passed to the Parabricks command line to customize the Parabricks run. For example, you can choose which tool to run and pass tool-specific options.

For example, use the following command to run the Parabricks fq2bam (BWA-MEM + GATK) tool using a Docker container. Refer to the tutorial for more information on how this command works.

$ docker run \
      --gpus all \
      --rm \
      --volume $(pwd):/workdir \
      --volume $(pwd):/outputdir \
    nvcr.io/nvidia/clara/clara-parabricks:4.8.0-1 \
    pbrun fq2bam \
      --ref /workdir/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
      --in-fq /workdir/parabricks_sample/Data/sample_1.fq.gz /workdir/parabricks_sample/Data/sample_2.fq.gz \
      --out-bam /outputdir/fq2bam_output.bam

Sample data is free and available. Refer to the Getting the Sample Data section in the Tutorials for instructions on obtaining the sample data and a step-by-step guide to using both fq2bam and Haplotype Caller.

Some useful Docker options to consider:

• --gpus all lets the Docker container use all the GPUs on the system. The GPUs available to Parabricks container can be limited using the --gpus "device=<list of GPUs>" option. Use nvidia-smi to see how many GPUs you have, and which one is which.

• --rm tells Docker to terminate the image once the command has finished.

• --volume $(pwd):/workdir and --volume $(pwd):/outputdir mount your current directory (a path on the server) in the Docker container at /workdir and /outputdir (paths inside the Docker container). To use other directories, replace $(pwd) with their paths, for example --volume /path/to/your/data:/workdir. Refer to Container Contract for how the commands in this documentation use the two mounts.

• --workdir tells Docker what working directory to execute the commands from (inside the container).

• The rest of the command is the Parabricks tool you want to run, followed by its arguments. For those familiar with pre-v4.0 versions of Parabricks and its pbrun command, this Docker invocation takes the place of pbrun.

Container Contract

The docker run commands in this documentation share the following conventions and defaults. Keep them in mind when you adapt a command to your own data.

Mounts: /workdir and /outputdir aren’t directories in the image. The commands mount host directories at those paths, and most mount your current directory at both, with --volume $(pwd):/workdir --volume $(pwd):/outputdir, so the two paths are the same directory. Inputs are read from /workdir and outputs are written to /outputdir. Multistep examples depend on this: a later step reads from /workdir a file that an earlier step wrote to /outputdir. If you mount different host directories at the two paths, update the paths in later steps to match.

File ownership: The container runs as root by default, so the files and directories it creates in your mounted directories are owned by root. You can’t edit those files, or remove those directories, without root privileges. To own the output yourself, run the container as your own user and group by adding --user $(id -u):$(id -g). Parabricks writes temporary files to the working directory by default, and a non-root user can’t write to the container’s default working directory, /, so also set --workdir to a mounted directory, or point --tmp-dir at one. Otherwise, the run stops with Could not create temporary directory. For example, this runs the fq2bam command from above as your own user:

$ docker run \
      --gpus all \
      --rm \
      --user $(id -u):$(id -g) \
      --volume $(pwd):/workdir \
      --volume $(pwd):/outputdir \
      --workdir /workdir \
    nvcr.io/nvidia/clara/clara-parabricks:4.8.0-1 \
    pbrun fq2bam \
      --ref /workdir/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
      --in-fq /workdir/parabricks_sample/Data/sample_1.fq.gz /workdir/parabricks_sample/Data/sample_2.fq.gz \
      --out-bam /outputdir/fq2bam_output.bam

Tools in the image: Besides pbrun, the image includes samtools and bcftools, but not bgzip or tabix. Use bcftools to compress a VCF with BGZF or to index it, for example when you prepare a panel of normals for prepon or postpon. To run samtools or bcftools, pass its name to --entrypoint and put its arguments after the image name:

# Compress a VCF with BGZF, in place of bgzip
$ docker run \
      --rm \
      --user $(id -u):$(id -g) \
      --volume $(pwd):/workdir \
      --workdir /workdir \
      --entrypoint bcftools \
    nvcr.io/nvidia/clara/clara-parabricks:4.8.0-1 \
    view --output-type z --output /workdir/${INPUT_VCF}.gz /workdir/${INPUT_VCF}
# Create a .tbi index next to the compressed VCF, in place of tabix
$ docker run \
      --rm \
      --user $(id -u):$(id -g) \
      --volume $(pwd):/workdir \
      --workdir /workdir \
      --entrypoint bcftools \
    nvcr.io/nvidia/clara/clara-parabricks:4.8.0-1 \
    index --tbi /workdir/${INPUT_VCF}.gz

Running Parabricks Using the Base Command Platform

An example command to launch a BaseCommand container on a single-GPU instance is:

ngc batch run --name "parabricks-germline" \
    --instance dgxa100.80g.1.norm \
    --commandline "pbrun germline \
        --ref /workspace/parabricks_sample/Ref/Homo_sapiens_assembly38.fasta \
        --in-fq /Data/HG002-NA24385-pFDA_S2_L002_R1_001-30x.fastq.gz /Data/HG002-NA24385-pFDA_S2_L002_R2_001-30x.fastq.gz \
        --knownSites /workspace/parabricks_sample/Ref/Homo_sapiens_assembly38.known_indels.vcf.gz \
        --out-bam output.bam \
        --out-variants output.vcf \
        --out-recal-file report.txt \
        --run-partition \
        --no-alt-contigs" \
    --result /results \
    --image "nvcr.io/nvidia/clara/clara-parabricks:4.8.0-1"