Parabricks DeepVariant NIM

Important

NVIDIA NIM currently is in limited availability, sign up here to get notified when the latest NIMs are available to download.

Parabricks DeepVariant (the tool behind the Universal Variant Calling Microservice) is a deep learning model that can help identify variants in short- and long-read sequencing datasets. The model is packaged in the container, therefore there is no need to explicitly pull the model as is common with other NIMs.

After alignment, the next step in the genomic analysis is variant calling. This NIM allows for efficiently setting up a microservice capable of scheduling multiple jobs to run one after another. This is especially handy for running large quantities of data, as the NIM ensures that when one job finishes, the next one starts, thus maximizing throughput. In addition, the NIM runs on an open port, so it can be configured such that multiple users can submit jobs to the same queue, thus creating a centralized compute environment. Thirdly, NIMs can be combined. For example, the output of the Parabricks FQ2BAM NIM can be used as the input for this NIM, thus enabling end-to-end workflows.

For more information on the Parabricks DeepVariant workflow, see the Parabricks documentation.

Specific Requirements

Operating System

• Linux(x86_64/amd64): Linux distributions supported by the NVIDIA Container Toolkit.

Hardware

• Hopper, Ampere, Ada GPUs with Minimum 24GB of GPU (VRAM) Memory

Software

• Minimum Driver version: 535.161.07

Quickstart Guide

Note

This page assumes Prerequisite Software (Docker, NGC CLI, NGC registry access) is installed and set up.

1. Pull the NIM container.

   docker pull nvcr.io/nvidia/nim/universal-variant-calling:24.03.01
   

2. Run NIM

   docker run --rm -d \
      --volume /etc/passwd:/etc/passwd:ro \
      --volume /etc/group:/etc/group:ro \
      --volume /etc/shadow:/etc/shadow:ro \
      --volume $PWD:/workspace \
      --shm-size=2g \
      --ulimit memlock=-1 \
      --ulimit stack=67108864 \
      --gpus all \
      --p 8003:8003  \
      --name universal-variant-calling \
      nvcr.io/nvidia/nim/universal-variant-calling:24.03.01
   

3. Download the data.

   mkdir test-data
   cd test-data
   wget https://s3.amazonaws.com/parabricks.sample/nim/References_chr22.fasta.tar
   wget https://s3.amazonaws.com/parabricks.sample/nim/test-datachr22.bam
   wget https://s3.amazonaws.com/parabricks.sample/nim/test-datachr22.bam.bai
   cd ..
   mkdir results
   

4. Wait until the health check returns true before proceeding.

   curl localhost:8003/v2/health/ready
   

5. Run universal variant calling from the local NIM instance.

   curl -X 'POST' \
   'http://localhost:8003/genomics/parabricks/universal-variant-calling/run' \
   -H 'accept: application/json' \
   -H 'Content-Type: application/json' \
   -d '{
      "additional_args": "--tmp-dir /workspace",
      "in_bai": "/workspace/test-data/test-datachr22.bam.bai",
      "in_bam": "/workspace/test-data/test-datachr22.bam",
      "in_ref_tarball": "/workspace/test-data/References_chr22.fasta.tar",
      "mode": "shortread",
      "out_stderr": "/workspace/results/test-datachr22.stderr",
      "out_stdout": "/workspace/results/test-datachr22.stdout",
      "out_variants": ["/workspace/results/test-datachr22.vcf"],
      "out_variants_parts_manifest": "/workspace/results/test-datachr22.bam.parts_manifest.txt",
      "use_gvcf": "False",
      "use_wes_model": "False"
   }'
   

Detailed Instructions

In this workflow, we will submit a ‘.bam’ file (Chromosome 22 from HG002 sampled at 100x) for variant calling. The microservice will use Parabricks DeepVariant to perform the variant calling and return a .vcf file.

The Genomics NIMs take either local file paths or pre-signed URLs. A local file path is one that is mounted on the computer on which the server is running. This could be locally attached storage, an NFS share, etc and looks like /path/to/file/name.ext. A pre-signed URL looks like https://s3.us-east-1.amazonaws.com/your.bucket/path/to/file.ext?etcetcetc.

If a local file path is given in the job request then it must be a path as seen from inside the server. To specify a local file path correctly you’ll need to know what local directories were made available to the server when it was started and where they were mounted. If the microservice is started as follows:

docker run --rm -it \
   ...
   --volume $PWD:/workspace \
   ...

then the --volume $PWD:/workspace line makes the current directory ($PWD) available as the /workspace directory inside the server. If a BAM file is found in $PWD/my_input_files/batch1/filename.bam on the server then the cURL command (see below for the complete command) would need to be:

curl -X 'POST' \
'http://localhost:8003/genomics/parabricks/universal-variant-calling/run' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
   "in_bam": "/workspace/my_input_files/batch1/filename.bam",
   ...
   }'

For remote files it doesn’t matter what directories were mounted on the server. Specify a pre-signed URL like this:

curl -X 'POST' \
'http://localhost:8003/genomics/parabricks/universal-variant-calling/run' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
   "in_bai": "https://s3.us-east-1.amazonaws.com/your.bucket/path/to/file.ext?etcetcetc",
   ...
   }'

Requests can mix local and remote file paths at will. For example, you might read all your input files from S3 and write all your output files to a local drive.

Pull Container Image

1. Container image tags can be retrieved using the following command.

   ngc registry image info nvcr.io/nvidia/nim/universal-variant-calling
   

   Image Repository Information
      Name: universal-variant-calling
      Display Name: DeepVariant
      Short Description: Run a GPU-accelerated DeepVariant algorithm, with accelerated models supporting data from all major sequencing platforms.
      Built By:
      Publisher:
      Multinode Support: False
      Multi-Arch Support: False
      Logo: https://nvdam.widen.net/content/jmiqcpgjsx/jpeg/m48-dna-256px-grn.jpeg
      Labels: Genome Sequencing, Healthcare, Life Sciences, NVIDIA AI Enterprise Supported, Clara Parabricks, Genomics
      Public: No
      Access Type:
      Associated Products: []
      Last Updated: Mar 15, 2024
      Latest Image Size: 2.41 GB
      Signed Tag?: False
      Latest Tag: 24.03.01
      Tags:
           24.03.01
   

2. Pull the container image

   Docker

   docker pull nvcr.io/nvidia/nim/universal-variant-calling:24.03.01
   

   NGC

   ngc registry image pull nvcr.io/nvidia/nim/universal-variant-calling:24.03.01
   

Launch the Microservice

Launch the container. Start-up may take a couple of minutes but the logs will say Application startup complete. when the service is available.

docker run --rm -it \
   --volume /etc/passwd:/etc/passwd:ro \
   --volume /etc/group:/etc/group:ro \
   --volume /etc/shadow:/etc/shadow:ro \
   --volume $PWD:/workspace \
   --shm-size=2g \
   --ulimit memlock=-1 \
   --ulimit stack=67108864 \
   --gpus all \
   -p 8003:8003 \
   --name universal-variant-calling \
   nvcr.io/nvidia/nim/universal-variant-calling:24.03.01

The current directory ($PWD) will be accessible inside the container as the “/workspace” directory. Subsequent example commands will put input files in this directory; output files will also be written here. If you wish to place your files elsewhere change $PWD to the desired path.

Flag	Description
--name	Sets the name of the container.
--gpus all	Tells Docker to use all the GPUs on the system.
--shm-size=2G	Increases the maximum shared memory size from the default of 64MB to 2GB.
--volume <host>:<docker>	Mounts directories from the host machine into Docker.
--ulimit memlock=-1	A value of -1 means there will be no limit on the memory that will not be paged out. All other values represent memory limit in kilobytes. The microservice will use a lot of memory so we recommend setting this to unlimited.
--ulimit stack=67108864	Sets the maximum stack size in bytes. We do not recommend lowering this value.
-p 8003:8003	Forwards the ports from the host machine into Docker. This NIM requires port 8003 to be open.

Health and Liveness checks

The container exposes health and liveness endpoints for integration into existing systems such as Kubernetes at /v2/health/ready and /v2/health/live. These endpoints only return an HTTP 200 OK status code if the service is ready or live, respectively.

Run these in a new terminal.

curl localhost:8003/v2/health/ready
true

curl localhost:8003/v2/health/live
true

Download Sample Data

1. Download a small reference file. The FASTA file and all its associated indices must be packaged into a single .tar file. This is the same .tar file downloaded for the FQ2BAM example.

   mkdir test-data
   cd test-data
   wget https://s3.amazonaws.com/parabricks.sample/nim/References_chr22.fasta.tar
   

2. Download the BAM (Binary Alignment Map) file and its index.

   wget https://s3.amazonaws.com/parabricks.sample/nim/test-datachr22.bam
   wget https://s3.amazonaws.com/parabricks.sample/nim/test-datachr22.bam.bai
   cd ..
   mkdir results
   

3. The current working directory should have at least these files:

   tree
   ├── results
   └── test-data
       ├── test-datachr22.bam
       ├── test-datachr22.bam.bai
       └── References_chr22.fasta.tar
   

Run Variant Calling

1. Running DeepVariant using an example CURL command. It is assumed that the folder containing input data files is mounted via docker run while launching the container.

   #!/bin/bash
   
   # Script: uvc_curl.sh
   # Usage: sh uvc_curl.sh
   # Doc/schemas: http://localhost:8003/docs#/Default/parabricks_universal_variant_calling
   
   curl -X 'POST' \
   'http://localhost:8003/genomics/parabricks/universal-variant-calling/run' \
   -H 'accept: application/json' \
   -H 'Content-Type: application/json' \
   -d '{
      "additional_args": "--tmp-dir /workspace",
      "in_bai": "/workspace/test-data/test-datachr22.bam.bai",
      "in_bam": "/workspace/test-data/test-datachr22.bam",
      "in_ref_tarball": "/workspace/test-data/References_chr22.fasta.tar",
      "mode": "shortread",
      "out_stderr": "/workspace/results/test-datachr22.stderr",
      "out_stdout": "/workspace/results/test-datachr22.stdout",
      "out_variants": ["/workspace/results/test-datachr22.vcf"],
      "out_variants_parts_manifest": "/workspace/results/test-datachr22.bam.parts_manifest.txt",
      "use_gvcf": "False",
      "use_wes_model": "False"
   }'
   

   Below is a description of what each flag in the payload does, and how it can be customized to any workflow:

   Flag	Description
   additional_args	Other flags found in the documentation can be passed through here as a string. For example, "--tmp-dir /workspace" passes that flag directly to the Parabricks run command inside the NIM.
   in_bai	Path to the index for the index file for the .bam file.
   in_bam	Path to the .bam file.
   in_ref_tarball	Reference tarball in the same format as the sample data ‘References_chr22.fasta.tar’. Untar to see associated files.
   mode	Value can be one of [shortread, pacbio, ont]. By default, it is shortread. If mode is set to pacbio, the following defaults are used: –norealign-reads, –alt-aligned-pileup diff_channels, –vsc-min-fraction-indels 0.12. If mode is set to ont, the following defaults are used: -norealign-reads, –variant-caller VCF_CANDIDATE_IMPORTER.
   out_stderr	Path for output of stderr during the job.
   out_stdout	Path for output of stdout during the job.
   out_variants	Path to the output .vcf file.
   out_variants_parts_manifest	Path to the parts manifest for the variants.
   use_gvcf	Generate variant calls in .gvcf format.
   use_wes_model	If passed, the WES model file will be used. Only used in shortread mode.

   When the job is finished, the terminal will do the following:

   {"out-stdout":"/workspace/results/test-datachr22.stdout","out-stderr":"/workspace/results/test-datachr22.stderr","out-variants-parts-manifest":"/workspace/results/test-datachr22.bam.parts_manifest.txt"}
   

2. You can submit a request using a Python script. Similar to the example above, it is assumed that the folder containing input data files is mounted via docker run while launching the container.

   #!/usr/bin/env python3
   
   # Script: uvc_request.py
   # Usage: python3 uvc_rquest.py
   # Model doc/Schemas:  http://localhost:8003/docs#/Default/parabricks_universal_variant_calling
   
   import requests
   import json
   
   invoke_url = "http://localhost:8003/genomics/parabricks/universal-variant-calling/run"
   
   headers = {
      "accept": "application/json",
      "content-type": "application/json",
   }
   
   # The /workspace directory referred to here is the /workspace directory within container.
   # Files mounted during the docker launch are present in /workspace directory.
   
   data = {
      "in_ref_tarball": "/workspace/test-data/References_chr22.fasta.tar",
      "in_bam": "/workspace/test-data/test-datachr22.bam",
      "in_bai": "/workspace/test-data/test-datachr22.bam.bai",
      "out_variants": ["/workspace/results/test-datachr22.vcf"],
      "out_variants_parts_manifest": "/workspace/results/test-datachr22.bam.parts_manifest.txt",
      "mode": "shortread",
      "use_gvcf" : "False",
      "use_wes_model" : "False",
      "out_stderr": "/workspace/results/test-datachr22.stderr",
      "out_stdout": "/workspace/results/test-datachr22.stdout",
      "additional_args": ""
   }
   
   resp = requests.post(invoke_url, json=data)
   
   # The status of the request should be '200' if it is processed
   print("Request status code: ", resp.status_code)
   

   See the above table for a description of the arguments.

3. Please note that it takes 30 seconds for the request to process. When it succeeds, the terminal will output the following:

   Request status code:  200
   

   The outputs will be in the results folder:

   ls results/
   test-datachr22.vcf
   test-datachr22.stderr
   test-datachr22.stdout
   test-datachr22.bam.parts_manifest.txt
   

Stopping the Container

When you’re done testing the endpoint, you can bring down the container by running docker stop universal-variant-calling in a new terminal.