pangenome_germline

Run a GPU-accelerated end-to-end germline pipeline from FASTQ to VCF using pangenome alignment (Giraffe) and Pangenome-aware DeepVariant.

Refer to the pangenome_germline Reference section for a detailed listing of all available options.

What is the Pangenome Germline Pipeline?

The pangenome germline pipeline combines two tools into a single end-to-end workflow:

1. Giraffe (giraffe (vg giraffe + GATK)) — aligns short reads to a pangenome graph, producing a coordinate-sorted, duplicate-marked BAM file.

2. Pangenome-aware DeepVariant (pangenome_aware_deepvariant) — calls variants from the aligned BAM using a pangenome-aware model, producing a VCF file.

This pipeline is inspired by the vgteam Giraffe-DeepVariant WDL workflow, but uses Pangenome-aware DeepVariant instead of standard DeepVariant for improved variant calling accuracy in complex and highly variable genomic regions.

Generating Required Files

The pangenome germline pipeline requires the following input files:

For Giraffe (alignment):

• .gbz — pangenome graph
• .dist — distance index
• .min — minimizer index
• .zipcodes — zipcodes index
• .paths.sub — filtered reference paths

For Pangenome-aware DeepVariant (variant calling):

• .fa — reference FASTA extracted from the graph
• .fa.fai — FASTA index

The reference FASTA (--ref) must be derived from the same pangenome graph used for alignment (--gbz-name) and must match the contigs listed in the reference paths file (--ref-paths).

The index files can be generated from a GBZ graph using vg autoindex, and the reference FASTA can be extracted from the graph using vg paths. The following example uses the HPRC v1.1 Minigraph-Cactus pangenome graph aligned to GRCh38:

1
# Download GBZ
2
# https://s3-us-west-2.amazonaws.com/human-pangenomics/pangenomes/freeze/freeze1/minigraph-cactus/hprc-v1.1-mc-grch38/hprc-v1.1-mc-grch38.d9.gbz
3
aws s3 cp \
4
    s3://human-pangenomics/pangenomes/freeze/freeze1/minigraph-cactus/hprc-v1.1-mc-grch38/hprc-v1.1-mc-grch38.d9.gbz \
5
    . \
6
    --no-sign-request
7
8
# Extract index files from GBZ
9
docker run --rm --volume $(pwd):/workdir \
10
    --workdir /workdir \
11
    --user $(id -u):$(id -g) \
12
    quay.io/vgteam/vg:v1.70.0 \
13
    vg autoindex \
14
        -p hprc-v1.1-mc-grch38.d9.autoindex.1.70 \
15
        -G hprc-v1.1-mc-grch38.d9.gbz \
16
        -w giraffe
17
18
# Extract paths from GBZ
19
docker run --rm \
20
    --user $(id -u):$(id -g) \
21
    --volume $(pwd):/workdir \
22
    --workdir /workdir \
23
    quay.io/vgteam/vg:v1.70.0 \
24
    vg paths -x hprc-v1.1-mc-grch38.d9.gbz \
25
    -L --paths-by GRCh38 > hprc-v1.1-mc-grch38.d9.paths
26
27
# Filter paths to keep only chromosomes
28
grep -v _decoy hprc-v1.1-mc-grch38.d9.paths \
29
    | grep -v _random \
30
    | grep -v chrUn_ \
31
    | grep -v chrEBV \
32
    | grep -v chrM \
33
    | grep -v chain_ > hprc-v1.1-mc-grch38.d9.paths.sub
34
35
# Extract the sequences corresponding to the list of paths to a FASTA file
36
docker run --rm \
37
    --user $(id -u):$(id -g) \
38
    --volume $(pwd):/workdir \
39
    --workdir /workdir \
40
    quay.io/vgteam/vg:v1.70.0 \
41
    vg paths -x hprc-v1.1-mc-grch38.d9.gbz \
42
        -p hprc-v1.1-mc-grch38.d9.paths.sub \
43
        -F > hprc-v1.1-mc-grch38.d9.fa
44
45
# Index the FASTA file
46
docker run --rm \
47
    --user $(id -u):$(id -g) \
48
    --volume $(pwd):/workdir \
49
    --workdir /workdir \
50
    quay.io/biocontainers/samtools:1.17--hd87286a_2 \
51
    samtools faidx hprc-v1.1-mc-grch38.d9.fa

Reference Verification

The pipeline provides an optional pre-flight verification step enabled by the --run-ref-verification flag. When enabled, the pipeline checks the consistency of the reference FASTA against the GBZ graph before alignment begins.

The verification checks that:

• Every contig listed in --ref-paths is present in the reference FASTA.
• Every contig listed in --ref-paths is present in the GBZ graph.
• The sequences and lengths of those contigs match between the FASTA and the graph.

If the verification fails, the pipeline exits with a descriptive error message indicating which contigs are missing or mismatched, and how to fix the issue. Users can remove the --run-ref-verification flag to bypass this check, but the resulting variants may be incorrect if the reference files are inconsistent.

When --run-ref-verification is not set, the pipeline will still suggest using the flag if an error occurs during alignment or variant calling, as the error may be caused by inconsistent reference files.

Quick Start

Before running the pangenome germline pipeline, ensure you have generated the required files. Refer to the file generation section above for instructions.

1
# This command assumes all the inputs are in the current working directory and all the outputs go to the same place.
2
docker run --rm --gpus all --volume $(pwd):/workdir --volume $(pwd):/outputdir \
3
    --workdir /workdir \
4
    nvcr.io/nvidia/clara/clara-parabricks:4.7.1-1 \
5
    pbrun pangenome_germline \
6
    --ref /workdir/hprc-v1.1-mc-grch38.d9.fa \
7
    --gbz-name /workdir/hprc-v1.1-mc-grch38.d9.gbz \
8
    --dist-name /workdir/hprc-v1.1-mc-grch38.d9.autoindex.1.70.dist \
9
    --minimizer-name /workdir/hprc-v1.1-mc-grch38.d9.autoindex.1.70.shortread.withzip.min \
10
    --zipcodes-name /workdir/hprc-v1.1-mc-grch38.d9.autoindex.1.70.shortread.zipcodes \
11
    --ref-paths /workdir/hprc-v1.1-mc-grch38.d9.paths.sub \
12
    --in-fq /workdir/${INPUT_FASTQ_1} /workdir/${INPUT_FASTQ_2} \
13
    --out-bam /outputdir/${OUTPUT_BAM} \
14
    --out-variants /outputdir/${OUTPUT_VCF} \
15
    --run-ref-verification

System Requirements and Performance

The pangenome germline pipeline inherits performance options from both Giraffe and Pangenome-aware DeepVariant. The key tuning parameters are:

Giraffe (alignment step):

• --nstreams: Number of CUDA streams per GPU (default: auto).
• --num-cpu-threads-per-gpu: Number of CPU worker threads per GPU (default: 16).
• --minimizers-gpu: Enable GPU-accelerated minimizer computation (SE only).
• --low-memory: For GPUs with less than 22 GB of device memory.

Pangenome-aware DeepVariant (variant calling step):

• --num-streams-per-gpu: Number of streams per GPU for variant calling (default: auto).
• --run-partition: More efficiently splits work across multiple GPUs.

For detailed performance guidance on the alignment step, refer to System Requirements and Useful Options for Performance. For the variant calling step, refer to Best Performance for Deepvariant.

Compatible CPU-based Commands

The commands below are the CPU counterpart of the Parabricks pangenome germline pipeline. The output from these commands will be identical to the output from the above command. Refer to the Output Comparison page for comparing the results.

The index files used below are generated in the file generation section.

1
# Stage 1: Run vg giraffe and pipe the output to create a sorted BAM.
2
$ vg giraffe \
3
    -t 16 \
4
    -Z /workdir/hprc-v1.1-mc-grch38.d9.gbz \
5
    -d /workdir/hprc-v1.1-mc-grch38.d9.autoindex.1.70.dist \
6
    -m /workdir/hprc-v1.1-mc-grch38.d9.autoindex.1.70.shortread.withzip.min \
7
    -z /workdir/hprc-v1.1-mc-grch38.d9.autoindex.1.70.shortread.zipcodes \
8
    --ref-paths /workdir/hprc-v1.1-mc-grch38.d9.paths.sub \
9
    -f /workdir/${INPUT_FASTQ_1} \
10
    -f /workdir/${INPUT_FASTQ_2} \
11
    --output-format bam | \
12
  gatk SortSam \
13
    --java-options -Xmx30g \
14
    --MAX_RECORDS_IN_RAM 5000000 \
15
    -I /dev/stdin \
16
    -O cpu.bam \
17
    --SORT_ORDER coordinate
18
19
# Mark duplicates.
20
$ gatk MarkDuplicates \
21
    -I cpu.bam \
22
    -O cpu.markdup.bam \
23
    -M metrics.txt
24
25
# Stage 2: Run Google pangenome-aware DeepVariant.
26
sudo docker run \
27
--volume <INPUT_DIR>:/input \
28
--volume <OUTPUT_DIR>:/output \
29
--shm-size 12gb \
30
google/pangenome_aware_deepvariant-1.9.0 \
31
/opt/deepvariant/bin/run_pangenome_aware_deepvariant \
32
--model_type WGS \
33
--ref /input/hprc-v1.1-mc-grch38.d9.fa \
34
--reads /input/cpu.markdup.bam \
35
--pangenome /input/hprc-v1.1-mc-grch38.d9.gbz \
36
--output_vcf /output/${OUTPUT_VCF} \
37
--num_shards $(nproc) \
38
--make_examples_extra_args "ws_use_window_selector_model=true"

Source of Mismatches

The pangenome germline pipeline combines Giraffe and Pangenome-aware DeepVariant. Sources of mismatches between Parabricks and CPU-based tools can arise from either stage. For detailed information, refer to:

• Giraffe Source of Mismatches — differences in the alignment stage (baseline container requirements, unmapped read sorting).

• Pangenome-aware DeepVariant Source of Mismatches — differences in the variant calling stage (CNN inference, read sorting, GBZ reader caching).

Options