pangenome_germline

Run a GPU-accelerated end-to-end germline pipeline from FASTQ to VCF using pangenome alignment (Giraffe) and Pangenome-aware DeepVariant.

See the pangenome_germline Reference section for a detailed listing of all available options.

The pangenome germline pipeline combines two tools into a single end-to-end workflow:

1. Giraffe (giraffe (vg giraffe + GATK)) -- aligns short reads to a pangenome graph, producing a coordinate-sorted, duplicate-marked BAM file.

2. Pangenome-aware DeepVariant (pangenome_aware_deepvariant) -- calls variants from the aligned BAM using a pangenome-aware model, producing a VCF file.

This pipeline is inspired by the vgteam Giraffe-DeepVariant WDL workflow, but uses Pangenome-aware DeepVariant instead of standard DeepVariant for improved variant calling accuracy in complex and highly variable genomic regions.

The pangenome germline pipeline requires the following input files:

For Giraffe (alignment):

• .gbz -- pangenome graph

• .dist -- distance index

• .min -- minimizer index

• .zipcodes -- zipcodes index

• .paths.sub -- filtered reference paths

For Pangenome-aware DeepVariant (variant calling):

• .fa -- reference FASTA extracted from the graph

• .fa.fai -- FASTA index

The reference FASTA (--ref) must be derived from the same pangenome graph used for alignment (--gbz-name) and must match the contigs listed in the reference paths file (--ref-paths).

The index files can be generated from a GBZ graph using vg autoindex, and the reference FASTA can be extracted from the graph using vg paths. The following example uses the HPRC v1.1 Minigraph-Cactus pangenome graph aligned to GRCh38:

            

            
# Download GBZ
# https://s3-us-west-2.amazonaws.com/human-pangenomics/pangenomes/freeze/freeze1/minigraph-cactus/hprc-v1.1-mc-grch38/hprc-v1.1-mc-grch38.d9.gbz
aws s3 cp \
    s3://human-pangenomics/pangenomes/freeze/freeze1/minigraph-cactus/hprc-v1.1-mc-grch38/hprc-v1.1-mc-grch38.d9.gbz \
    . \
    --no-sign-request

# Extract index files from GBZ
docker run --rm --volume $(pwd):/workdir \
    --workdir /workdir \
    --user $(id -u):$(id -g) \
    quay.io/vgteam/vg:v1.70.0 \
    vg autoindex \
        -p hprc-v1.1-mc-grch38.d9.autoindex.1.70 \
        -G hprc-v1.1-mc-grch38.d9.gbz \
        -w giraffe

# Extract paths from GBZ
docker run --rm \
    --user $(id -u):$(id -g) \
    --volume $(pwd):/workdir \
    --workdir /workdir \
    quay.io/vgteam/vg:v1.70.0 \
    vg paths -x hprc-v1.1-mc-grch38.d9.gbz \
    -L --paths-by GRCh38 > hprc-v1.1-mc-grch38.d9.paths

# Filter paths to keep only chromosomes
grep -v _decoy hprc-v1.1-mc-grch38.d9.paths \
    | grep -v _random \
    | grep -v chrUn_ \
    | grep -v chrEBV \
    | grep -v chrM \
    | grep -v chain_ > hprc-v1.1-mc-grch38.d9.paths.sub

# Extract the sequences corresponding to the list of paths to a FASTA file
docker run --rm \
    --user $(id -u):$(id -g) \
    --volume $(pwd):/workdir \
    --workdir /workdir \
    quay.io/vgteam/vg:v1.70.0 \
    vg paths -x hprc-v1.1-mc-grch38.d9.gbz \
        -p hprc-v1.1-mc-grch38.d9.paths.sub \
        -F > hprc-v1.1-mc-grch38.d9.fa

# Index the FASTA file
docker run --rm \
    --user $(id -u):$(id -g) \
    --volume $(pwd):/workdir \
    --workdir /workdir \
    quay.io/biocontainers/samtools:1.17--hd87286a_2 \
    samtools faidx hprc-v1.1-mc-grch38.d9.fa
        

The pipeline provides an optional pre-flight verification step enabled by the --run-ref-verification flag. When enabled, the pipeline checks the consistency of the reference FASTA against the GBZ graph before alignment begins.

The verification checks that:

• Every contig listed in --ref-paths is present in the reference FASTA.

• Every contig listed in --ref-paths is present in the GBZ graph.

• The sequences and lengths of those contigs match between the FASTA and the graph.

If the verification fails, the pipeline exits with a descriptive error message indicating which contigs are missing or mismatched, and how to fix the issue. Users can remove the --run-ref-verification flag to bypass this check, but the resulting variants may be incorrect if the reference files are inconsistent.

When --run-ref-verification is not set, the pipeline will still suggest using the flag if an error occurs during alignment or variant calling, as the error may be caused by inconsistent reference files.

Before running the pangenome germline pipeline, ensure you have generated the required files. See the file generation section above for instructions.

            

            
# This command assumes all the inputs are in the current working directory and all the outputs go to the same place.
docker run --rm --gpus all --volume $(pwd):/workdir --volume $(pwd):/outputdir \
    --workdir /workdir \
    nvcr.io/nvidia/clara/clara-parabricks:4.7.0-1 \
    pbrun pangenome_germline \
    --ref /workdir/hprc-v1.1-mc-grch38.d9.fa \
    --gbz-name /workdir/hprc-v1.1-mc-grch38.d9.gbz \
    --dist-name /workdir/hprc-v1.1-mc-grch38.d9.autoindex.1.70.dist \
    --minimizer-name /workdir/hprc-v1.1-mc-grch38.d9.autoindex.1.70.shortread.withzip.min \
    --zipcodes-name /workdir/hprc-v1.1-mc-grch38.d9.autoindex.1.70.shortread.zipcodes \
    --ref-paths /workdir/hprc-v1.1-mc-grch38.d9.paths.sub \
    --in-fq /workdir/${INPUT_FASTQ_1} /workdir/${INPUT_FASTQ_2} \
    --out-bam /outputdir/${OUTPUT_BAM} \
    --out-variants /outputdir/${OUTPUT_VCF} \
    --run-ref-verification
        

The pangenome germline pipeline inherits performance options from both Giraffe and Pangenome-aware DeepVariant. The key tuning parameters are:

Giraffe (alignment step):

• --nstreams: Number of CUDA streams per GPU (default: auto).

• --num-cpu-threads-per-gpu: Number of CPU worker threads per GPU (default: 16).

• --minimizers-gpu: Enable GPU-accelerated minimizer computation (SE only).

• --low-memory: For GPUs with less than 22 GB of device memory.

Pangenome-aware DeepVariant (variant calling step):

• --num-streams-per-gpu: Number of streams per GPU for variant calling (default: auto).

• --run-partition: More efficiently splits work across multiple GPUs.

For detailed performance guidance on the alignment step, see System Requirements and Useful Options for Performance. For the variant calling step, see Best Performance for Deepvariant.

The commands below are the CPU counterpart of the Parabricks pangenome germline pipeline. The output from these commands will be identical to the output from the above command. See the Output Comparison page for comparing the results.

The index files used below are generated in the file generation section.

            

            
# Stage 1: Run vg giraffe and pipe the output to create a sorted BAM.
$ vg giraffe \
    -t 16 \
    -Z /workdir/hprc-v1.1-mc-grch38.d9.gbz \
    -d /workdir/hprc-v1.1-mc-grch38.d9.autoindex.1.70.dist \
    -m /workdir/hprc-v1.1-mc-grch38.d9.autoindex.1.70.shortread.withzip.min \
    -z /workdir/hprc-v1.1-mc-grch38.d9.autoindex.1.70.shortread.zipcodes \
    --ref-paths /workdir/hprc-v1.1-mc-grch38.d9.paths.sub \
    -f /workdir/${INPUT_FASTQ_1} \
    -f /workdir/${INPUT_FASTQ_2} \
    --output-format bam | \
  gatk SortSam \
    --java-options -Xmx30g \
    --MAX_RECORDS_IN_RAM 5000000 \
    -I /dev/stdin \
    -O cpu.bam \
    --SORT_ORDER coordinate

# Mark duplicates.
$ gatk MarkDuplicates \
    -I cpu.bam \
    -O cpu.markdup.bam \
    -M metrics.txt

# Stage 2: Run Google pangenome-aware DeepVariant.
sudo docker run \
--volume <INPUT_DIR>:/input \
--volume <OUTPUT_DIR>:/output \
--shm-size 12gb \
google/pangenome_aware_deepvariant-1.9.0 \
/opt/deepvariant/bin/run_pangenome_aware_deepvariant \
--model_type WGS \
--ref /input/hprc-v1.1-mc-grch38.d9.fa \
--reads /input/cpu.markdup.bam \
--pangenome /input/hprc-v1.1-mc-grch38.d9.gbz \
--output_vcf /output/${OUTPUT_VCF} \
--num_shards $(nproc) \
--make_examples_extra_args "ws_use_window_selector_model=true"
        

The pangenome germline pipeline combines Giraffe and Pangenome-aware DeepVariant. Sources of mismatches between Parabricks and CPU-based tools can arise from either stage. For detailed information, refer to:

• Giraffe Source of Mismatches -- differences in the alignment stage (baseline container requirements, unmapped read sorting).

• Pangenome-aware DeepVariant Source of Mismatches -- differences in the variant calling stage (CNN inference, read sorting, GBZ reader caching).

Run the germline pipeline from FASTQ to VCF using pangenome alignment (Giraffe) and pangenome-aware DeepVariant. The reference FASTA (--ref) must be the same reference used for the graph (GBZ) and for the ref-paths file.