Benchmarks

BioNeMo Inference Runtime (BioIR) measured against two OSS PyTorch baselines on every GPU it is tested on. These are the numbers a run of the published procedure should reproduce; they are not peak performance, and no attempt was made to tune any configuration per GPU.

Both baselines are reported for every measurement, and the pair is the point. Against OSS PyTorch eager, BioIR is faster on every GPU and model here, by 1.19x at worst. Against OSS torch.compile the answer depends on the architecture: 1.5-3.0x on Ampere, Hopper and Ada, but 1.0-2.1x on Blackwell, where torch.compile closes most of the gap. Either baseline quoted alone overstates the result in one direction or understates it in the other, so figures below 1.00x are printed rather than dropped.

torch.compile is ahead on 19% of individual measurements, and where it is ahead is not random: never below 256 residues, 16% of inputs between 257 and 768, and around 29% above 769 — and only ever on Boltz-2 and OpenFold3, never on OpenFold2 or Protenix. Long inputs on Blackwell are the case to check before quoting a single figure.

Per-sample speedup against input size, all four models on H100. The dashed line is parity; crosses mark the largest input measured. The curves are far from flat — high on the smallest inputs, where launch and dispatch overhead dominates and BioIR has less of it, lowest in the few-hundred-residue range, then climbing with length as the fused kernels start to matter. That shape is why every table below is broken out by residue count, and why an overall geomean alone is not enough to quote.

Contents

• Measurements
• Boltz-2
• OpenFold3
• OpenFold2 / AlphaFold2
• Protenix
• Reproducing

Measurements

Hardware

One GPU per measurement, one process, no replicas. Absolute latencies move with clocks, power limit and thermal headroom; the ratios are what should be comparable.

What Is Compared

Three backends fold the same inputs on the same GPU in the same container:

The OSS side runs its own inference script: eager always, and torch.compile additionally when it passes a dynamic-shape probe. Protenix has no compile path, which is why its compile column is empty.

Units

Speedup is OSS forward / BioIR forward per sample; above 1 favours BioIR. Every figure is a geometric mean of per-sample speedups, not a ratio of totals, so each sample counts equally regardless of how long it runs. n is the sample count and differs by model.

Speedups are broken out by input size in residues because the spread is wide — Boltz-2 on H200 is 1.80x below 512 residues and 3.61x above 1024 against OSS torch.compile. The overall geomean alone hides a factor of two.

Peak memory is torch.cuda.max_memory_allocated over the run. Accuracy is OpenStructure lDDT over every sample and DockQ over the subset with a supported protein interface.

Boltz-2

Bucket cells are OSS torch.compile / OSS PyTorch eager. Below 1.00x means the baseline was faster.

Forward Latency

Geomean of the per-sample forward time, over the samples the ratios above are taken over. Quote these when comparing two releases: a ratio that moved does not say which side did.

Peak Allocated Memory

Below 1.00x means BioIR allocates less than the baseline; above 1.00x means it allocates more.

Accuracy

Each cell is BioIR / OSS torch.compile / OSS PyTorch eager. lDDT is scored over every sample; DockQ covers the subset with a supported protein interface, which is why its count is lower. The three should agree — a speedup that moved them would not be the same answer.

OpenFold3

Bucket cells are OSS torch.compile / OSS PyTorch eager. Below 1.00x means the baseline was faster.

Forward Latency

Geomean of the per-sample forward time, over the samples the ratios above are taken over. Quote these when comparing two releases: a ratio that moved does not say which side did.

Peak Allocated Memory

Below 1.00x means BioIR allocates less than the baseline; above 1.00x means it allocates more.

Accuracy

Each cell is BioIR / OSS torch.compile / OSS PyTorch eager. lDDT is scored over every sample; DockQ covers the subset with a supported protein interface, which is why its count is lower. The three should agree — a speedup that moved them would not be the same answer.

OpenFold2 / AlphaFold2

Monomer

Bucket cells are OSS torch.compile / OSS PyTorch eager. Below 1.00x means the baseline was faster.

Forward Latency

Geomean of the per-sample forward time, over the samples the ratios above are taken over. Quote these when comparing two releases: a ratio that moved does not say which side did.

Peak Allocated Memory

Below 1.00x means BioIR allocates less than the baseline; above 1.00x means it allocates more.

Accuracy

Each cell is BioIR / OSS torch.compile / OSS PyTorch eager. lDDT is scored over every sample; DockQ covers the subset with a supported protein interface, which is why its count is lower. The three should agree — a speedup that moved them would not be the same answer.

Multimer

Bucket cells are OSS torch.compile / OSS PyTorch eager. Below 1.00x means the baseline was faster.

Forward Latency

Geomean of the per-sample forward time, over the samples the ratios above are taken over. Quote these when comparing two releases: a ratio that moved does not say which side did.

Peak Allocated Memory

Below 1.00x means BioIR allocates less than the baseline; above 1.00x means it allocates more.

Accuracy

Each cell is BioIR / OSS torch.compile / OSS PyTorch eager. lDDT is scored over every sample; DockQ covers the subset with a supported protein interface, which is why its count is lower. The three should agree — a speedup that moved them would not be the same answer.

Protenix

Bucket cells are OSS torch.compile / OSS PyTorch eager. Below 1.00x means the baseline was faster.

Forward Latency

Geomean of the per-sample forward time, over the samples the ratios above are taken over. Quote these when comparing two releases: a ratio that moved does not say which side did.

Peak Allocated Memory

Below 1.00x means BioIR allocates less than the baseline; above 1.00x means it allocates more.

Accuracy

Each cell is BioIR / OSS torch.compile / OSS PyTorch eager. lDDT is scored over every sample; DockQ covers the subset with a supported protein interface, which is why its count is lower. The three should agree — a speedup that moved them would not be the same answer.

Reproducing

These numbers are not tied to a build artifact: they come from main, on one GPU, with public inputs and upstream baselines at pinned tags. The procedure is the bench-perf-oss agent skill — point an agent at it rather than following steps by hand. It installs BioIR and each pinned baseline in isolation, runs one warmup and one timed forward per sample, scores the output, and aggregates the geomeans above.

Have these ready first; the skill cannot supply them:

• One NVIDIA GPU on driver 580 or newer. The container ships a CUDA 13.2 build of PyTorch. On a 535 driver it runs only through the forward-compatibility shim, and measuring on that path produced silent hangs and SIGFPE crashes part-way through a sweep — wrong numbers, not merely slow ones. Check the ECC counters are zero while you are there.
• NGC_API_KEY exported, for the MSA Search NIM. rebuild_dataset.py builds the dataset-0.3.0 inputs from RCSB and that endpoint rather than downloading a release; the structures need no credential at all (--structures-only). Free key at build.nvidia.com.
• HF_TOKEN exported. OpenFold3’s checkpoint is gated and an anonymous fetch returns 401. Registering is free. Checkpoint staging generally is Model Weights.
• A built extension, per Development Workflow.

When a ratio does not match, suspect the baseline environment before the code. The most common cause by far is an upstream package resolved against a different torch than BioIR’s, at which point the ratio compares two PyTorch builds rather than two implementations.

---

Container nvcr.io/nvidia/pytorch:26.05-py3 · driver 580.105.08–595.58.03 · measured 2026-09-01