> This page is for RFD4-Proteina, version 1.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 1.0.0 (Latest) (default): https://docs.nvidia.com/nim/bionemo/rfd4-proteina/1.0.0/llms.txt
> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/bionemo/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/bionemo/_mcp/server.
# Performance in RFD4-Proteina NIM
## Execution Backends
The RFD4-Proteina NIM can run three execution backends, selected per
deployment (refer to [Configuration Alternatives](/nim/bionemo/rfd4-proteina/configuration-alternatives)):
- `oss_eager`: the upstream reference network, uncompiled.
- `oss_torch_compile`: the same network under `torch.compile`.
- `bioir`: an NVIDIA-accelerated trunk built on the BioNeMo Inference
Runtime, with its own fused kernels and CUDA-graph capture.
Every response reports which backend actually ran it in
`model.execution_backend`, alongside `runtime.cuda_graphs` and
`runtime.cuda_graph_cache`, which describe what the loaded runtime currently
holds. Results are comparable in kind across backends but not bit-identical,
since compilation and BioIR's own kernels both change floating-point
numerics.
## Benchmark Configuration
The table in this section comes from a single release-candidate sweep of the
synthetic latency suite, `1.0.0-rc.20261012093607`, measured 2026-10-12.
| Parameter | Setting |
| --- | --- |
| Task | `generation` |
| Input | synthetic poly-glycine backbone, `annotated_cif` |
| `sampling.recycle_strategy` | `thinking_medium` — the checkpoint default, 60 steps over 185 trunk passes |
| `sampling.num_replicates`, `sampling.batch_size` | 1, 2, and 4, swept at every token bucket |
| `output.structure_format` | `none` |
| Token buckets | 128, 160, 192, 224, 256, 288, 320, 352, 384, 448, 512, 640, 768, 896, 1024 |
| Runs per shape | 3, of which the cold first run is discarded and the warm median reported |
| Backends | `oss_eager`, `oss_torch_compile`, `bioir` |
Design time is the client's end-to-end wall clock around one HTTP request. It
therefore covers input parsing, sampling, metrics, and response encoding, not
only the trunk forward passes. Because the cold run is excluded, these are
steady-state figures; refer to
[Cold-Start Behavior](#cold-start-behavior) for what the first request into a
bucket costs instead.
Replicates are batched into one forward pass, so a batch-`n` request is a
single request returning `n` designs. Divide by the batch size for a
per-design cost. Refer to [Replicates and
Batching](/nim/bionemo/rfd4-proteina/prerequisites#replicates-and-batching).
Five GPUs sweep batch 1 only — GB10, both 40 GB A100s, L40S, and RTX 6000 Ada
Generation — so their tabs carry 15 rows rather than 45, and each one says so.
## Performance Across Execution Backends
All three execution backends over the full sweep, at every token bucket and
every batch size that GPU was swept at. Select a GPU to switch the table; the
two rightmost columns are `bioir`'s speedup over each OSS baseline, where
higher means `bioir` is faster. A dash marks a shape that did not complete.
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 6.82 | 3.12 | 1.17 | 5.83x | 2.67x |
| 128 | 2 | 7.04 | 3.35 | 1.44 | 4.88x | 2.33x |
| 128 | 4 | 8.25 | 3.93 | 2.44 | 3.39x | 1.61x |
| 160 | 1 | 6.85 | 3.26 | 1.23 | 5.56x | 2.65x |
| 160 | 2 | 7.09 | 3.51 | 2.00 | 3.54x | 1.75x |
| 160 | 4 | 8.87 | 4.74 | 3.34 | 2.66x | 1.42x |
| 192 | 1 | 6.87 | 3.35 | 1.49 | 4.60x | 2.24x |
| 192 | 2 | 8.71 | 3.75 | 2.39 | 3.64x | 1.57x |
| 192 | 4 | 9.14 | 5.62 | 4.12 | 2.22x | 1.36x |
| 224 | 1 | 6.93 | 3.40 | 1.65 | 4.19x | 2.06x |
| 224 | 2 | 11.10 | 4.51 | 2.76 | 4.02x | 1.63x |
| 224 | 4 | 10.69 | 6.86 | 5.02 | 2.13x | 1.37x |
| 256 | 1 | 8.05 | 3.88 | 1.91 | 4.22x | 2.03x |
| 256 | 2 | 8.67 | 4.79 | 3.28 | 2.64x | 1.46x |
| 256 | 4 | 13.18 | 8.36 | 6.09 | 2.16x | 1.37x |
| 288 | 1 | 8.20 | 3.96 | 2.28 | 3.60x | 1.74x |
| 288 | 2 | 8.90 | 5.70 | 3.99 | 2.23x | 1.43x |
| 288 | 4 | 16.00 | 10.17 | 7.55 | 2.12x | 1.35x |
| 320 | 1 | 8.24 | 4.02 | 2.63 | 3.14x | 1.53x |
| 320 | 2 | 10.32 | 6.50 | 4.73 | 2.18x | 1.37x |
| 320 | 4 | 19.13 | 11.87 | 8.90 | 2.15x | 1.33x |
| 352 | 1 | 8.30 | 4.49 | 2.94 | 2.82x | 1.53x |
| 352 | 2 | 12.00 | 7.57 | 5.36 | 2.24x | 1.41x |
| 352 | 4 | 22.57 | 13.88 | 10.40 | 2.17x | 1.34x |
| 384 | 1 | 8.53 | 4.74 | 3.30 | 2.59x | 1.44x |
| 384 | 2 | 13.81 | 8.47 | 6.18 | 2.24x | 1.37x |
| 384 | 4 | 26.29 | 16.15 | 11.85 | 2.22x | 1.36x |
| 448 | 1 | 9.75 | 5.93 | 4.21 | 2.32x | 1.41x |
| 448 | 2 | 18.03 | 10.97 | 7.95 | 2.27x | 1.38x |
| 448 | 4 | 34.64 | 21.48 | 15.61 | 2.22x | 1.38x |
| 512 | 1 | 12.02 | 7.29 | 5.17 | 2.32x | 1.41x |
| 512 | 2 | 22.79 | 13.62 | 9.98 | 2.28x | 1.36x |
| 512 | 4 | 44.20 | 26.46 | 19.32 | 2.29x | 1.37x |
| 640 | 1 | 17.69 | 10.56 | 7.79 | 2.27x | 1.36x |
| 640 | 2 | 34.38 | 20.32 | 15.00 | 2.29x | 1.35x |
| 640 | 4 | 67.22 | 39.86 | 29.35 | 2.29x | 1.36x |
| 768 | 1 | 24.65 | 14.58 | 10.68 | 2.31x | 1.37x |
| 768 | 2 | 48.16 | 28.44 | 20.57 | 2.34x | 1.38x |
| 768 | 4 | 95.19 | 55.93 | 40.69 | 2.34x | 1.37x |
| 896 | 1 | 32.82 | 19.45 | 14.36 | 2.29x | 1.35x |
| 896 | 2 | 65.12 | 38.30 | 28.10 | 2.32x | 1.36x |
| 896 | 4 | 128.90 | 75.59 | 56.00 | 2.30x | 1.35x |
| 1024 | 1 | 42.05 | 24.74 | 17.89 | 2.35x | 1.38x |
| 1024 | 2 | 83.45 | 49.12 | 35.31 | 2.36x | 1.39x |
| 1024 | 4 | 166.68 | 97.17 | 69.98 | 2.38x | 1.39x |
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 6.15 | 2.66 | 0.91 | 6.79x | 2.93x |
| 128 | 2 | 6.12 | 2.84 | 1.30 | 4.70x | 2.18x |
| 128 | 4 | 7.42 | 3.35 | 2.16 | 3.43x | 1.55x |
| 160 | 1 | 6.01 | 2.81 | 1.10 | 5.47x | 2.55x |
| 160 | 2 | 6.25 | 3.05 | 1.75 | 3.57x | 1.74x |
| 160 | 4 | 7.73 | 3.95 | 2.88 | 2.69x | 1.37x |
| 192 | 1 | 6.03 | 2.86 | 1.32 | 4.57x | 2.16x |
| 192 | 2 | 7.82 | 3.14 | 2.08 | 3.76x | 1.51x |
| 192 | 4 | 7.99 | 4.62 | 3.62 | 2.21x | 1.28x |
| 224 | 1 | 6.01 | 2.94 | 1.44 | 4.17x | 2.04x |
| 224 | 2 | 9.99 | 3.64 | 2.44 | 4.10x | 1.50x |
| 224 | 4 | 9.55 | 5.82 | 4.45 | 2.15x | 1.31x |
| 256 | 1 | 7.11 | 3.37 | 1.69 | 4.20x | 1.99x |
| 256 | 2 | 7.65 | 3.97 | 2.90 | 2.64x | 1.37x |
| 256 | 4 | 11.73 | 6.97 | 5.43 | 2.16x | 1.28x |
| 288 | 1 | 7.23 | 3.48 | 2.00 | 3.62x | 1.74x |
| 288 | 2 | 7.92 | 4.72 | 3.56 | 2.23x | 1.33x |
| 288 | 4 | 14.33 | 8.52 | 6.79 | 2.11x | 1.25x |
| 320 | 1 | 7.24 | 3.48 | 2.33 | 3.11x | 1.49x |
| 320 | 2 | 9.18 | 5.38 | 4.21 | 2.18x | 1.28x |
| 320 | 4 | 17.05 | 10.11 | 7.98 | 2.14x | 1.27x |
| 352 | 1 | 6.99 | 3.67 | 2.60 | 2.69x | 1.41x |
| 352 | 2 | 10.72 | 6.33 | 4.81 | 2.23x | 1.32x |
| 352 | 4 | 20.16 | 11.91 | 9.29 | 2.17x | 1.28x |
| 384 | 1 | 7.50 | 4.09 | 2.94 | 2.55x | 1.39x |
| 384 | 2 | 12.40 | 7.32 | 5.53 | 2.24x | 1.32x |
| 384 | 4 | 23.42 | 13.67 | 10.71 | 2.19x | 1.28x |
| 448 | 1 | 8.76 | 5.04 | 3.77 | 2.32x | 1.34x |
| 448 | 2 | 16.10 | 9.29 | 7.22 | 2.23x | 1.29x |
| 448 | 4 | 30.90 | 18.13 | 14.18 | 2.18x | 1.28x |
| 512 | 1 | 10.85 | 6.15 | 4.67 | 2.32x | 1.32x |
| 512 | 2 | 20.41 | 11.71 | 9.01 | 2.26x | 1.30x |
| 512 | 4 | 39.49 | 22.50 | 17.74 | 2.23x | 1.27x |
| 640 | 1 | 15.86 | 9.11 | 7.16 | 2.22x | 1.27x |
| 640 | 2 | 30.93 | 17.50 | 13.88 | 2.23x | 1.26x |
| 640 | 4 | 60.43 | 34.44 | 27.37 | 2.21x | 1.26x |
| 768 | 1 | 22.10 | 12.57 | 9.72 | 2.27x | 1.29x |
| 768 | 2 | 43.42 | 24.79 | 19.34 | 2.24x | 1.28x |
| 768 | 4 | 85.86 | 48.55 | 38.26 | 2.24x | 1.27x |
| 896 | 1 | 29.54 | 16.98 | 13.35 | 2.21x | 1.27x |
| 896 | 2 | 59.09 | 33.23 | 26.32 | 2.25x | 1.26x |
| 896 | 4 | 116.96 | 66.69 | 52.88 | 2.21x | 1.26x |
| 1024 | 1 | 38.02 | 21.71 | 16.95 | 2.24x | 1.28x |
| 1024 | 2 | 75.89 | 42.80 | 33.74 | 2.25x | 1.27x |
| 1024 | 4 | 151.45 | 84.61 | 67.08 | 2.26x | 1.26x |
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 9.63 | 4.17 | 1.75 | 5.51x | 2.39x |
| 128 | 2 | 9.95 | 4.14 | 2.82 | 3.53x | 1.47x |
| 128 | 4 | 13.31 | 6.16 | 4.84 | 2.75x | 1.27x |
| 160 | 1 | 9.67 | 4.00 | 2.25 | 4.29x | 1.78x |
| 160 | 2 | 10.81 | 4.99 | 3.82 | 2.83x | 1.31x |
| 160 | 4 | 19.42 | 8.57 | 6.98 | 2.78x | 1.23x |
| 192 | 1 | 9.68 | 4.52 | 2.81 | 3.44x | 1.61x |
| 192 | 2 | 14.77 | 6.25 | 4.97 | 2.97x | 1.26x |
| 192 | 4 | 27.22 | 11.41 | 9.35 | 2.91x | 1.22x |
| 224 | 1 | 11.28 | 5.97 | 3.50 | 3.23x | 1.71x |
| 224 | 2 | 20.34 | 7.95 | 6.25 | 3.25x | 1.27x |
| 224 | 4 | 38.45 | 14.64 | 11.81 | 3.26x | 1.24x |
| 256 | 1 | 14.87 | 7.53 | 4.17 | 3.57x | 1.81x |
| 256 | 2 | 27.42 | 9.69 | 7.61 | 3.60x | 1.27x |
| 256 | 4 | 52.48 | 18.04 | 14.71 | 3.57x | 1.23x |
| 288 | 1 | 17.21 | 8.69 | 5.12 | 3.36x | 1.70x |
| 288 | 2 | 32.31 | 12.12 | 9.76 | 3.31x | 1.24x |
| 288 | 4 | 62.39 | 22.79 | 18.63 | 3.35x | 1.22x |
| 320 | 1 | 20.40 | 10.36 | 6.04 | 3.38x | 1.72x |
| 320 | 2 | 38.61 | 14.11 | 11.47 | 3.37x | 1.23x |
| 320 | 4 | 75.90 | 27.19 | 22.31 | 3.40x | 1.22x |
| 352 | 1 | 25.21 | 12.49 | 6.91 | 3.65x | 1.81x |
| 352 | 2 | 48.20 | 16.61 | 13.24 | 3.64x | 1.25x |
| 352 | 4 | 94.67 | 31.67 | 26.34 | 3.59x | 1.20x |
| 384 | 1 | 29.69 | 14.44 | 8.00 | 3.71x | 1.81x |
| 384 | 2 | 56.99 | 19.20 | 15.32 | 3.72x | 1.25x |
| 384 | 4 | 112.61 | 36.72 | 30.72 | 3.67x | 1.20x |
| 448 | 1 | 41.48 | 20.53 | 10.64 | 3.90x | 1.93x |
| 448 | 2 | 81.03 | 25.39 | 20.65 | 3.92x | 1.23x |
| 448 | 4 | 160.89 | 49.59 | 40.79 | 3.94x | 1.22x |
| 512 | 1 | 58.76 | 31.01 | 13.34 | 4.41x | 2.33x |
| 512 | 2 | 116.25 | 32.01 | 26.01 | 4.47x | 1.23x |
| 512 | 4 | 231.44 | 63.17 | 52.27 | 4.43x | 1.21x |
| 640 | 1 | 99.39 | 56.56 | 20.50 | 4.85x | 2.76x |
| 640 | 2 | 198.37 | 49.26 | 40.74 | 4.87x | 1.21x |
| 640 | 4 | 396.50 | 97.48 | 80.69 | 4.91x | 1.21x |
| 768 | 1 | 147.68 | 84.16 | 29.08 | 5.08x | 2.89x |
| 768 | 2 | 294.72 | 70.07 | 57.69 | 5.11x | 1.21x |
| 768 | 4 | 589.56 | 138.93 | 115.12 | 5.12x | 1.21x |
| 896 | 1 | 200.99 | 114.39 | 39.60 | 5.08x | 2.89x |
| 896 | 2 | 402.19 | 95.22 | 78.92 | 5.10x | 1.21x |
| 896 | 4 | 805.30 | 189.86 | 157.50 | 5.11x | 1.21x |
| 1024 | 1 | 262.70 | 149.65 | 51.28 | 5.12x | 2.92x |
| 1024 | 2 | 525.71 | 124.05 | 102.48 | 5.13x | 1.21x |
| 1024 | 4 | 1055.63 | 247.65 | 205.40 | 5.14x | 1.21x |
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 5.10 | 2.33 | 0.93 | 5.49x | 2.51x |
| 128 | 2 | 5.42 | 2.56 | 1.18 | 4.59x | 2.17x |
| 128 | 4 | 5.80 | 2.95 | 1.93 | 3.00x | 1.53x |
| 160 | 1 | 5.22 | 2.44 | 0.99 | 5.27x | 2.47x |
| 160 | 2 | 5.41 | 2.65 | 1.48 | 3.65x | 1.79x |
| 160 | 4 | 8.11 | 3.11 | 2.51 | 3.23x | 1.24x |
| 192 | 1 | 5.26 | 2.47 | 1.17 | 4.50x | 2.12x |
| 192 | 2 | 6.03 | 2.74 | 1.85 | 3.26x | 1.48x |
| 192 | 4 | 7.31 | 3.85 | 3.14 | 2.33x | 1.23x |
| 224 | 1 | 5.32 | 2.52 | 1.35 | 3.94x | 1.87x |
| 224 | 2 | 7.71 | 2.81 | 2.18 | 3.53x | 1.29x |
| 224 | 4 | 7.47 | 4.10 | 3.92 | 1.91x | 1.05x |
| 256 | 1 | 5.36 | 2.57 | 1.55 | 3.45x | 1.66x |
| 256 | 2 | 7.01 | 3.56 | 2.57 | 2.72x | 1.38x |
| 256 | 4 | 8.94 | 4.92 | 4.77 | 1.87x | 1.03x |
| 288 | 1 | 6.46 | 2.93 | 1.80 | 3.59x | 1.63x |
| 288 | 2 | 7.08 | 3.69 | 3.07 | 2.30x | 1.20x |
| 288 | 4 | 10.88 | 5.91 | 5.80 | 1.88x | 1.02x |
| 320 | 1 | 7.63 | 2.96 | 2.01 | 3.79x | 1.47x |
| 320 | 2 | 7.13 | 3.79 | 3.55 | 2.01x | 1.07x |
| 320 | 4 | 12.96 | 6.84 | 6.81 | 1.90x | 1.00x |
| 352 | 1 | 8.91 | 3.38 | 2.29 | 3.88x | 1.47x |
| 352 | 2 | 8.12 | 4.42 | 4.12 | 1.97x | 1.07x |
| 352 | 4 | 15.13 | 7.93 | 7.99 | 1.89x | 0.99x |
| 384 | 1 | 6.77 | 3.27 | 2.58 | 2.63x | 1.27x |
| 384 | 2 | 9.33 | 4.85 | 4.79 | 1.95x | 1.01x |
| 384 | 4 | 17.63 | 9.11 | 9.16 | 1.92x | 0.99x |
| 448 | 1 | 6.84 | 3.54 | 3.22 | 2.12x | 1.10x |
| 448 | 2 | 12.15 | 6.19 | 6.36 | 1.91x | 0.97x |
| 448 | 4 | 23.40 | 11.88 | 11.94 | 1.96x | 1.00x |
| 512 | 1 | 8.10 | 4.16 | 4.03 | 2.01x | 1.03x |
| 512 | 2 | 15.26 | 7.72 | 7.67 | 1.99x | 1.01x |
| 512 | 4 | 29.54 | 14.92 | 15.04 | 1.96x | 0.99x |
| 640 | 1 | 11.91 | 6.04 | 5.97 | 1.99x | 1.01x |
| 640 | 2 | 22.98 | 11.50 | 11.54 | 1.99x | 1.00x |
| 640 | 4 | 45.33 | 22.50 | 22.75 | 1.99x | 0.99x |
| 768 | 1 | 16.41 | 8.15 | 8.31 | 1.97x | 0.98x |
| 768 | 2 | 32.05 | 15.74 | 15.89 | 2.02x | 0.99x |
| 768 | 4 | 63.90 | 31.16 | 31.77 | 2.01x | 0.98x |
| 896 | 1 | 21.91 | 11.07 | 11.02 | 1.99x | 1.00x |
| 896 | 2 | 43.37 | 21.43 | 21.65 | 2.00x | 0.99x |
| 896 | 4 | 86.71 | 42.62 | 43.32 | 2.00x | 0.98x |
| 1024 | 1 | 27.89 | 13.66 | 13.85 | 2.01x | 0.99x |
| 1024 | 2 | 56.12 | 27.10 | 27.12 | 2.07x | 1.00x |
| 1024 | 4 | 112.14 | 54.03 | 54.62 | 2.05x | 0.99x |
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 9.08 | 3.90 | 1.10 | 8.23x | 3.53x |
| 128 | 2 | 5.38 | 4.33 | 1.11 | 4.85x | 3.91x |
| 128 | 4 | 5.76 | 4.96 | 1.70 | 3.40x | 2.93x |
| 160 | 1 | 5.31 | 4.13 | 0.89 | 5.99x | 4.66x |
| 160 | 2 | 5.47 | 4.48 | 1.31 | 4.18x | 3.42x |
| 160 | 4 | 7.76 | 5.33 | 2.26 | 3.43x | 2.36x |
| 192 | 1 | 5.31 | 4.32 | 1.16 | 4.59x | 3.73x |
| 192 | 2 | 7.65 | 4.78 | 2.20 | 3.47x | 2.17x |
| 192 | 4 | 12.52 | 6.55 | 3.74 | 3.34x | 1.75x |
| 224 | 1 | 9.14 | 4.36 | 1.61 | 5.68x | 2.71x |
| 224 | 2 | 9.76 | 4.89 | 2.60 | 3.76x | 1.88x |
| 224 | 4 | 12.79 | 6.99 | 4.60 | 2.78x | 1.52x |
| 256 | 1 | 9.19 | 4.38 | 1.81 | 5.07x | 2.41x |
| 256 | 2 | 11.92 | 5.96 | 3.04 | 3.92x | 1.96x |
| 256 | 4 | 13.02 | 7.28 | 4.23 | 3.08x | 1.72x |
| 288 | 1 | 9.32 | 4.45 | 1.67 | 5.59x | 2.67x |
| 288 | 2 | 12.14 | 6.19 | 2.76 | 4.39x | 2.24x |
| 288 | 4 | 13.34 | 7.59 | 5.15 | 2.59x | 1.47x |
| 320 | 1 | 9.35 | 4.64 | 1.85 | 5.06x | 2.51x |
| 320 | 2 | 12.26 | 6.34 | 4.00 | 3.07x | 1.59x |
| 320 | 4 | 13.84 | 8.07 | 7.61 | 1.82x | 1.06x |
| 352 | 1 | 9.49 | 4.57 | 2.66 | 3.57x | 1.72x |
| 352 | 2 | 12.31 | 6.51 | 4.61 | 2.67x | 1.41x |
| 352 | 4 | 16.20 | 9.26 | 8.71 | 1.86x | 1.06x |
| 384 | 1 | 11.67 | 6.19 | 2.87 | 4.07x | 2.16x |
| 384 | 2 | 12.35 | 6.68 | 5.33 | 2.32x | 1.25x |
| 384 | 4 | 18.65 | 8.75 | 10.03 | 1.86x | 0.87x |
| 448 | 1 | 11.76 | 3.51 | 3.57 | 3.30x | 0.99x |
| 448 | 2 | 12.83 | 6.42 | 6.59 | 1.95x | 0.97x |
| 448 | 4 | 24.37 | 11.13 | 12.70 | 1.92x | 0.88x |
| 512 | 1 | 11.91 | 5.83 | 4.29 | 2.77x | 1.36x |
| 512 | 2 | 15.82 | 8.56 | 8.15 | 1.94x | 1.05x |
| 512 | 4 | 30.52 | 14.99 | 15.98 | 1.91x | 0.94x |
| 640 | 1 | 12.30 | 6.62 | 8.13 | 1.51x | 0.81x |
| 640 | 2 | 23.45 | 10.72 | 11.98 | 1.96x | 0.89x |
| 640 | 4 | 46.00 | 21.02 | 20.68 | 2.22x | 1.02x |
| 768 | 1 | 16.69 | 7.60 | 7.33 | 2.28x | 1.04x |
| 768 | 2 | 32.49 | 16.84 | 14.40 | 2.26x | 1.17x |
| 768 | 4 | 62.31 | 33.02 | 30.41 | 2.05x | 1.09x |
| 896 | 1 | 22.11 | 11.41 | 9.93 | 2.23x | 1.15x |
| 896 | 2 | 43.57 | 22.38 | 19.61 | 2.22x | 1.14x |
| 896 | 4 | 86.16 | 45.55 | 39.16 | 2.20x | 1.16x |
| 1024 | 1 | 28.02 | 14.18 | 12.54 | 2.24x | 1.13x |
| 1024 | 2 | 55.42 | 27.61 | 26.00 | 2.13x | 1.06x |
| 1024 | 4 | 111.25 | 56.06 | 50.61 | 2.20x | 1.11x |
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 9.84 | 5.44 | 1.16 | 8.50x | 4.70x |
| 128 | 2 | 10.14 | 5.56 | 1.35 | 7.52x | 4.12x |
| 128 | 4 | 10.35 | 6.97 | 2.10 | 4.94x | 3.32x |
| 160 | 1 | 10.14 | 6.13 | 1.08 | 9.35x | 5.65x |
| 160 | 2 | 10.31 | 6.61 | 1.59 | 6.46x | 4.14x |
| 160 | 4 | 10.82 | 7.68 | 2.59 | 4.18x | 2.97x |
| 192 | 1 | 10.27 | 6.26 | 1.30 | 7.92x | 4.83x |
| 192 | 2 | 10.21 | 5.79 | 1.95 | 5.25x | 2.98x |
| 192 | 4 | 12.98 | 8.09 | 3.21 | 4.05x | 2.52x |
| 224 | 1 | 10.20 | 6.56 | 1.51 | 6.77x | 4.36x |
| 224 | 2 | 10.65 | 6.85 | 2.33 | 4.58x | 2.94x |
| 224 | 4 | 12.69 | 8.43 | 3.81 | 3.33x | 2.21x |
| 256 | 1 | 10.09 | 5.96 | 1.69 | 5.95x | 3.52x |
| 256 | 2 | 11.97 | 7.99 | 2.68 | 4.47x | 2.98x |
| 256 | 4 | 12.79 | 8.75 | 4.56 | 2.80x | 1.92x |
| 288 | 1 | 10.06 | 6.24 | 1.89 | 5.33x | 3.30x |
| 288 | 2 | 12.29 | 7.21 | 3.03 | 4.05x | 2.38x |
| 288 | 4 | 12.85 | 9.02 | 5.49 | 2.34x | 1.64x |
| 320 | 1 | 10.26 | 6.56 | 2.10 | 4.90x | 3.13x |
| 320 | 2 | 12.58 | 7.69 | 3.48 | 3.61x | 2.21x |
| 320 | 4 | 13.86 | 10.08 | 6.47 | 2.14x | 1.56x |
| 352 | 1 | 10.36 | 6.27 | 2.44 | 4.25x | 2.57x |
| 352 | 2 | 12.39 | 8.27 | 3.99 | 3.11x | 2.07x |
| 352 | 4 | 14.67 | 9.21 | 7.60 | 1.93x | 1.21x |
| 384 | 1 | 12.51 | 7.90 | 2.73 | 4.59x | 2.90x |
| 384 | 2 | 12.99 | 8.59 | 4.56 | 2.85x | 1.89x |
| 384 | 4 | 16.80 | 10.38 | 8.60 | 1.95x | 1.21x |
| 448 | 1 | 11.93 | 7.18 | 3.12 | 3.82x | 2.30x |
| 448 | 2 | 12.78 | 7.92 | 5.69 | 2.24x | 1.39x |
| 448 | 4 | 21.97 | 12.62 | 11.01 | 2.00x | 1.15x |
| 512 | 1 | 12.04 | 8.04 | 3.79 | 3.18x | 2.12x |
| 512 | 2 | 14.53 | 9.18 | 7.13 | 2.04x | 1.29x |
| 512 | 4 | 28.12 | 15.09 | 13.73 | 2.05x | 1.10x |
| 640 | 1 | 12.62 | 7.53 | 9.30 | 1.36x | 0.81x |
| 640 | 2 | 21.73 | 11.43 | 10.64 | 2.04x | 1.07x |
| 640 | 4 | 42.19 | 21.79 | 20.70 | 2.04x | 1.05x |
| 768 | 1 | 15.51 | 8.64 | 9.05 | 1.71x | 0.95x |
| 768 | 2 | 30.07 | 15.66 | 14.78 | 2.03x | 1.06x |
| 768 | 4 | 59.00 | 29.94 | 28.91 | 2.04x | 1.04x |
| 896 | 1 | 20.65 | 10.99 | 10.15 | 2.04x | 1.08x |
| 896 | 2 | 40.62 | 20.28 | 19.82 | 2.05x | 1.02x |
| 896 | 4 | 79.96 | 39.62 | 39.03 | 2.05x | 1.02x |
| 1024 | 1 | 26.13 | 13.00 | 12.61 | 2.07x | 1.03x |
| 1024 | 2 | 51.82 | 25.27 | 25.09 | 2.07x | 1.01x |
| 1024 | 4 | 102.82 | 51.02 | 49.89 | 2.06x | 1.02x |
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 11.04 | 4.25 | 1.12 | 9.85x | 3.79x |
| 128 | 2 | 11.79 | 4.67 | 1.31 | 8.99x | 3.56x |
| 128 | 4 | 12.35 | 5.20 | 1.96 | 6.29x | 2.65x |
| 160 | 1 | 11.72 | 4.70 | 1.06 | 11.04x | 4.43x |
| 160 | 2 | 12.05 | 5.58 | 1.56 | 7.73x | 3.58x |
| 160 | 4 | 12.14 | 5.79 | 2.65 | 4.58x | 2.19x |
| 192 | 1 | 11.50 | 5.41 | 1.33 | 8.67x | 4.07x |
| 192 | 2 | 12.33 | 5.01 | 1.94 | 6.36x | 2.59x |
| 192 | 4 | 15.55 | 6.99 | 3.15 | 4.93x | 2.21x |
| 224 | 1 | 11.22 | 4.91 | 1.46 | 7.66x | 3.35x |
| 224 | 2 | 12.62 | 5.45 | 2.28 | 5.54x | 2.39x |
| 224 | 4 | 14.87 | 6.64 | 3.86 | 3.85x | 1.72x |
| 256 | 1 | 10.70 | 4.93 | 1.78 | 6.03x | 2.77x |
| 256 | 2 | 13.65 | 5.75 | 2.58 | 5.28x | 2.22x |
| 256 | 4 | 14.61 | 7.59 | 4.52 | 3.23x | 1.68x |
| 288 | 1 | 12.53 | 5.68 | 1.89 | 6.65x | 3.01x |
| 288 | 2 | 15.02 | 7.01 | 3.01 | 4.98x | 2.33x |
| 288 | 4 | 15.16 | 7.64 | 5.43 | 2.79x | 1.41x |
| 320 | 1 | 13.00 | 5.04 | 2.11 | 6.15x | 2.38x |
| 320 | 2 | 14.81 | 6.45 | 3.45 | 4.29x | 1.87x |
| 320 | 4 | 16.28 | 7.77 | 6.32 | 2.57x | 1.23x |
| 352 | 1 | 12.45 | 5.64 | 2.45 | 5.09x | 2.31x |
| 352 | 2 | 15.00 | 6.64 | 3.99 | 3.76x | 1.66x |
| 352 | 4 | 16.70 | 8.49 | 7.25 | 2.30x | 1.17x |
| 384 | 1 | 13.61 | 6.05 | 2.59 | 5.26x | 2.34x |
| 384 | 2 | 14.67 | 7.02 | 4.44 | 3.30x | 1.58x |
| 384 | 4 | 17.53 | 9.09 | 8.28 | 2.12x | 1.10x |
| 448 | 1 | 13.81 | 6.39 | 3.09 | 4.46x | 2.07x |
| 448 | 2 | 14.12 | 6.94 | 5.58 | 2.53x | 1.24x |
| 448 | 4 | 22.30 | 11.18 | 10.45 | 2.13x | 1.07x |
| 512 | 1 | 13.69 | 7.95 | 3.69 | 3.71x | 2.16x |
| 512 | 2 | 14.95 | 7.57 | 6.92 | 2.16x | 1.10x |
| 512 | 4 | 28.24 | 13.77 | 13.04 | 2.17x | 1.06x |
| 640 | 1 | 13.43 | 6.07 | 8.81 | 1.52x | 0.69x |
| 640 | 2 | 21.87 | 11.61 | 10.25 | 2.13x | 1.13x |
| 640 | 4 | 42.51 | 20.56 | 19.36 | 2.20x | 1.06x |
| 768 | 1 | 15.73 | 7.55 | 8.86 | 1.77x | 0.85x |
| 768 | 2 | 30.30 | 14.16 | 13.56 | 2.23x | 1.04x |
| 768 | 4 | 59.17 | 29.05 | 26.46 | 2.24x | 1.10x |
| 896 | 1 | 21.00 | 10.49 | 9.51 | 2.21x | 1.10x |
| 896 | 2 | 40.63 | 19.03 | 18.31 | 2.22x | 1.04x |
| 896 | 4 | 80.21 | 37.40 | 35.96 | 2.23x | 1.04x |
| 1024 | 1 | 26.40 | 12.16 | 11.64 | 2.27x | 1.05x |
| 1024 | 2 | 52.08 | 23.69 | 22.55 | 2.31x | 1.05x |
| 1024 | 4 | 103.28 | 45.49 | 45.10 | 2.29x | 1.01x |
*This SKU's benchmark sweeps batch 1 only.*
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 11.08 | 6.38 | 6.23 | 1.78x | 1.03x |
| 160 | 1 | 18.70 | 10.21 | 10.69 | 1.75x | 0.95x |
| 192 | 1 | 27.06 | 14.23 | 15.53 | 1.74x | 0.92x |
| 224 | 1 | 36.49 | 19.29 | 21.21 | 1.72x | 0.91x |
| 256 | 1 | 47.35 | 25.01 | 27.58 | 1.72x | 0.91x |
| 288 | 1 | 60.26 | 31.65 | 35.10 | 1.72x | 0.90x |
| 320 | 1 | 74.05 | 39.65 | 44.15 | 1.68x | 0.90x |
| 352 | 1 | 89.95 | 47.84 | 53.98 | 1.67x | 0.89x |
| 384 | 1 | 107.38 | 56.81 | 64.74 | 1.66x | 0.88x |
| 448 | 1 | 165.16 | 100.91 | 116.68 | 1.42x | 0.86x |
| 512 | 1 | 272.33 | 192.18 | 195.32 | 1.39x | 0.98x |
| 640 | 1 | 440.08 | 296.56 | 332.75 | 1.32x | 0.89x |
| 768 | 1 | 618.50 | 414.79 | 465.30 | 1.33x | 0.89x |
| 896 | 1 | 850.69 | 572.41 | 649.30 | 1.31x | 0.88x |
| 1024 | 1 | 1131.61 | 774.21 | 833.86 | 1.36x | 0.93x |
*This SKU's benchmark sweeps batch 1 only.*
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 6.55 | 4.88 | 2.08 | 3.16x | 2.35x |
| 160 | 1 | 6.57 | 4.95 | 2.56 | 2.56x | 1.93x |
| 192 | 1 | 8.77 | 5.24 | 3.38 | 2.60x | 1.55x |
| 224 | 1 | 11.79 | 7.14 | 4.08 | 2.89x | 1.75x |
| 256 | 1 | 15.58 | 8.70 | 4.94 | 3.15x | 1.76x |
| 288 | 1 | 18.44 | 10.13 | 6.11 | 3.02x | 1.66x |
| 320 | 1 | 22.11 | 11.89 | 7.37 | 3.00x | 1.61x |
| 352 | 1 | 27.21 | 14.71 | 8.39 | 3.24x | 1.75x |
| 384 | 1 | 32.37 | 17.15 | 9.74 | 3.32x | 1.76x |
| 448 | 1 | 45.69 | 24.62 | 13.00 | 3.52x | 1.89x |
| 512 | 1 | 65.67 | 36.29 | 16.44 | 4.00x | 2.21x |
| 640 | 1 | 114.42 | 68.76 | 25.49 | 4.49x | 2.70x |
| 768 | 1 | 169.72 | 102.43 | 36.24 | 4.68x | 2.83x |
| 896 | 1 | 231.09 | 139.09 | 49.20 | 4.70x | 2.83x |
| 1024 | 1 | 302.18 | 181.64 | 64.80 | 4.66x | 2.80x |
*This SKU's benchmark sweeps batch 1 only.*
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 10.42 | 4.33 | 1.92 | 5.42x | 2.25x |
| 160 | 1 | 10.50 | 4.44 | 2.36 | 4.45x | 1.88x |
| 192 | 1 | 10.51 | 4.79 | 2.93 | 3.58x | 1.63x |
| 224 | 1 | 11.92 | 6.30 | 3.60 | 3.31x | 1.75x |
| 256 | 1 | 15.57 | 8.04 | 4.34 | 3.59x | 1.85x |
| 288 | 1 | 18.11 | 9.23 | 5.28 | 3.43x | 1.75x |
| 320 | 1 | 21.52 | 10.86 | 6.22 | 3.46x | 1.75x |
| 352 | 1 | 26.54 | 13.45 | 7.11 | 3.73x | 1.89x |
| 384 | 1 | 31.37 | 15.44 | 8.24 | 3.81x | 1.87x |
| 448 | 1 | 44.16 | 22.24 | 10.89 | 4.06x | 2.04x |
| 512 | 1 | 63.31 | 33.73 | 13.66 | 4.63x | 2.47x |
| 640 | 1 | 111.98 | 67.36 | 21.11 | 5.31x | 3.19x |
| 768 | 1 | 166.56 | 100.33 | 29.86 | 5.58x | 3.36x |
| 896 | 1 | 226.38 | 136.43 | 40.47 | 5.59x | 3.37x |
| 1024 | 1 | 295.86 | 178.09 | 52.76 | 5.61x | 3.38x |
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 8.69 | 3.77 | 1.76 | 4.95x | 2.15x |
| 128 | 2 | 8.93 | 3.84 | 2.76 | 3.23x | 1.39x |
| 128 | 4 | 13.48 | 6.25 | 4.88 | 2.76x | 1.28x |
| 160 | 1 | 9.02 | 3.83 | 2.21 | 4.07x | 1.73x |
| 160 | 2 | 10.90 | 5.10 | 3.82 | 2.85x | 1.33x |
| 160 | 4 | 19.96 | 8.73 | 7.20 | 2.77x | 1.21x |
| 192 | 1 | 8.86 | 4.54 | 2.85 | 3.11x | 1.59x |
| 192 | 2 | 15.04 | 6.44 | 5.04 | 2.99x | 1.28x |
| 192 | 4 | 28.39 | 11.59 | 9.45 | 3.00x | 1.23x |
| 224 | 1 | 11.35 | 6.02 | 3.52 | 3.22x | 1.71x |
| 224 | 2 | 20.73 | 8.14 | 6.46 | 3.21x | 1.26x |
| 224 | 4 | 39.69 | 14.98 | 12.21 | 3.25x | 1.23x |
| 256 | 1 | 14.92 | 7.64 | 4.25 | 3.51x | 1.80x |
| 256 | 2 | 27.87 | 10.05 | 7.95 | 3.51x | 1.27x |
| 256 | 4 | 54.16 | 18.80 | 15.39 | 3.52x | 1.22x |
| 288 | 1 | 17.55 | 9.03 | 5.33 | 3.29x | 1.69x |
| 288 | 2 | 33.28 | 12.76 | 10.18 | 3.27x | 1.25x |
| 288 | 4 | 64.88 | 23.70 | 19.64 | 3.30x | 1.21x |
| 320 | 1 | 20.89 | 10.48 | 6.31 | 3.31x | 1.66x |
| 320 | 2 | 40.17 | 14.86 | 12.09 | 3.32x | 1.23x |
| 320 | 4 | 78.76 | 28.51 | 23.73 | 3.32x | 1.20x |
| 352 | 1 | 25.73 | 13.19 | 7.27 | 3.54x | 1.81x |
| 352 | 2 | 49.98 | 17.49 | 14.26 | 3.50x | 1.23x |
| 352 | 4 | 98.47 | 33.47 | 27.64 | 3.56x | 1.21x |
| 384 | 1 | 30.41 | 15.09 | 8.42 | 3.61x | 1.79x |
| 384 | 2 | 59.27 | 20.12 | 16.43 | 3.61x | 1.23x |
| 384 | 4 | 117.63 | 39.04 | 32.12 | 3.66x | 1.22x |
| 448 | 1 | 42.84 | 21.84 | 11.28 | 3.80x | 1.94x |
| 448 | 2 | 84.39 | 26.93 | 22.09 | 3.82x | 1.22x |
| 448 | 4 | 168.09 | 52.98 | 43.73 | 3.84x | 1.21x |
| 512 | 1 | 60.71 | 32.09 | 14.28 | 4.25x | 2.25x |
| 512 | 2 | 120.97 | 34.24 | 28.07 | 4.31x | 1.22x |
| 512 | 4 | 241.06 | 67.74 | 55.62 | 4.33x | 1.22x |
| 640 | 1 | 103.00 | 58.53 | 22.12 | 4.66x | 2.65x |
| 640 | 2 | 205.48 | 52.94 | 43.69 | 4.70x | 1.21x |
| 640 | 4 | 410.53 | 104.66 | 86.81 | 4.73x | 1.21x |
| 768 | 1 | 152.80 | 86.93 | 31.47 | 4.85x | 2.76x |
| 768 | 2 | 304.74 | 75.43 | 62.38 | 4.88x | 1.21x |
| 768 | 4 | 611.95 | 150.51 | 124.35 | 4.92x | 1.21x |
| 896 | 1 | 208.04 | 118.37 | 42.66 | 4.88x | 2.77x |
| 896 | 2 | 415.65 | 102.95 | 84.90 | 4.90x | 1.21x |
| 896 | 4 | 838.80 | 204.82 | 170.07 | 4.93x | 1.20x |
| 1024 | 1 | 271.61 | 154.49 | 55.85 | 4.86x | 2.77x |
| 1024 | 2 | 545.38 | 133.94 | 111.49 | 4.89x | 1.20x |
| 1024 | 4 | 1101.63 | 267.61 | 222.79 | 4.94x | 1.20x |
*This SKU's benchmark sweeps batch 1 only.*
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 6.58 | 2.91 | 1.42 | 4.63x | 2.05x |
| 160 | 1 | 6.67 | 3.51 | 1.81 | 3.67x | 1.93x |
| 192 | 1 | 7.42 | 4.70 | 2.49 | 2.98x | 1.89x |
| 224 | 1 | 10.62 | 6.35 | 3.36 | 3.16x | 1.89x |
| 256 | 1 | 15.49 | 8.28 | 4.42 | 3.51x | 1.88x |
| 288 | 1 | 20.30 | 10.53 | 5.79 | 3.50x | 1.82x |
| 320 | 1 | 25.75 | 13.36 | 7.16 | 3.60x | 1.87x |
| 352 | 1 | 31.95 | 16.81 | 10.02 | 3.19x | 1.68x |
| 384 | 1 | 38.58 | 20.22 | 12.92 | 2.99x | 1.56x |
| 448 | 1 | 53.22 | 27.86 | 14.15 | 3.76x | 1.97x |
| 512 | 1 | 71.50 | 37.42 | 18.29 | 3.91x | 2.05x |
| 640 | 1 | 110.99 | 58.05 | 28.17 | 3.94x | 2.06x |
| 768 | 1 | 163.30 | 85.47 | 41.57 | 3.93x | 2.06x |
| 896 | 1 | 272.86 | 171.07 | 57.52 | 4.74x | 2.97x |
| 1024 | 1 | 449.06 | 307.58 | 76.08 | 5.90x | 4.04x |
*This SKU's benchmark sweeps batch 1 only.*
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 8.04 | 3.33 | 1.39 | 5.79x | 2.39x |
| 160 | 1 | 8.00 | 3.54 | 1.95 | 4.09x | 1.81x |
| 192 | 1 | 8.11 | 4.65 | 2.62 | 3.10x | 1.77x |
| 224 | 1 | 10.20 | 6.49 | 3.63 | 2.81x | 1.79x |
| 256 | 1 | 14.30 | 8.32 | 4.86 | 2.95x | 1.71x |
| 288 | 1 | 18.64 | 10.78 | 6.36 | 2.93x | 1.70x |
| 320 | 1 | 23.18 | 13.07 | 8.05 | 2.88x | 1.63x |
| 352 | 1 | 28.56 | 16.56 | 10.59 | 2.70x | 1.56x |
| 384 | 1 | 34.18 | 19.73 | 12.82 | 2.67x | 1.54x |
| 448 | 1 | 47.39 | 27.22 | 17.33 | 2.73x | 1.57x |
| 512 | 1 | 63.69 | 36.58 | 22.72 | 2.80x | 1.61x |
| 640 | 1 | 99.74 | 57.80 | 36.52 | 2.73x | 1.58x |
| 768 | 1 | 148.09 | 86.48 | 53.80 | 2.75x | 1.61x |
| 896 | 1 | 246.66 | 160.44 | 74.02 | 3.33x | 2.17x |
| 1024 | 1 | 407.31 | 292.16 | 97.86 | 4.16x | 2.99x |
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 6.89 | 2.82 | 1.23 | 5.60x | 2.29x |
| 128 | 2 | 7.06 | 3.07 | 1.72 | 4.10x | 1.78x |
| 128 | 4 | 10.29 | 4.54 | 2.86 | 3.60x | 1.59x |
| 160 | 1 | 7.00 | 3.05 | 1.30 | 5.39x | 2.34x |
| 160 | 2 | 8.28 | 3.67 | 2.22 | 3.73x | 1.66x |
| 160 | 4 | 8.98 | 6.00 | 3.99 | 2.25x | 1.50x |
| 192 | 1 | 7.06 | 3.14 | 1.62 | 4.35x | 1.93x |
| 192 | 2 | 11.08 | 4.60 | 2.88 | 3.85x | 1.60x |
| 192 | 4 | 11.07 | 6.97 | 5.21 | 2.13x | 1.34x |
| 224 | 1 | 7.94 | 3.46 | 2.03 | 3.92x | 1.71x |
| 224 | 2 | 8.47 | 5.14 | 3.50 | 2.42x | 1.47x |
| 224 | 4 | 14.27 | 8.98 | 6.46 | 2.21x | 1.39x |
| 256 | 1 | 7.96 | 3.57 | 2.41 | 3.31x | 1.48x |
| 256 | 2 | 9.62 | 6.03 | 4.24 | 2.27x | 1.42x |
| 256 | 4 | 18.11 | 11.30 | 8.06 | 2.25x | 1.40x |
| 288 | 1 | 8.09 | 4.34 | 2.88 | 2.81x | 1.51x |
| 288 | 2 | 11.80 | 7.43 | 5.34 | 2.21x | 1.39x |
| 288 | 4 | 22.43 | 13.71 | 10.05 | 2.23x | 1.36x |
| 320 | 1 | 8.39 | 5.02 | 3.42 | 2.45x | 1.47x |
| 320 | 2 | 14.16 | 8.59 | 6.31 | 2.24x | 1.36x |
| 320 | 4 | 27.48 | 16.56 | 12.02 | 2.29x | 1.38x |
| 352 | 1 | 9.81 | 5.84 | 3.92 | 2.50x | 1.49x |
| 352 | 2 | 16.77 | 10.36 | 7.35 | 2.28x | 1.41x |
| 352 | 4 | 32.38 | 19.55 | 14.29 | 2.27x | 1.37x |
| 384 | 1 | 10.37 | 6.37 | 4.54 | 2.29x | 1.40x |
| 384 | 2 | 19.54 | 11.72 | 8.62 | 2.27x | 1.36x |
| 384 | 4 | 37.90 | 22.82 | 16.70 | 2.27x | 1.37x |
| 448 | 1 | 13.50 | 8.15 | 5.83 | 2.31x | 1.40x |
| 448 | 2 | 25.89 | 15.50 | 11.23 | 2.31x | 1.38x |
| 448 | 4 | 50.71 | 29.99 | 22.01 | 2.30x | 1.36x |
| 512 | 1 | 17.11 | 10.34 | 7.42 | 2.31x | 1.39x |
| 512 | 2 | 33.26 | 19.78 | 14.20 | 2.34x | 1.39x |
| 512 | 4 | 65.47 | 38.62 | 28.02 | 2.34x | 1.38x |
| 640 | 1 | 25.80 | 15.24 | 11.27 | 2.29x | 1.35x |
| 640 | 2 | 50.77 | 29.69 | 21.81 | 2.33x | 1.36x |
| 640 | 4 | 100.71 | 58.66 | 43.06 | 2.34x | 1.36x |
| 768 | 1 | 36.41 | 21.62 | 15.44 | 2.36x | 1.40x |
| 768 | 2 | 72.20 | 41.99 | 30.09 | 2.40x | 1.40x |
| 768 | 4 | 143.25 | 83.43 | 61.09 | 2.35x | 1.37x |
| 896 | 1 | 49.10 | 28.61 | 21.23 | 2.31x | 1.35x |
| 896 | 2 | 98.27 | 56.92 | 42.12 | 2.33x | 1.35x |
| 896 | 4 | 195.63 | 112.82 | 84.84 | 2.31x | 1.33x |
| 1024 | 1 | 63.56 | 36.92 | 27.02 | 2.35x | 1.37x |
| 1024 | 2 | 127.40 | 73.16 | 53.87 | 2.36x | 1.36x |
| 1024 | 4 | 255.10 | 146.18 | 108.00 | 2.36x | 1.35x |
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 6.74 | 2.78 | 1.06 | 6.33x | 2.61x |
| 128 | 2 | 7.06 | 2.94 | 1.43 | 4.92x | 2.05x |
| 128 | 4 | 8.31 | 3.48 | 2.50 | 3.32x | 1.39x |
| 160 | 1 | 6.76 | 2.89 | 1.22 | 5.56x | 2.38x |
| 160 | 2 | 7.41 | 3.11 | 2.25 | 3.29x | 1.38x |
| 160 | 4 | 8.42 | 4.46 | 3.53 | 2.38x | 1.26x |
| 192 | 1 | 7.04 | 2.96 | 1.49 | 4.74x | 1.99x |
| 192 | 2 | 8.81 | 3.50 | 2.50 | 3.53x | 1.40x |
| 192 | 4 | 8.89 | 5.60 | 4.45 | 2.00x | 1.26x |
| 224 | 1 | 7.00 | 3.10 | 1.72 | 4.06x | 1.80x |
| 224 | 2 | 11.47 | 4.44 | 3.03 | 3.79x | 1.47x |
| 224 | 4 | 11.46 | 7.14 | 5.58 | 2.05x | 1.28x |
| 256 | 1 | 7.70 | 3.26 | 2.04 | 3.77x | 1.60x |
| 256 | 2 | 8.27 | 4.76 | 3.68 | 2.25x | 1.30x |
| 256 | 4 | 14.40 | 8.86 | 6.91 | 2.08x | 1.28x |
| 288 | 1 | 7.85 | 3.44 | 2.49 | 3.16x | 1.38x |
| 288 | 2 | 9.41 | 5.93 | 4.58 | 2.05x | 1.30x |
| 288 | 4 | 18.01 | 10.93 | 8.66 | 2.08x | 1.26x |
| 320 | 1 | 7.85 | 3.90 | 2.99 | 2.63x | 1.31x |
| 320 | 2 | 11.29 | 6.86 | 5.43 | 2.08x | 1.26x |
| 320 | 4 | 21.79 | 13.16 | 10.39 | 2.10x | 1.27x |
| 352 | 1 | 7.87 | 4.61 | 3.38 | 2.33x | 1.36x |
| 352 | 2 | 13.32 | 8.19 | 6.20 | 2.15x | 1.32x |
| 352 | 4 | 25.97 | 15.43 | 12.12 | 2.14x | 1.27x |
| 384 | 1 | 8.22 | 4.96 | 3.78 | 2.17x | 1.31x |
| 384 | 2 | 15.57 | 9.25 | 7.15 | 2.18x | 1.29x |
| 384 | 4 | 30.26 | 17.90 | 14.01 | 2.16x | 1.28x |
| 448 | 1 | 10.72 | 6.49 | 4.93 | 2.17x | 1.32x |
| 448 | 2 | 20.82 | 12.24 | 9.74 | 2.14x | 1.26x |
| 448 | 4 | 40.83 | 24.18 | 18.88 | 2.16x | 1.28x |
| 512 | 1 | 13.62 | 8.07 | 6.12 | 2.22x | 1.32x |
| 512 | 2 | 26.55 | 15.48 | 12.04 | 2.20x | 1.29x |
| 512 | 4 | 52.54 | 30.45 | 23.85 | 2.20x | 1.28x |
| 640 | 1 | 20.70 | 12.13 | 9.51 | 2.18x | 1.28x |
| 640 | 2 | 40.66 | 23.92 | 18.54 | 2.19x | 1.29x |
| 640 | 4 | 80.70 | 47.71 | 36.99 | 2.18x | 1.29x |
| 768 | 1 | 29.19 | 17.05 | 13.15 | 2.22x | 1.30x |
| 768 | 2 | 57.88 | 34.16 | 26.21 | 2.21x | 1.30x |
| 768 | 4 | 114.95 | 68.55 | 53.13 | 2.16x | 1.29x |
| 896 | 1 | 39.69 | 23.34 | 18.01 | 2.20x | 1.30x |
| 896 | 2 | 79.05 | 46.67 | 35.90 | 2.20x | 1.30x |
| 896 | 4 | 157.34 | 92.67 | 73.74 | 2.13x | 1.26x |
| 1024 | 1 | 51.19 | 30.12 | 23.25 | 2.20x | 1.30x |
| 1024 | 2 | 102.52 | 60.16 | 46.91 | 2.19x | 1.28x |
| 1024 | 4 | 204.19 | 120.15 | 94.36 | 2.16x | 1.27x |
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 4.92 | 2.86 | 0.96 | 5.13x | 2.98x |
| 128 | 2 | 5.10 | 2.93 | 1.33 | 3.84x | 2.21x |
| 128 | 4 | 7.74 | 3.94 | 2.13 | 3.64x | 1.85x |
| 160 | 1 | 4.97 | 2.94 | 1.27 | 3.91x | 2.31x |
| 160 | 2 | 6.31 | 2.97 | 1.80 | 3.50x | 1.65x |
| 160 | 4 | 6.56 | 3.98 | 2.90 | 2.26x | 1.37x |
| 192 | 1 | 5.22 | 2.99 | 1.36 | 3.85x | 2.20x |
| 192 | 2 | 8.23 | 3.30 | 2.10 | 3.93x | 1.57x |
| 192 | 4 | 7.89 | 4.80 | 3.58 | 2.21x | 1.34x |
| 224 | 1 | 6.05 | 2.93 | 1.50 | 4.04x | 1.96x |
| 224 | 2 | 10.53 | 3.87 | 2.49 | 4.23x | 1.56x |
| 224 | 4 | 9.95 | 5.96 | 4.44 | 2.24x | 1.34x |
| 256 | 1 | 5.98 | 3.21 | 1.73 | 3.45x | 1.85x |
| 256 | 2 | 6.90 | 4.19 | 2.96 | 2.33x | 1.42x |
| 256 | 4 | 12.34 | 7.32 | 5.50 | 2.24x | 1.33x |
| 288 | 1 | 6.16 | 3.35 | 2.05 | 3.00x | 1.63x |
| 288 | 2 | 8.29 | 4.94 | 3.65 | 2.27x | 1.35x |
| 288 | 4 | 15.13 | 8.83 | 6.87 | 2.20x | 1.29x |
| 320 | 1 | 6.12 | 3.50 | 2.44 | 2.51x | 1.43x |
| 320 | 2 | 9.74 | 5.61 | 4.31 | 2.26x | 1.30x |
| 320 | 4 | 18.10 | 10.45 | 8.19 | 2.21x | 1.28x |
| 352 | 1 | 6.77 | 3.94 | 2.72 | 2.49x | 1.45x |
| 352 | 2 | 11.34 | 6.59 | 4.94 | 2.29x | 1.33x |
| 352 | 4 | 21.43 | 12.26 | 9.57 | 2.24x | 1.28x |
| 384 | 1 | 7.23 | 4.20 | 3.02 | 2.39x | 1.39x |
| 384 | 2 | 13.13 | 7.52 | 5.71 | 2.30x | 1.32x |
| 384 | 4 | 25.00 | 14.24 | 11.04 | 2.27x | 1.29x |
| 448 | 1 | 9.24 | 5.34 | 3.95 | 2.34x | 1.35x |
| 448 | 2 | 17.22 | 9.76 | 7.50 | 2.30x | 1.30x |
| 448 | 4 | 33.13 | 18.88 | 14.68 | 2.26x | 1.29x |
| 512 | 1 | 11.50 | 6.52 | 4.82 | 2.39x | 1.35x |
| 512 | 2 | 21.83 | 12.24 | 9.40 | 2.32x | 1.30x |
| 512 | 4 | 42.43 | 23.71 | 18.44 | 2.30x | 1.29x |
| 640 | 1 | 17.07 | 10.26 | 7.55 | 2.26x | 1.36x |
| 640 | 2 | 33.01 | 18.40 | 14.57 | 2.27x | 1.26x |
| 640 | 4 | 65.15 | 36.41 | 28.75 | 2.27x | 1.27x |
| 768 | 1 | 23.84 | 13.49 | 10.19 | 2.34x | 1.32x |
| 768 | 2 | 46.91 | 26.10 | 20.15 | 2.33x | 1.29x |
| 768 | 4 | 93.03 | 51.79 | 40.47 | 2.30x | 1.28x |
| 896 | 1 | 31.95 | 17.93 | 14.04 | 2.28x | 1.28x |
| 896 | 2 | 64.49 | 36.09 | 27.81 | 2.32x | 1.30x |
| 896 | 4 | 128.13 | 71.59 | 56.06 | 2.29x | 1.28x |
| 1024 | 1 | 41.27 | 22.97 | 17.93 | 2.30x | 1.28x |
| 1024 | 2 | 82.18 | 45.80 | 35.87 | 2.29x | 1.28x |
| 1024 | 4 | 164.67 | 92.02 | 71.87 | 2.29x | 1.28x |
| Tokens | Batch | `oss_eager` (s) | `oss_torch_compile` (s) | `bioir` (s) | eager÷bioir | compile÷bioir |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 128 | 1 | 10.37 | 6.55 | 1.13 | 9.18x | 5.81x |
| 128 | 2 | 10.88 | 6.26 | 1.43 | 7.59x | 4.37x |
| 128 | 4 | 11.48 | 7.00 | 2.35 | 4.89x | 2.98x |
| 160 | 1 | 11.12 | 7.01 | 1.29 | 8.64x | 5.44x |
| 160 | 2 | 12.54 | 6.17 | 1.99 | 6.31x | 3.11x |
| 160 | 4 | 12.38 | 8.47 | 3.16 | 3.92x | 2.68x |
| 192 | 1 | 10.86 | 7.12 | 1.51 | 7.19x | 4.71x |
| 192 | 2 | 11.09 | 7.03 | 2.21 | 5.01x | 3.18x |
| 192 | 4 | 12.70 | 9.29 | 3.79 | 3.35x | 2.45x |
| 224 | 1 | 10.52 | 6.64 | 1.57 | 6.71x | 4.24x |
| 224 | 2 | 11.15 | 7.83 | 2.63 | 4.23x | 2.97x |
| 224 | 4 | 13.21 | 8.96 | 4.63 | 2.85x | 1.93x |
| 256 | 1 | 11.30 | 7.02 | 1.83 | 6.17x | 3.84x |
| 256 | 2 | 12.02 | 7.57 | 3.02 | 3.98x | 2.51x |
| 256 | 4 | 13.68 | 8.59 | 5.52 | 2.48x | 1.56x |
| 288 | 1 | 11.65 | 7.42 | 2.18 | 5.35x | 3.41x |
| 288 | 2 | 12.47 | 8.33 | 3.81 | 3.27x | 2.19x |
| 288 | 4 | 15.34 | 10.48 | 7.07 | 2.17x | 1.48x |
| 320 | 1 | 12.25 | 7.64 | 2.51 | 4.89x | 3.05x |
| 320 | 2 | 12.75 | 7.59 | 4.32 | 2.95x | 1.76x |
| 320 | 4 | 17.60 | 11.30 | 8.34 | 2.11x | 1.35x |
| 352 | 1 | 11.94 | 7.16 | 2.87 | 4.16x | 2.49x |
| 352 | 2 | 13.56 | 8.34 | 5.00 | 2.71x | 1.67x |
| 352 | 4 | 20.48 | 13.61 | 9.61 | 2.13x | 1.42x |
| 384 | 1 | 14.44 | 8.04 | 3.11 | 4.65x | 2.59x |
| 384 | 2 | 14.89 | 9.39 | 5.67 | 2.63x | 1.66x |
| 384 | 4 | 23.90 | 14.77 | 10.92 | 2.19x | 1.35x |
| 448 | 1 | 14.41 | 7.27 | 4.00 | 3.60x | 1.82x |
| 448 | 2 | 16.36 | 10.60 | 7.38 | 2.22x | 1.44x |
| 448 | 4 | 31.27 | 18.71 | 14.61 | 2.14x | 1.28x |
| 512 | 1 | 14.54 | 9.78 | 4.85 | 3.00x | 2.02x |
| 512 | 2 | 20.65 | 13.07 | 9.35 | 2.21x | 1.40x |
| 512 | 4 | 39.52 | 24.54 | 18.06 | 2.19x | 1.36x |
| 640 | 1 | 16.22 | 10.27 | 7.60 | 2.13x | 1.35x |
| 640 | 2 | 31.33 | 19.43 | 14.30 | 2.19x | 1.36x |
| 640 | 4 | 60.65 | 35.94 | 28.02 | 2.16x | 1.28x |
| 768 | 1 | 22.41 | 13.71 | 10.39 | 2.16x | 1.32x |
| 768 | 2 | 43.82 | 25.98 | 19.78 | 2.22x | 1.31x |
| 768 | 4 | 86.53 | 50.59 | 39.22 | 2.21x | 1.29x |
| 896 | 1 | 29.90 | 18.10 | 13.83 | 2.16x | 1.31x |
| 896 | 2 | 59.05 | 34.63 | 26.99 | 2.19x | 1.28x |
| 896 | 4 | 117.42 | 69.20 | 54.31 | 2.16x | 1.27x |
| 1024 | 1 | 38.37 | 22.61 | 17.47 | 2.20x | 1.29x |
| 1024 | 2 | 76.44 | 45.74 | 34.73 | 2.20x | 1.32x |
| 1024 | 4 | 152.35 | 88.67 | 68.90 | 2.21x | 1.29x |
Three things the grid shows.
**Design time is super-linear in token count**, because the pair track is
quadratic: on H100 80GB HBM3 with `bioir` at batch 1, an eightfold rise in
tokens, 128 to 1024, costs 15.3x the time. The spread between GPUs widens with
it. At 128 tokens the datacenter parts are about 2.3x apart; at 1024 they fall
into bands — 11.6 to 13.9 s on the Blackwell datacenter parts, 16.9 to 27.0 s
on Hopper, 51.3 to 64.8 s on Ampere, and 76.1 to 97.9 s on the Ada-generation
parts. GPU choice therefore matters far more for long designs than for short
ones. The GB10 (DGX Spark) sits well outside those bands at 833.9 s, which is
what a desktop part costs on the longest design this suite sends.
**Batching is close to free at short lengths and close to linear at long
ones.** On H100 at 128 tokens, `oss_eager` returns four designs in 8.25 s
against 6.82 s for one — 1.21x the time for 4x the output. `bioir` starts from
a far lower per-request floor and so saturates sooner, at 2.08x. By 1024
tokens the GPU is saturated either way and batch 4 costs about 3.9x on both. A
deployment serving short designs should batch; one serving long designs gains
throughput from batching but little latency.
**`bioir` leads on most parts, by a margin that narrows with both length and
batch.** On H100 it is 5.83x over `oss_eager` and 2.67x over
`oss_torch_compile` at 128 tokens and batch 1, 3.39x and 1.61x at the same
length and batch 4, and settles near 2.3x and 1.38x from 512 tokens upward
regardless of batch. The short-length, batch-1 figures are the most
flattering ones and should not be quoted as the general case.
## Accuracy Across Execution Backends
The eval suite's structural-quality metrics, on one H100 80GB HBM3, from the
same release candidate as the performance table above. Each backend ran the
same 87 conditioned designs. `n` is how many of those 87 the metric is defined
on.
| Metric | Better | `n` | `oss_eager` | `oss_torch_compile` | `bioir` |
| --- | :---: | ---: | ---: | ---: | ---: |
| `success_rate` | higher | 87 | 1.000 | 1.000 | 1.000 |
| `clash_frac` mean | lower | 87 | 0.0134 | 0.0105 | 0.0106 |
| `clash_frac` max | lower | 87 | 0.917 | 0.833 | 0.917 |
| `bond_length_unreasonable` mean | lower | 87 | 0.00023 | 0.00025 | 0.00031 |
| `bond_length_unreasonable` max | lower | 87 | 0.0082 | 0.0108 | 0.0041 |
| `sequence_adherence` mean | higher | 57 | 1.000 | 1.000 | 0.994 |
| `ss_conditioning_accuracy` mean | higher | 1 | 0.663 | 0.807 | 0.659 |
Every design completed on every backend, and the structural-quality means sit
within a few thousandths of one another. Results are comparable in kind across
backends but not bit-identical — compilation and BioIR's own kernels both
change floating-point numerics — so small differences between the three
columns are expected and none of the gaps here is larger than that.
These are means over a small evaluation set, not a published accuracy
benchmark, and two rows are weaker than the rest. `sequence_adherence` is
defined on 57 of the 87 designs, and `ss_conditioning_accuracy` on exactly
one — the spread from 0.659 to 0.807 in that last row is a single design and
carries no information about backend quality. Do not select a backend on
these numbers.
## Cold-Start Behavior
The NIM reports ready as soon as its weights are loaded; it does not pre-warm
every token-count bucket the `oss_torch_compile` backend compiles for. This
means the *first* request against a given token bucket after startup pays a
Triton JIT compilation cost, once per bucket per container start, and later requests against the same (or
a smaller, same-bucket) size run at steady-state speed. This is a deliberate
trade: a NIM that is servable in well under a minute, with the compile cost
spread across a handful of early requests, versus one that is uniformly fast
but takes several minutes to report ready.
Measured on one H100, `thinking_medium` (the checkpoint's default recycle
strategy), one design:
| Tokens | Bucket | Cold (first request in bucket) | Steady state | Penalty |
|---|---|---|---|---|
| 100 | 128 | 30.3 s | 3.1 s | 9.8x |
| 150 | 160 | 22.2 s | 3.6 s | 6.2x |
| 250 | 256 | 21.9 s | 4.4 s | 5.0x |
| 650 | 768 | 54.9 s | 28.4 s | 1.9x |
| 694 | 768 (already warm from 650) | 28.5 s | 28.4 s | 1.0x |
Two things follow from this table. First, the penalty is per *bucket*, not
"the first request ever": a deployment serving a wide range of input sizes
pays this cost once per distinct bucket its clients exercise, scattered
across whichever requests happen to land first. Second, after a bucket is
warm, a smaller request that rounds up into that same bucket (694 tokens
padding to the 768 bucket already warmed by a 650-token request) pays nothing
extra.
If a deployment or a demo needs a specific input size to respond quickly from
the first request, send one request of that size to warm its bucket before
the size matters.
## Memory Footprint
This NIM ships with no refolding oracle built in (refer to
[Oracles](/nim/bionemo/rfd4-proteina/oracle)), so the design model's own memory
footprint is what a deployment plans capacity around. That footprint grows
with the square of the token count and with the batch size, and
`sampling.batch_size` defaults to `sampling.num_replicates`: a multi-replicate
request is a single large forward pass unless it says otherwise. Refer to
[Replicates and
Batching](/nim/bionemo/rfd4-proteina/prerequisites#replicates-and-batching). A mounted oracle that
calls out to an independently hosted structure-prediction NIM (the
reference implementations this NIM ships against OpenFold3 and Boltz-2 both
work this way) adds no GPU memory cost of its own here, since the model it
calls lives entirely in that other service.