Performance in RFD4-Proteina NIM#
Execution Backends#
The RFD4-Proteina NIM can run three execution backends, selected per deployment (refer to Configuration Alternatives):
oss_eager: the upstream reference network, uncompiled.oss_torch_compile: the same network undertorch.compile.bioir: an NVIDIA-accelerated trunk built on the BioNeMo Inference Runtime, with its own fused kernels and CUDA-graph capture.
Every response reports which backend actually ran it in
model.execution_backend, alongside runtime.cuda_graphs and
runtime.cuda_graph_cache, which describe what the loaded runtime currently
holds. Results are comparable in kind across backends but not bit-identical,
since compilation and BioIR’s own kernels both change floating-point
numerics.
Benchmark Configuration#
The table in this section comes from a single release-candidate sweep of the
synthetic latency suite, 1.0.0-rc.20261012093607, measured 2026-10-12.
Parameter |
Setting |
|---|---|
Task |
|
Input |
synthetic poly-glycine backbone, |
|
|
|
1, 2, and 4, swept at every token bucket |
|
|
Token buckets |
128, 160, 192, 224, 256, 288, 320, 352, 384, 448, 512, 640, 768, 896, 1024 |
Runs per shape |
3, of which the cold first run is discarded and the warm median reported |
Backends |
|
Design time is the client’s end-to-end wall clock around one HTTP request. It therefore covers input parsing, sampling, metrics, and response encoding, not only the trunk forward passes. Because the cold run is excluded, these are steady-state figures; refer to Cold-Start Behavior for what the first request into a bucket costs instead.
Replicates are batched into one forward pass, so a batch-n request is a
single request returning n designs. Divide by the batch size for a
per-design cost. Refer to Replicates and
Batching.
Five GPUs sweep batch 1 only — GB10, both 40 GB A100s, L40S, and RTX 6000 Ada Generation — so their tabs carry 15 rows rather than 45, and each one says so.
Performance Across Execution Backends#
All three execution backends over the full sweep, at every token bucket and
every batch size that GPU was swept at. Select a GPU to switch the table; the
two rightmost columns are bioir’s speedup over each OSS baseline, where
higher means bioir is faster. A dash marks a shape that did not complete.
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
6.82 |
3.12 |
1.17 |
5.83x |
2.67x |
128 |
2 |
7.04 |
3.35 |
1.44 |
4.88x |
2.33x |
128 |
4 |
8.25 |
3.93 |
2.44 |
3.39x |
1.61x |
160 |
1 |
6.85 |
3.26 |
1.23 |
5.56x |
2.65x |
160 |
2 |
7.09 |
3.51 |
2.00 |
3.54x |
1.75x |
160 |
4 |
8.87 |
4.74 |
3.34 |
2.66x |
1.42x |
192 |
1 |
6.87 |
3.35 |
1.49 |
4.60x |
2.24x |
192 |
2 |
8.71 |
3.75 |
2.39 |
3.64x |
1.57x |
192 |
4 |
9.14 |
5.62 |
4.12 |
2.22x |
1.36x |
224 |
1 |
6.93 |
3.40 |
1.65 |
4.19x |
2.06x |
224 |
2 |
11.10 |
4.51 |
2.76 |
4.02x |
1.63x |
224 |
4 |
10.69 |
6.86 |
5.02 |
2.13x |
1.37x |
256 |
1 |
8.05 |
3.88 |
1.91 |
4.22x |
2.03x |
256 |
2 |
8.67 |
4.79 |
3.28 |
2.64x |
1.46x |
256 |
4 |
13.18 |
8.36 |
6.09 |
2.16x |
1.37x |
288 |
1 |
8.20 |
3.96 |
2.28 |
3.60x |
1.74x |
288 |
2 |
8.90 |
5.70 |
3.99 |
2.23x |
1.43x |
288 |
4 |
16.00 |
10.17 |
7.55 |
2.12x |
1.35x |
320 |
1 |
8.24 |
4.02 |
2.63 |
3.14x |
1.53x |
320 |
2 |
10.32 |
6.50 |
4.73 |
2.18x |
1.37x |
320 |
4 |
19.13 |
11.87 |
8.90 |
2.15x |
1.33x |
352 |
1 |
8.30 |
4.49 |
2.94 |
2.82x |
1.53x |
352 |
2 |
12.00 |
7.57 |
5.36 |
2.24x |
1.41x |
352 |
4 |
22.57 |
13.88 |
10.40 |
2.17x |
1.34x |
384 |
1 |
8.53 |
4.74 |
3.30 |
2.59x |
1.44x |
384 |
2 |
13.81 |
8.47 |
6.18 |
2.24x |
1.37x |
384 |
4 |
26.29 |
16.15 |
11.85 |
2.22x |
1.36x |
448 |
1 |
9.75 |
5.93 |
4.21 |
2.32x |
1.41x |
448 |
2 |
18.03 |
10.97 |
7.95 |
2.27x |
1.38x |
448 |
4 |
34.64 |
21.48 |
15.61 |
2.22x |
1.38x |
512 |
1 |
12.02 |
7.29 |
5.17 |
2.32x |
1.41x |
512 |
2 |
22.79 |
13.62 |
9.98 |
2.28x |
1.36x |
512 |
4 |
44.20 |
26.46 |
19.32 |
2.29x |
1.37x |
640 |
1 |
17.69 |
10.56 |
7.79 |
2.27x |
1.36x |
640 |
2 |
34.38 |
20.32 |
15.00 |
2.29x |
1.35x |
640 |
4 |
67.22 |
39.86 |
29.35 |
2.29x |
1.36x |
768 |
1 |
24.65 |
14.58 |
10.68 |
2.31x |
1.37x |
768 |
2 |
48.16 |
28.44 |
20.57 |
2.34x |
1.38x |
768 |
4 |
95.19 |
55.93 |
40.69 |
2.34x |
1.37x |
896 |
1 |
32.82 |
19.45 |
14.36 |
2.29x |
1.35x |
896 |
2 |
65.12 |
38.30 |
28.10 |
2.32x |
1.36x |
896 |
4 |
128.90 |
75.59 |
56.00 |
2.30x |
1.35x |
1024 |
1 |
42.05 |
24.74 |
17.89 |
2.35x |
1.38x |
1024 |
2 |
83.45 |
49.12 |
35.31 |
2.36x |
1.39x |
1024 |
4 |
166.68 |
97.17 |
69.98 |
2.38x |
1.39x |
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
6.15 |
2.66 |
0.91 |
6.79x |
2.93x |
128 |
2 |
6.12 |
2.84 |
1.30 |
4.70x |
2.18x |
128 |
4 |
7.42 |
3.35 |
2.16 |
3.43x |
1.55x |
160 |
1 |
6.01 |
2.81 |
1.10 |
5.47x |
2.55x |
160 |
2 |
6.25 |
3.05 |
1.75 |
3.57x |
1.74x |
160 |
4 |
7.73 |
3.95 |
2.88 |
2.69x |
1.37x |
192 |
1 |
6.03 |
2.86 |
1.32 |
4.57x |
2.16x |
192 |
2 |
7.82 |
3.14 |
2.08 |
3.76x |
1.51x |
192 |
4 |
7.99 |
4.62 |
3.62 |
2.21x |
1.28x |
224 |
1 |
6.01 |
2.94 |
1.44 |
4.17x |
2.04x |
224 |
2 |
9.99 |
3.64 |
2.44 |
4.10x |
1.50x |
224 |
4 |
9.55 |
5.82 |
4.45 |
2.15x |
1.31x |
256 |
1 |
7.11 |
3.37 |
1.69 |
4.20x |
1.99x |
256 |
2 |
7.65 |
3.97 |
2.90 |
2.64x |
1.37x |
256 |
4 |
11.73 |
6.97 |
5.43 |
2.16x |
1.28x |
288 |
1 |
7.23 |
3.48 |
2.00 |
3.62x |
1.74x |
288 |
2 |
7.92 |
4.72 |
3.56 |
2.23x |
1.33x |
288 |
4 |
14.33 |
8.52 |
6.79 |
2.11x |
1.25x |
320 |
1 |
7.24 |
3.48 |
2.33 |
3.11x |
1.49x |
320 |
2 |
9.18 |
5.38 |
4.21 |
2.18x |
1.28x |
320 |
4 |
17.05 |
10.11 |
7.98 |
2.14x |
1.27x |
352 |
1 |
6.99 |
3.67 |
2.60 |
2.69x |
1.41x |
352 |
2 |
10.72 |
6.33 |
4.81 |
2.23x |
1.32x |
352 |
4 |
20.16 |
11.91 |
9.29 |
2.17x |
1.28x |
384 |
1 |
7.50 |
4.09 |
2.94 |
2.55x |
1.39x |
384 |
2 |
12.40 |
7.32 |
5.53 |
2.24x |
1.32x |
384 |
4 |
23.42 |
13.67 |
10.71 |
2.19x |
1.28x |
448 |
1 |
8.76 |
5.04 |
3.77 |
2.32x |
1.34x |
448 |
2 |
16.10 |
9.29 |
7.22 |
2.23x |
1.29x |
448 |
4 |
30.90 |
18.13 |
14.18 |
2.18x |
1.28x |
512 |
1 |
10.85 |
6.15 |
4.67 |
2.32x |
1.32x |
512 |
2 |
20.41 |
11.71 |
9.01 |
2.26x |
1.30x |
512 |
4 |
39.49 |
22.50 |
17.74 |
2.23x |
1.27x |
640 |
1 |
15.86 |
9.11 |
7.16 |
2.22x |
1.27x |
640 |
2 |
30.93 |
17.50 |
13.88 |
2.23x |
1.26x |
640 |
4 |
60.43 |
34.44 |
27.37 |
2.21x |
1.26x |
768 |
1 |
22.10 |
12.57 |
9.72 |
2.27x |
1.29x |
768 |
2 |
43.42 |
24.79 |
19.34 |
2.24x |
1.28x |
768 |
4 |
85.86 |
48.55 |
38.26 |
2.24x |
1.27x |
896 |
1 |
29.54 |
16.98 |
13.35 |
2.21x |
1.27x |
896 |
2 |
59.09 |
33.23 |
26.32 |
2.25x |
1.26x |
896 |
4 |
116.96 |
66.69 |
52.88 |
2.21x |
1.26x |
1024 |
1 |
38.02 |
21.71 |
16.95 |
2.24x |
1.28x |
1024 |
2 |
75.89 |
42.80 |
33.74 |
2.25x |
1.27x |
1024 |
4 |
151.45 |
84.61 |
67.08 |
2.26x |
1.26x |
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
9.63 |
4.17 |
1.75 |
5.51x |
2.39x |
128 |
2 |
9.95 |
4.14 |
2.82 |
3.53x |
1.47x |
128 |
4 |
13.31 |
6.16 |
4.84 |
2.75x |
1.27x |
160 |
1 |
9.67 |
4.00 |
2.25 |
4.29x |
1.78x |
160 |
2 |
10.81 |
4.99 |
3.82 |
2.83x |
1.31x |
160 |
4 |
19.42 |
8.57 |
6.98 |
2.78x |
1.23x |
192 |
1 |
9.68 |
4.52 |
2.81 |
3.44x |
1.61x |
192 |
2 |
14.77 |
6.25 |
4.97 |
2.97x |
1.26x |
192 |
4 |
27.22 |
11.41 |
9.35 |
2.91x |
1.22x |
224 |
1 |
11.28 |
5.97 |
3.50 |
3.23x |
1.71x |
224 |
2 |
20.34 |
7.95 |
6.25 |
3.25x |
1.27x |
224 |
4 |
38.45 |
14.64 |
11.81 |
3.26x |
1.24x |
256 |
1 |
14.87 |
7.53 |
4.17 |
3.57x |
1.81x |
256 |
2 |
27.42 |
9.69 |
7.61 |
3.60x |
1.27x |
256 |
4 |
52.48 |
18.04 |
14.71 |
3.57x |
1.23x |
288 |
1 |
17.21 |
8.69 |
5.12 |
3.36x |
1.70x |
288 |
2 |
32.31 |
12.12 |
9.76 |
3.31x |
1.24x |
288 |
4 |
62.39 |
22.79 |
18.63 |
3.35x |
1.22x |
320 |
1 |
20.40 |
10.36 |
6.04 |
3.38x |
1.72x |
320 |
2 |
38.61 |
14.11 |
11.47 |
3.37x |
1.23x |
320 |
4 |
75.90 |
27.19 |
22.31 |
3.40x |
1.22x |
352 |
1 |
25.21 |
12.49 |
6.91 |
3.65x |
1.81x |
352 |
2 |
48.20 |
16.61 |
13.24 |
3.64x |
1.25x |
352 |
4 |
94.67 |
31.67 |
26.34 |
3.59x |
1.20x |
384 |
1 |
29.69 |
14.44 |
8.00 |
3.71x |
1.81x |
384 |
2 |
56.99 |
19.20 |
15.32 |
3.72x |
1.25x |
384 |
4 |
112.61 |
36.72 |
30.72 |
3.67x |
1.20x |
448 |
1 |
41.48 |
20.53 |
10.64 |
3.90x |
1.93x |
448 |
2 |
81.03 |
25.39 |
20.65 |
3.92x |
1.23x |
448 |
4 |
160.89 |
49.59 |
40.79 |
3.94x |
1.22x |
512 |
1 |
58.76 |
31.01 |
13.34 |
4.41x |
2.33x |
512 |
2 |
116.25 |
32.01 |
26.01 |
4.47x |
1.23x |
512 |
4 |
231.44 |
63.17 |
52.27 |
4.43x |
1.21x |
640 |
1 |
99.39 |
56.56 |
20.50 |
4.85x |
2.76x |
640 |
2 |
198.37 |
49.26 |
40.74 |
4.87x |
1.21x |
640 |
4 |
396.50 |
97.48 |
80.69 |
4.91x |
1.21x |
768 |
1 |
147.68 |
84.16 |
29.08 |
5.08x |
2.89x |
768 |
2 |
294.72 |
70.07 |
57.69 |
5.11x |
1.21x |
768 |
4 |
589.56 |
138.93 |
115.12 |
5.12x |
1.21x |
896 |
1 |
200.99 |
114.39 |
39.60 |
5.08x |
2.89x |
896 |
2 |
402.19 |
95.22 |
78.92 |
5.10x |
1.21x |
896 |
4 |
805.30 |
189.86 |
157.50 |
5.11x |
1.21x |
1024 |
1 |
262.70 |
149.65 |
51.28 |
5.12x |
2.92x |
1024 |
2 |
525.71 |
124.05 |
102.48 |
5.13x |
1.21x |
1024 |
4 |
1055.63 |
247.65 |
205.40 |
5.14x |
1.21x |
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
5.10 |
2.33 |
0.93 |
5.49x |
2.51x |
128 |
2 |
5.42 |
2.56 |
1.18 |
4.59x |
2.17x |
128 |
4 |
5.80 |
2.95 |
1.93 |
3.00x |
1.53x |
160 |
1 |
5.22 |
2.44 |
0.99 |
5.27x |
2.47x |
160 |
2 |
5.41 |
2.65 |
1.48 |
3.65x |
1.79x |
160 |
4 |
8.11 |
3.11 |
2.51 |
3.23x |
1.24x |
192 |
1 |
5.26 |
2.47 |
1.17 |
4.50x |
2.12x |
192 |
2 |
6.03 |
2.74 |
1.85 |
3.26x |
1.48x |
192 |
4 |
7.31 |
3.85 |
3.14 |
2.33x |
1.23x |
224 |
1 |
5.32 |
2.52 |
1.35 |
3.94x |
1.87x |
224 |
2 |
7.71 |
2.81 |
2.18 |
3.53x |
1.29x |
224 |
4 |
7.47 |
4.10 |
3.92 |
1.91x |
1.05x |
256 |
1 |
5.36 |
2.57 |
1.55 |
3.45x |
1.66x |
256 |
2 |
7.01 |
3.56 |
2.57 |
2.72x |
1.38x |
256 |
4 |
8.94 |
4.92 |
4.77 |
1.87x |
1.03x |
288 |
1 |
6.46 |
2.93 |
1.80 |
3.59x |
1.63x |
288 |
2 |
7.08 |
3.69 |
3.07 |
2.30x |
1.20x |
288 |
4 |
10.88 |
5.91 |
5.80 |
1.88x |
1.02x |
320 |
1 |
7.63 |
2.96 |
2.01 |
3.79x |
1.47x |
320 |
2 |
7.13 |
3.79 |
3.55 |
2.01x |
1.07x |
320 |
4 |
12.96 |
6.84 |
6.81 |
1.90x |
1.00x |
352 |
1 |
8.91 |
3.38 |
2.29 |
3.88x |
1.47x |
352 |
2 |
8.12 |
4.42 |
4.12 |
1.97x |
1.07x |
352 |
4 |
15.13 |
7.93 |
7.99 |
1.89x |
0.99x |
384 |
1 |
6.77 |
3.27 |
2.58 |
2.63x |
1.27x |
384 |
2 |
9.33 |
4.85 |
4.79 |
1.95x |
1.01x |
384 |
4 |
17.63 |
9.11 |
9.16 |
1.92x |
0.99x |
448 |
1 |
6.84 |
3.54 |
3.22 |
2.12x |
1.10x |
448 |
2 |
12.15 |
6.19 |
6.36 |
1.91x |
0.97x |
448 |
4 |
23.40 |
11.88 |
11.94 |
1.96x |
1.00x |
512 |
1 |
8.10 |
4.16 |
4.03 |
2.01x |
1.03x |
512 |
2 |
15.26 |
7.72 |
7.67 |
1.99x |
1.01x |
512 |
4 |
29.54 |
14.92 |
15.04 |
1.96x |
0.99x |
640 |
1 |
11.91 |
6.04 |
5.97 |
1.99x |
1.01x |
640 |
2 |
22.98 |
11.50 |
11.54 |
1.99x |
1.00x |
640 |
4 |
45.33 |
22.50 |
22.75 |
1.99x |
0.99x |
768 |
1 |
16.41 |
8.15 |
8.31 |
1.97x |
0.98x |
768 |
2 |
32.05 |
15.74 |
15.89 |
2.02x |
0.99x |
768 |
4 |
63.90 |
31.16 |
31.77 |
2.01x |
0.98x |
896 |
1 |
21.91 |
11.07 |
11.02 |
1.99x |
1.00x |
896 |
2 |
43.37 |
21.43 |
21.65 |
2.00x |
0.99x |
896 |
4 |
86.71 |
42.62 |
43.32 |
2.00x |
0.98x |
1024 |
1 |
27.89 |
13.66 |
13.85 |
2.01x |
0.99x |
1024 |
2 |
56.12 |
27.10 |
27.12 |
2.07x |
1.00x |
1024 |
4 |
112.14 |
54.03 |
54.62 |
2.05x |
0.99x |
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
9.08 |
3.90 |
1.10 |
8.23x |
3.53x |
128 |
2 |
5.38 |
4.33 |
1.11 |
4.85x |
3.91x |
128 |
4 |
5.76 |
4.96 |
1.70 |
3.40x |
2.93x |
160 |
1 |
5.31 |
4.13 |
0.89 |
5.99x |
4.66x |
160 |
2 |
5.47 |
4.48 |
1.31 |
4.18x |
3.42x |
160 |
4 |
7.76 |
5.33 |
2.26 |
3.43x |
2.36x |
192 |
1 |
5.31 |
4.32 |
1.16 |
4.59x |
3.73x |
192 |
2 |
7.65 |
4.78 |
2.20 |
3.47x |
2.17x |
192 |
4 |
12.52 |
6.55 |
3.74 |
3.34x |
1.75x |
224 |
1 |
9.14 |
4.36 |
1.61 |
5.68x |
2.71x |
224 |
2 |
9.76 |
4.89 |
2.60 |
3.76x |
1.88x |
224 |
4 |
12.79 |
6.99 |
4.60 |
2.78x |
1.52x |
256 |
1 |
9.19 |
4.38 |
1.81 |
5.07x |
2.41x |
256 |
2 |
11.92 |
5.96 |
3.04 |
3.92x |
1.96x |
256 |
4 |
13.02 |
7.28 |
4.23 |
3.08x |
1.72x |
288 |
1 |
9.32 |
4.45 |
1.67 |
5.59x |
2.67x |
288 |
2 |
12.14 |
6.19 |
2.76 |
4.39x |
2.24x |
288 |
4 |
13.34 |
7.59 |
5.15 |
2.59x |
1.47x |
320 |
1 |
9.35 |
4.64 |
1.85 |
5.06x |
2.51x |
320 |
2 |
12.26 |
6.34 |
4.00 |
3.07x |
1.59x |
320 |
4 |
13.84 |
8.07 |
7.61 |
1.82x |
1.06x |
352 |
1 |
9.49 |
4.57 |
2.66 |
3.57x |
1.72x |
352 |
2 |
12.31 |
6.51 |
4.61 |
2.67x |
1.41x |
352 |
4 |
16.20 |
9.26 |
8.71 |
1.86x |
1.06x |
384 |
1 |
11.67 |
6.19 |
2.87 |
4.07x |
2.16x |
384 |
2 |
12.35 |
6.68 |
5.33 |
2.32x |
1.25x |
384 |
4 |
18.65 |
8.75 |
10.03 |
1.86x |
0.87x |
448 |
1 |
11.76 |
3.51 |
3.57 |
3.30x |
0.99x |
448 |
2 |
12.83 |
6.42 |
6.59 |
1.95x |
0.97x |
448 |
4 |
24.37 |
11.13 |
12.70 |
1.92x |
0.88x |
512 |
1 |
11.91 |
5.83 |
4.29 |
2.77x |
1.36x |
512 |
2 |
15.82 |
8.56 |
8.15 |
1.94x |
1.05x |
512 |
4 |
30.52 |
14.99 |
15.98 |
1.91x |
0.94x |
640 |
1 |
12.30 |
6.62 |
8.13 |
1.51x |
0.81x |
640 |
2 |
23.45 |
10.72 |
11.98 |
1.96x |
0.89x |
640 |
4 |
46.00 |
21.02 |
20.68 |
2.22x |
1.02x |
768 |
1 |
16.69 |
7.60 |
7.33 |
2.28x |
1.04x |
768 |
2 |
32.49 |
16.84 |
14.40 |
2.26x |
1.17x |
768 |
4 |
62.31 |
33.02 |
30.41 |
2.05x |
1.09x |
896 |
1 |
22.11 |
11.41 |
9.93 |
2.23x |
1.15x |
896 |
2 |
43.57 |
22.38 |
19.61 |
2.22x |
1.14x |
896 |
4 |
86.16 |
45.55 |
39.16 |
2.20x |
1.16x |
1024 |
1 |
28.02 |
14.18 |
12.54 |
2.24x |
1.13x |
1024 |
2 |
55.42 |
27.61 |
26.00 |
2.13x |
1.06x |
1024 |
4 |
111.25 |
56.06 |
50.61 |
2.20x |
1.11x |
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
9.84 |
5.44 |
1.16 |
8.50x |
4.70x |
128 |
2 |
10.14 |
5.56 |
1.35 |
7.52x |
4.12x |
128 |
4 |
10.35 |
6.97 |
2.10 |
4.94x |
3.32x |
160 |
1 |
10.14 |
6.13 |
1.08 |
9.35x |
5.65x |
160 |
2 |
10.31 |
6.61 |
1.59 |
6.46x |
4.14x |
160 |
4 |
10.82 |
7.68 |
2.59 |
4.18x |
2.97x |
192 |
1 |
10.27 |
6.26 |
1.30 |
7.92x |
4.83x |
192 |
2 |
10.21 |
5.79 |
1.95 |
5.25x |
2.98x |
192 |
4 |
12.98 |
8.09 |
3.21 |
4.05x |
2.52x |
224 |
1 |
10.20 |
6.56 |
1.51 |
6.77x |
4.36x |
224 |
2 |
10.65 |
6.85 |
2.33 |
4.58x |
2.94x |
224 |
4 |
12.69 |
8.43 |
3.81 |
3.33x |
2.21x |
256 |
1 |
10.09 |
5.96 |
1.69 |
5.95x |
3.52x |
256 |
2 |
11.97 |
7.99 |
2.68 |
4.47x |
2.98x |
256 |
4 |
12.79 |
8.75 |
4.56 |
2.80x |
1.92x |
288 |
1 |
10.06 |
6.24 |
1.89 |
5.33x |
3.30x |
288 |
2 |
12.29 |
7.21 |
3.03 |
4.05x |
2.38x |
288 |
4 |
12.85 |
9.02 |
5.49 |
2.34x |
1.64x |
320 |
1 |
10.26 |
6.56 |
2.10 |
4.90x |
3.13x |
320 |
2 |
12.58 |
7.69 |
3.48 |
3.61x |
2.21x |
320 |
4 |
13.86 |
10.08 |
6.47 |
2.14x |
1.56x |
352 |
1 |
10.36 |
6.27 |
2.44 |
4.25x |
2.57x |
352 |
2 |
12.39 |
8.27 |
3.99 |
3.11x |
2.07x |
352 |
4 |
14.67 |
9.21 |
7.60 |
1.93x |
1.21x |
384 |
1 |
12.51 |
7.90 |
2.73 |
4.59x |
2.90x |
384 |
2 |
12.99 |
8.59 |
4.56 |
2.85x |
1.89x |
384 |
4 |
16.80 |
10.38 |
8.60 |
1.95x |
1.21x |
448 |
1 |
11.93 |
7.18 |
3.12 |
3.82x |
2.30x |
448 |
2 |
12.78 |
7.92 |
5.69 |
2.24x |
1.39x |
448 |
4 |
21.97 |
12.62 |
11.01 |
2.00x |
1.15x |
512 |
1 |
12.04 |
8.04 |
3.79 |
3.18x |
2.12x |
512 |
2 |
14.53 |
9.18 |
7.13 |
2.04x |
1.29x |
512 |
4 |
28.12 |
15.09 |
13.73 |
2.05x |
1.10x |
640 |
1 |
12.62 |
7.53 |
9.30 |
1.36x |
0.81x |
640 |
2 |
21.73 |
11.43 |
10.64 |
2.04x |
1.07x |
640 |
4 |
42.19 |
21.79 |
20.70 |
2.04x |
1.05x |
768 |
1 |
15.51 |
8.64 |
9.05 |
1.71x |
0.95x |
768 |
2 |
30.07 |
15.66 |
14.78 |
2.03x |
1.06x |
768 |
4 |
59.00 |
29.94 |
28.91 |
2.04x |
1.04x |
896 |
1 |
20.65 |
10.99 |
10.15 |
2.04x |
1.08x |
896 |
2 |
40.62 |
20.28 |
19.82 |
2.05x |
1.02x |
896 |
4 |
79.96 |
39.62 |
39.03 |
2.05x |
1.02x |
1024 |
1 |
26.13 |
13.00 |
12.61 |
2.07x |
1.03x |
1024 |
2 |
51.82 |
25.27 |
25.09 |
2.07x |
1.01x |
1024 |
4 |
102.82 |
51.02 |
49.89 |
2.06x |
1.02x |
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
11.04 |
4.25 |
1.12 |
9.85x |
3.79x |
128 |
2 |
11.79 |
4.67 |
1.31 |
8.99x |
3.56x |
128 |
4 |
12.35 |
5.20 |
1.96 |
6.29x |
2.65x |
160 |
1 |
11.72 |
4.70 |
1.06 |
11.04x |
4.43x |
160 |
2 |
12.05 |
5.58 |
1.56 |
7.73x |
3.58x |
160 |
4 |
12.14 |
5.79 |
2.65 |
4.58x |
2.19x |
192 |
1 |
11.50 |
5.41 |
1.33 |
8.67x |
4.07x |
192 |
2 |
12.33 |
5.01 |
1.94 |
6.36x |
2.59x |
192 |
4 |
15.55 |
6.99 |
3.15 |
4.93x |
2.21x |
224 |
1 |
11.22 |
4.91 |
1.46 |
7.66x |
3.35x |
224 |
2 |
12.62 |
5.45 |
2.28 |
5.54x |
2.39x |
224 |
4 |
14.87 |
6.64 |
3.86 |
3.85x |
1.72x |
256 |
1 |
10.70 |
4.93 |
1.78 |
6.03x |
2.77x |
256 |
2 |
13.65 |
5.75 |
2.58 |
5.28x |
2.22x |
256 |
4 |
14.61 |
7.59 |
4.52 |
3.23x |
1.68x |
288 |
1 |
12.53 |
5.68 |
1.89 |
6.65x |
3.01x |
288 |
2 |
15.02 |
7.01 |
3.01 |
4.98x |
2.33x |
288 |
4 |
15.16 |
7.64 |
5.43 |
2.79x |
1.41x |
320 |
1 |
13.00 |
5.04 |
2.11 |
6.15x |
2.38x |
320 |
2 |
14.81 |
6.45 |
3.45 |
4.29x |
1.87x |
320 |
4 |
16.28 |
7.77 |
6.32 |
2.57x |
1.23x |
352 |
1 |
12.45 |
5.64 |
2.45 |
5.09x |
2.31x |
352 |
2 |
15.00 |
6.64 |
3.99 |
3.76x |
1.66x |
352 |
4 |
16.70 |
8.49 |
7.25 |
2.30x |
1.17x |
384 |
1 |
13.61 |
6.05 |
2.59 |
5.26x |
2.34x |
384 |
2 |
14.67 |
7.02 |
4.44 |
3.30x |
1.58x |
384 |
4 |
17.53 |
9.09 |
8.28 |
2.12x |
1.10x |
448 |
1 |
13.81 |
6.39 |
3.09 |
4.46x |
2.07x |
448 |
2 |
14.12 |
6.94 |
5.58 |
2.53x |
1.24x |
448 |
4 |
22.30 |
11.18 |
10.45 |
2.13x |
1.07x |
512 |
1 |
13.69 |
7.95 |
3.69 |
3.71x |
2.16x |
512 |
2 |
14.95 |
7.57 |
6.92 |
2.16x |
1.10x |
512 |
4 |
28.24 |
13.77 |
13.04 |
2.17x |
1.06x |
640 |
1 |
13.43 |
6.07 |
8.81 |
1.52x |
0.69x |
640 |
2 |
21.87 |
11.61 |
10.25 |
2.13x |
1.13x |
640 |
4 |
42.51 |
20.56 |
19.36 |
2.20x |
1.06x |
768 |
1 |
15.73 |
7.55 |
8.86 |
1.77x |
0.85x |
768 |
2 |
30.30 |
14.16 |
13.56 |
2.23x |
1.04x |
768 |
4 |
59.17 |
29.05 |
26.46 |
2.24x |
1.10x |
896 |
1 |
21.00 |
10.49 |
9.51 |
2.21x |
1.10x |
896 |
2 |
40.63 |
19.03 |
18.31 |
2.22x |
1.04x |
896 |
4 |
80.21 |
37.40 |
35.96 |
2.23x |
1.04x |
1024 |
1 |
26.40 |
12.16 |
11.64 |
2.27x |
1.05x |
1024 |
2 |
52.08 |
23.69 |
22.55 |
2.31x |
1.05x |
1024 |
4 |
103.28 |
45.49 |
45.10 |
2.29x |
1.01x |
This SKU’s benchmark sweeps batch 1 only.
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
11.08 |
6.38 |
6.23 |
1.78x |
1.03x |
160 |
1 |
18.70 |
10.21 |
10.69 |
1.75x |
0.95x |
192 |
1 |
27.06 |
14.23 |
15.53 |
1.74x |
0.92x |
224 |
1 |
36.49 |
19.29 |
21.21 |
1.72x |
0.91x |
256 |
1 |
47.35 |
25.01 |
27.58 |
1.72x |
0.91x |
288 |
1 |
60.26 |
31.65 |
35.10 |
1.72x |
0.90x |
320 |
1 |
74.05 |
39.65 |
44.15 |
1.68x |
0.90x |
352 |
1 |
89.95 |
47.84 |
53.98 |
1.67x |
0.89x |
384 |
1 |
107.38 |
56.81 |
64.74 |
1.66x |
0.88x |
448 |
1 |
165.16 |
100.91 |
116.68 |
1.42x |
0.86x |
512 |
1 |
272.33 |
192.18 |
195.32 |
1.39x |
0.98x |
640 |
1 |
440.08 |
296.56 |
332.75 |
1.32x |
0.89x |
768 |
1 |
618.50 |
414.79 |
465.30 |
1.33x |
0.89x |
896 |
1 |
850.69 |
572.41 |
649.30 |
1.31x |
0.88x |
1024 |
1 |
1131.61 |
774.21 |
833.86 |
1.36x |
0.93x |
This SKU’s benchmark sweeps batch 1 only.
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
6.55 |
4.88 |
2.08 |
3.16x |
2.35x |
160 |
1 |
6.57 |
4.95 |
2.56 |
2.56x |
1.93x |
192 |
1 |
8.77 |
5.24 |
3.38 |
2.60x |
1.55x |
224 |
1 |
11.79 |
7.14 |
4.08 |
2.89x |
1.75x |
256 |
1 |
15.58 |
8.70 |
4.94 |
3.15x |
1.76x |
288 |
1 |
18.44 |
10.13 |
6.11 |
3.02x |
1.66x |
320 |
1 |
22.11 |
11.89 |
7.37 |
3.00x |
1.61x |
352 |
1 |
27.21 |
14.71 |
8.39 |
3.24x |
1.75x |
384 |
1 |
32.37 |
17.15 |
9.74 |
3.32x |
1.76x |
448 |
1 |
45.69 |
24.62 |
13.00 |
3.52x |
1.89x |
512 |
1 |
65.67 |
36.29 |
16.44 |
4.00x |
2.21x |
640 |
1 |
114.42 |
68.76 |
25.49 |
4.49x |
2.70x |
768 |
1 |
169.72 |
102.43 |
36.24 |
4.68x |
2.83x |
896 |
1 |
231.09 |
139.09 |
49.20 |
4.70x |
2.83x |
1024 |
1 |
302.18 |
181.64 |
64.80 |
4.66x |
2.80x |
This SKU’s benchmark sweeps batch 1 only.
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
10.42 |
4.33 |
1.92 |
5.42x |
2.25x |
160 |
1 |
10.50 |
4.44 |
2.36 |
4.45x |
1.88x |
192 |
1 |
10.51 |
4.79 |
2.93 |
3.58x |
1.63x |
224 |
1 |
11.92 |
6.30 |
3.60 |
3.31x |
1.75x |
256 |
1 |
15.57 |
8.04 |
4.34 |
3.59x |
1.85x |
288 |
1 |
18.11 |
9.23 |
5.28 |
3.43x |
1.75x |
320 |
1 |
21.52 |
10.86 |
6.22 |
3.46x |
1.75x |
352 |
1 |
26.54 |
13.45 |
7.11 |
3.73x |
1.89x |
384 |
1 |
31.37 |
15.44 |
8.24 |
3.81x |
1.87x |
448 |
1 |
44.16 |
22.24 |
10.89 |
4.06x |
2.04x |
512 |
1 |
63.31 |
33.73 |
13.66 |
4.63x |
2.47x |
640 |
1 |
111.98 |
67.36 |
21.11 |
5.31x |
3.19x |
768 |
1 |
166.56 |
100.33 |
29.86 |
5.58x |
3.36x |
896 |
1 |
226.38 |
136.43 |
40.47 |
5.59x |
3.37x |
1024 |
1 |
295.86 |
178.09 |
52.76 |
5.61x |
3.38x |
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
8.69 |
3.77 |
1.76 |
4.95x |
2.15x |
128 |
2 |
8.93 |
3.84 |
2.76 |
3.23x |
1.39x |
128 |
4 |
13.48 |
6.25 |
4.88 |
2.76x |
1.28x |
160 |
1 |
9.02 |
3.83 |
2.21 |
4.07x |
1.73x |
160 |
2 |
10.90 |
5.10 |
3.82 |
2.85x |
1.33x |
160 |
4 |
19.96 |
8.73 |
7.20 |
2.77x |
1.21x |
192 |
1 |
8.86 |
4.54 |
2.85 |
3.11x |
1.59x |
192 |
2 |
15.04 |
6.44 |
5.04 |
2.99x |
1.28x |
192 |
4 |
28.39 |
11.59 |
9.45 |
3.00x |
1.23x |
224 |
1 |
11.35 |
6.02 |
3.52 |
3.22x |
1.71x |
224 |
2 |
20.73 |
8.14 |
6.46 |
3.21x |
1.26x |
224 |
4 |
39.69 |
14.98 |
12.21 |
3.25x |
1.23x |
256 |
1 |
14.92 |
7.64 |
4.25 |
3.51x |
1.80x |
256 |
2 |
27.87 |
10.05 |
7.95 |
3.51x |
1.27x |
256 |
4 |
54.16 |
18.80 |
15.39 |
3.52x |
1.22x |
288 |
1 |
17.55 |
9.03 |
5.33 |
3.29x |
1.69x |
288 |
2 |
33.28 |
12.76 |
10.18 |
3.27x |
1.25x |
288 |
4 |
64.88 |
23.70 |
19.64 |
3.30x |
1.21x |
320 |
1 |
20.89 |
10.48 |
6.31 |
3.31x |
1.66x |
320 |
2 |
40.17 |
14.86 |
12.09 |
3.32x |
1.23x |
320 |
4 |
78.76 |
28.51 |
23.73 |
3.32x |
1.20x |
352 |
1 |
25.73 |
13.19 |
7.27 |
3.54x |
1.81x |
352 |
2 |
49.98 |
17.49 |
14.26 |
3.50x |
1.23x |
352 |
4 |
98.47 |
33.47 |
27.64 |
3.56x |
1.21x |
384 |
1 |
30.41 |
15.09 |
8.42 |
3.61x |
1.79x |
384 |
2 |
59.27 |
20.12 |
16.43 |
3.61x |
1.23x |
384 |
4 |
117.63 |
39.04 |
32.12 |
3.66x |
1.22x |
448 |
1 |
42.84 |
21.84 |
11.28 |
3.80x |
1.94x |
448 |
2 |
84.39 |
26.93 |
22.09 |
3.82x |
1.22x |
448 |
4 |
168.09 |
52.98 |
43.73 |
3.84x |
1.21x |
512 |
1 |
60.71 |
32.09 |
14.28 |
4.25x |
2.25x |
512 |
2 |
120.97 |
34.24 |
28.07 |
4.31x |
1.22x |
512 |
4 |
241.06 |
67.74 |
55.62 |
4.33x |
1.22x |
640 |
1 |
103.00 |
58.53 |
22.12 |
4.66x |
2.65x |
640 |
2 |
205.48 |
52.94 |
43.69 |
4.70x |
1.21x |
640 |
4 |
410.53 |
104.66 |
86.81 |
4.73x |
1.21x |
768 |
1 |
152.80 |
86.93 |
31.47 |
4.85x |
2.76x |
768 |
2 |
304.74 |
75.43 |
62.38 |
4.88x |
1.21x |
768 |
4 |
611.95 |
150.51 |
124.35 |
4.92x |
1.21x |
896 |
1 |
208.04 |
118.37 |
42.66 |
4.88x |
2.77x |
896 |
2 |
415.65 |
102.95 |
84.90 |
4.90x |
1.21x |
896 |
4 |
838.80 |
204.82 |
170.07 |
4.93x |
1.20x |
1024 |
1 |
271.61 |
154.49 |
55.85 |
4.86x |
2.77x |
1024 |
2 |
545.38 |
133.94 |
111.49 |
4.89x |
1.20x |
1024 |
4 |
1101.63 |
267.61 |
222.79 |
4.94x |
1.20x |
This SKU’s benchmark sweeps batch 1 only.
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
6.58 |
2.91 |
1.42 |
4.63x |
2.05x |
160 |
1 |
6.67 |
3.51 |
1.81 |
3.67x |
1.93x |
192 |
1 |
7.42 |
4.70 |
2.49 |
2.98x |
1.89x |
224 |
1 |
10.62 |
6.35 |
3.36 |
3.16x |
1.89x |
256 |
1 |
15.49 |
8.28 |
4.42 |
3.51x |
1.88x |
288 |
1 |
20.30 |
10.53 |
5.79 |
3.50x |
1.82x |
320 |
1 |
25.75 |
13.36 |
7.16 |
3.60x |
1.87x |
352 |
1 |
31.95 |
16.81 |
10.02 |
3.19x |
1.68x |
384 |
1 |
38.58 |
20.22 |
12.92 |
2.99x |
1.56x |
448 |
1 |
53.22 |
27.86 |
14.15 |
3.76x |
1.97x |
512 |
1 |
71.50 |
37.42 |
18.29 |
3.91x |
2.05x |
640 |
1 |
110.99 |
58.05 |
28.17 |
3.94x |
2.06x |
768 |
1 |
163.30 |
85.47 |
41.57 |
3.93x |
2.06x |
896 |
1 |
272.86 |
171.07 |
57.52 |
4.74x |
2.97x |
1024 |
1 |
449.06 |
307.58 |
76.08 |
5.90x |
4.04x |
This SKU’s benchmark sweeps batch 1 only.
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
8.04 |
3.33 |
1.39 |
5.79x |
2.39x |
160 |
1 |
8.00 |
3.54 |
1.95 |
4.09x |
1.81x |
192 |
1 |
8.11 |
4.65 |
2.62 |
3.10x |
1.77x |
224 |
1 |
10.20 |
6.49 |
3.63 |
2.81x |
1.79x |
256 |
1 |
14.30 |
8.32 |
4.86 |
2.95x |
1.71x |
288 |
1 |
18.64 |
10.78 |
6.36 |
2.93x |
1.70x |
320 |
1 |
23.18 |
13.07 |
8.05 |
2.88x |
1.63x |
352 |
1 |
28.56 |
16.56 |
10.59 |
2.70x |
1.56x |
384 |
1 |
34.18 |
19.73 |
12.82 |
2.67x |
1.54x |
448 |
1 |
47.39 |
27.22 |
17.33 |
2.73x |
1.57x |
512 |
1 |
63.69 |
36.58 |
22.72 |
2.80x |
1.61x |
640 |
1 |
99.74 |
57.80 |
36.52 |
2.73x |
1.58x |
768 |
1 |
148.09 |
86.48 |
53.80 |
2.75x |
1.61x |
896 |
1 |
246.66 |
160.44 |
74.02 |
3.33x |
2.17x |
1024 |
1 |
407.31 |
292.16 |
97.86 |
4.16x |
2.99x |
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
6.89 |
2.82 |
1.23 |
5.60x |
2.29x |
128 |
2 |
7.06 |
3.07 |
1.72 |
4.10x |
1.78x |
128 |
4 |
10.29 |
4.54 |
2.86 |
3.60x |
1.59x |
160 |
1 |
7.00 |
3.05 |
1.30 |
5.39x |
2.34x |
160 |
2 |
8.28 |
3.67 |
2.22 |
3.73x |
1.66x |
160 |
4 |
8.98 |
6.00 |
3.99 |
2.25x |
1.50x |
192 |
1 |
7.06 |
3.14 |
1.62 |
4.35x |
1.93x |
192 |
2 |
11.08 |
4.60 |
2.88 |
3.85x |
1.60x |
192 |
4 |
11.07 |
6.97 |
5.21 |
2.13x |
1.34x |
224 |
1 |
7.94 |
3.46 |
2.03 |
3.92x |
1.71x |
224 |
2 |
8.47 |
5.14 |
3.50 |
2.42x |
1.47x |
224 |
4 |
14.27 |
8.98 |
6.46 |
2.21x |
1.39x |
256 |
1 |
7.96 |
3.57 |
2.41 |
3.31x |
1.48x |
256 |
2 |
9.62 |
6.03 |
4.24 |
2.27x |
1.42x |
256 |
4 |
18.11 |
11.30 |
8.06 |
2.25x |
1.40x |
288 |
1 |
8.09 |
4.34 |
2.88 |
2.81x |
1.51x |
288 |
2 |
11.80 |
7.43 |
5.34 |
2.21x |
1.39x |
288 |
4 |
22.43 |
13.71 |
10.05 |
2.23x |
1.36x |
320 |
1 |
8.39 |
5.02 |
3.42 |
2.45x |
1.47x |
320 |
2 |
14.16 |
8.59 |
6.31 |
2.24x |
1.36x |
320 |
4 |
27.48 |
16.56 |
12.02 |
2.29x |
1.38x |
352 |
1 |
9.81 |
5.84 |
3.92 |
2.50x |
1.49x |
352 |
2 |
16.77 |
10.36 |
7.35 |
2.28x |
1.41x |
352 |
4 |
32.38 |
19.55 |
14.29 |
2.27x |
1.37x |
384 |
1 |
10.37 |
6.37 |
4.54 |
2.29x |
1.40x |
384 |
2 |
19.54 |
11.72 |
8.62 |
2.27x |
1.36x |
384 |
4 |
37.90 |
22.82 |
16.70 |
2.27x |
1.37x |
448 |
1 |
13.50 |
8.15 |
5.83 |
2.31x |
1.40x |
448 |
2 |
25.89 |
15.50 |
11.23 |
2.31x |
1.38x |
448 |
4 |
50.71 |
29.99 |
22.01 |
2.30x |
1.36x |
512 |
1 |
17.11 |
10.34 |
7.42 |
2.31x |
1.39x |
512 |
2 |
33.26 |
19.78 |
14.20 |
2.34x |
1.39x |
512 |
4 |
65.47 |
38.62 |
28.02 |
2.34x |
1.38x |
640 |
1 |
25.80 |
15.24 |
11.27 |
2.29x |
1.35x |
640 |
2 |
50.77 |
29.69 |
21.81 |
2.33x |
1.36x |
640 |
4 |
100.71 |
58.66 |
43.06 |
2.34x |
1.36x |
768 |
1 |
36.41 |
21.62 |
15.44 |
2.36x |
1.40x |
768 |
2 |
72.20 |
41.99 |
30.09 |
2.40x |
1.40x |
768 |
4 |
143.25 |
83.43 |
61.09 |
2.35x |
1.37x |
896 |
1 |
49.10 |
28.61 |
21.23 |
2.31x |
1.35x |
896 |
2 |
98.27 |
56.92 |
42.12 |
2.33x |
1.35x |
896 |
4 |
195.63 |
112.82 |
84.84 |
2.31x |
1.33x |
1024 |
1 |
63.56 |
36.92 |
27.02 |
2.35x |
1.37x |
1024 |
2 |
127.40 |
73.16 |
53.87 |
2.36x |
1.36x |
1024 |
4 |
255.10 |
146.18 |
108.00 |
2.36x |
1.35x |
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
6.74 |
2.78 |
1.06 |
6.33x |
2.61x |
128 |
2 |
7.06 |
2.94 |
1.43 |
4.92x |
2.05x |
128 |
4 |
8.31 |
3.48 |
2.50 |
3.32x |
1.39x |
160 |
1 |
6.76 |
2.89 |
1.22 |
5.56x |
2.38x |
160 |
2 |
7.41 |
3.11 |
2.25 |
3.29x |
1.38x |
160 |
4 |
8.42 |
4.46 |
3.53 |
2.38x |
1.26x |
192 |
1 |
7.04 |
2.96 |
1.49 |
4.74x |
1.99x |
192 |
2 |
8.81 |
3.50 |
2.50 |
3.53x |
1.40x |
192 |
4 |
8.89 |
5.60 |
4.45 |
2.00x |
1.26x |
224 |
1 |
7.00 |
3.10 |
1.72 |
4.06x |
1.80x |
224 |
2 |
11.47 |
4.44 |
3.03 |
3.79x |
1.47x |
224 |
4 |
11.46 |
7.14 |
5.58 |
2.05x |
1.28x |
256 |
1 |
7.70 |
3.26 |
2.04 |
3.77x |
1.60x |
256 |
2 |
8.27 |
4.76 |
3.68 |
2.25x |
1.30x |
256 |
4 |
14.40 |
8.86 |
6.91 |
2.08x |
1.28x |
288 |
1 |
7.85 |
3.44 |
2.49 |
3.16x |
1.38x |
288 |
2 |
9.41 |
5.93 |
4.58 |
2.05x |
1.30x |
288 |
4 |
18.01 |
10.93 |
8.66 |
2.08x |
1.26x |
320 |
1 |
7.85 |
3.90 |
2.99 |
2.63x |
1.31x |
320 |
2 |
11.29 |
6.86 |
5.43 |
2.08x |
1.26x |
320 |
4 |
21.79 |
13.16 |
10.39 |
2.10x |
1.27x |
352 |
1 |
7.87 |
4.61 |
3.38 |
2.33x |
1.36x |
352 |
2 |
13.32 |
8.19 |
6.20 |
2.15x |
1.32x |
352 |
4 |
25.97 |
15.43 |
12.12 |
2.14x |
1.27x |
384 |
1 |
8.22 |
4.96 |
3.78 |
2.17x |
1.31x |
384 |
2 |
15.57 |
9.25 |
7.15 |
2.18x |
1.29x |
384 |
4 |
30.26 |
17.90 |
14.01 |
2.16x |
1.28x |
448 |
1 |
10.72 |
6.49 |
4.93 |
2.17x |
1.32x |
448 |
2 |
20.82 |
12.24 |
9.74 |
2.14x |
1.26x |
448 |
4 |
40.83 |
24.18 |
18.88 |
2.16x |
1.28x |
512 |
1 |
13.62 |
8.07 |
6.12 |
2.22x |
1.32x |
512 |
2 |
26.55 |
15.48 |
12.04 |
2.20x |
1.29x |
512 |
4 |
52.54 |
30.45 |
23.85 |
2.20x |
1.28x |
640 |
1 |
20.70 |
12.13 |
9.51 |
2.18x |
1.28x |
640 |
2 |
40.66 |
23.92 |
18.54 |
2.19x |
1.29x |
640 |
4 |
80.70 |
47.71 |
36.99 |
2.18x |
1.29x |
768 |
1 |
29.19 |
17.05 |
13.15 |
2.22x |
1.30x |
768 |
2 |
57.88 |
34.16 |
26.21 |
2.21x |
1.30x |
768 |
4 |
114.95 |
68.55 |
53.13 |
2.16x |
1.29x |
896 |
1 |
39.69 |
23.34 |
18.01 |
2.20x |
1.30x |
896 |
2 |
79.05 |
46.67 |
35.90 |
2.20x |
1.30x |
896 |
4 |
157.34 |
92.67 |
73.74 |
2.13x |
1.26x |
1024 |
1 |
51.19 |
30.12 |
23.25 |
2.20x |
1.30x |
1024 |
2 |
102.52 |
60.16 |
46.91 |
2.19x |
1.28x |
1024 |
4 |
204.19 |
120.15 |
94.36 |
2.16x |
1.27x |
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
4.92 |
2.86 |
0.96 |
5.13x |
2.98x |
128 |
2 |
5.10 |
2.93 |
1.33 |
3.84x |
2.21x |
128 |
4 |
7.74 |
3.94 |
2.13 |
3.64x |
1.85x |
160 |
1 |
4.97 |
2.94 |
1.27 |
3.91x |
2.31x |
160 |
2 |
6.31 |
2.97 |
1.80 |
3.50x |
1.65x |
160 |
4 |
6.56 |
3.98 |
2.90 |
2.26x |
1.37x |
192 |
1 |
5.22 |
2.99 |
1.36 |
3.85x |
2.20x |
192 |
2 |
8.23 |
3.30 |
2.10 |
3.93x |
1.57x |
192 |
4 |
7.89 |
4.80 |
3.58 |
2.21x |
1.34x |
224 |
1 |
6.05 |
2.93 |
1.50 |
4.04x |
1.96x |
224 |
2 |
10.53 |
3.87 |
2.49 |
4.23x |
1.56x |
224 |
4 |
9.95 |
5.96 |
4.44 |
2.24x |
1.34x |
256 |
1 |
5.98 |
3.21 |
1.73 |
3.45x |
1.85x |
256 |
2 |
6.90 |
4.19 |
2.96 |
2.33x |
1.42x |
256 |
4 |
12.34 |
7.32 |
5.50 |
2.24x |
1.33x |
288 |
1 |
6.16 |
3.35 |
2.05 |
3.00x |
1.63x |
288 |
2 |
8.29 |
4.94 |
3.65 |
2.27x |
1.35x |
288 |
4 |
15.13 |
8.83 |
6.87 |
2.20x |
1.29x |
320 |
1 |
6.12 |
3.50 |
2.44 |
2.51x |
1.43x |
320 |
2 |
9.74 |
5.61 |
4.31 |
2.26x |
1.30x |
320 |
4 |
18.10 |
10.45 |
8.19 |
2.21x |
1.28x |
352 |
1 |
6.77 |
3.94 |
2.72 |
2.49x |
1.45x |
352 |
2 |
11.34 |
6.59 |
4.94 |
2.29x |
1.33x |
352 |
4 |
21.43 |
12.26 |
9.57 |
2.24x |
1.28x |
384 |
1 |
7.23 |
4.20 |
3.02 |
2.39x |
1.39x |
384 |
2 |
13.13 |
7.52 |
5.71 |
2.30x |
1.32x |
384 |
4 |
25.00 |
14.24 |
11.04 |
2.27x |
1.29x |
448 |
1 |
9.24 |
5.34 |
3.95 |
2.34x |
1.35x |
448 |
2 |
17.22 |
9.76 |
7.50 |
2.30x |
1.30x |
448 |
4 |
33.13 |
18.88 |
14.68 |
2.26x |
1.29x |
512 |
1 |
11.50 |
6.52 |
4.82 |
2.39x |
1.35x |
512 |
2 |
21.83 |
12.24 |
9.40 |
2.32x |
1.30x |
512 |
4 |
42.43 |
23.71 |
18.44 |
2.30x |
1.29x |
640 |
1 |
17.07 |
10.26 |
7.55 |
2.26x |
1.36x |
640 |
2 |
33.01 |
18.40 |
14.57 |
2.27x |
1.26x |
640 |
4 |
65.15 |
36.41 |
28.75 |
2.27x |
1.27x |
768 |
1 |
23.84 |
13.49 |
10.19 |
2.34x |
1.32x |
768 |
2 |
46.91 |
26.10 |
20.15 |
2.33x |
1.29x |
768 |
4 |
93.03 |
51.79 |
40.47 |
2.30x |
1.28x |
896 |
1 |
31.95 |
17.93 |
14.04 |
2.28x |
1.28x |
896 |
2 |
64.49 |
36.09 |
27.81 |
2.32x |
1.30x |
896 |
4 |
128.13 |
71.59 |
56.06 |
2.29x |
1.28x |
1024 |
1 |
41.27 |
22.97 |
17.93 |
2.30x |
1.28x |
1024 |
2 |
82.18 |
45.80 |
35.87 |
2.29x |
1.28x |
1024 |
4 |
164.67 |
92.02 |
71.87 |
2.29x |
1.28x |
Tokens |
Batch |
|
|
|
eager÷bioir |
compile÷bioir |
|---|---|---|---|---|---|---|
128 |
1 |
10.37 |
6.55 |
1.13 |
9.18x |
5.81x |
128 |
2 |
10.88 |
6.26 |
1.43 |
7.59x |
4.37x |
128 |
4 |
11.48 |
7.00 |
2.35 |
4.89x |
2.98x |
160 |
1 |
11.12 |
7.01 |
1.29 |
8.64x |
5.44x |
160 |
2 |
12.54 |
6.17 |
1.99 |
6.31x |
3.11x |
160 |
4 |
12.38 |
8.47 |
3.16 |
3.92x |
2.68x |
192 |
1 |
10.86 |
7.12 |
1.51 |
7.19x |
4.71x |
192 |
2 |
11.09 |
7.03 |
2.21 |
5.01x |
3.18x |
192 |
4 |
12.70 |
9.29 |
3.79 |
3.35x |
2.45x |
224 |
1 |
10.52 |
6.64 |
1.57 |
6.71x |
4.24x |
224 |
2 |
11.15 |
7.83 |
2.63 |
4.23x |
2.97x |
224 |
4 |
13.21 |
8.96 |
4.63 |
2.85x |
1.93x |
256 |
1 |
11.30 |
7.02 |
1.83 |
6.17x |
3.84x |
256 |
2 |
12.02 |
7.57 |
3.02 |
3.98x |
2.51x |
256 |
4 |
13.68 |
8.59 |
5.52 |
2.48x |
1.56x |
288 |
1 |
11.65 |
7.42 |
2.18 |
5.35x |
3.41x |
288 |
2 |
12.47 |
8.33 |
3.81 |
3.27x |
2.19x |
288 |
4 |
15.34 |
10.48 |
7.07 |
2.17x |
1.48x |
320 |
1 |
12.25 |
7.64 |
2.51 |
4.89x |
3.05x |
320 |
2 |
12.75 |
7.59 |
4.32 |
2.95x |
1.76x |
320 |
4 |
17.60 |
11.30 |
8.34 |
2.11x |
1.35x |
352 |
1 |
11.94 |
7.16 |
2.87 |
4.16x |
2.49x |
352 |
2 |
13.56 |
8.34 |
5.00 |
2.71x |
1.67x |
352 |
4 |
20.48 |
13.61 |
9.61 |
2.13x |
1.42x |
384 |
1 |
14.44 |
8.04 |
3.11 |
4.65x |
2.59x |
384 |
2 |
14.89 |
9.39 |
5.67 |
2.63x |
1.66x |
384 |
4 |
23.90 |
14.77 |
10.92 |
2.19x |
1.35x |
448 |
1 |
14.41 |
7.27 |
4.00 |
3.60x |
1.82x |
448 |
2 |
16.36 |
10.60 |
7.38 |
2.22x |
1.44x |
448 |
4 |
31.27 |
18.71 |
14.61 |
2.14x |
1.28x |
512 |
1 |
14.54 |
9.78 |
4.85 |
3.00x |
2.02x |
512 |
2 |
20.65 |
13.07 |
9.35 |
2.21x |
1.40x |
512 |
4 |
39.52 |
24.54 |
18.06 |
2.19x |
1.36x |
640 |
1 |
16.22 |
10.27 |
7.60 |
2.13x |
1.35x |
640 |
2 |
31.33 |
19.43 |
14.30 |
2.19x |
1.36x |
640 |
4 |
60.65 |
35.94 |
28.02 |
2.16x |
1.28x |
768 |
1 |
22.41 |
13.71 |
10.39 |
2.16x |
1.32x |
768 |
2 |
43.82 |
25.98 |
19.78 |
2.22x |
1.31x |
768 |
4 |
86.53 |
50.59 |
39.22 |
2.21x |
1.29x |
896 |
1 |
29.90 |
18.10 |
13.83 |
2.16x |
1.31x |
896 |
2 |
59.05 |
34.63 |
26.99 |
2.19x |
1.28x |
896 |
4 |
117.42 |
69.20 |
54.31 |
2.16x |
1.27x |
1024 |
1 |
38.37 |
22.61 |
17.47 |
2.20x |
1.29x |
1024 |
2 |
76.44 |
45.74 |
34.73 |
2.20x |
1.32x |
1024 |
4 |
152.35 |
88.67 |
68.90 |
2.21x |
1.29x |
Three things the grid shows.
Design time is super-linear in token count, because the pair track is
quadratic: on H100 80GB HBM3 with bioir at batch 1, an eightfold rise in
tokens, 128 to 1024, costs 15.3x the time. The spread between GPUs widens with
it. At 128 tokens the datacenter parts are about 2.3x apart; at 1024 they fall
into bands — 11.6 to 13.9 s on the Blackwell datacenter parts, 16.9 to 27.0 s
on Hopper, 51.3 to 64.8 s on Ampere, and 76.1 to 97.9 s on the Ada-generation
parts. GPU choice therefore matters far more for long designs than for short
ones. The GB10 (DGX Spark) sits well outside those bands at 833.9 s, which is
what a desktop part costs on the longest design this suite sends.
Batching is close to free at short lengths and close to linear at long
ones. On H100 at 128 tokens, oss_eager returns four designs in 8.25 s
against 6.82 s for one — 1.21x the time for 4x the output. bioir starts from
a far lower per-request floor and so saturates sooner, at 2.08x. By 1024
tokens the GPU is saturated either way and batch 4 costs about 3.9x on both. A
deployment serving short designs should batch; one serving long designs gains
throughput from batching but little latency.
bioir leads on most parts, by a margin that narrows with both length and
batch. On H100 it is 5.83x over oss_eager and 2.67x over
oss_torch_compile at 128 tokens and batch 1, 3.39x and 1.61x at the same
length and batch 4, and settles near 2.3x and 1.38x from 512 tokens upward
regardless of batch. The short-length, batch-1 figures are the most
flattering ones and should not be quoted as the general case.
Accuracy Across Execution Backends#
The eval suite’s structural-quality metrics, on one H100 80GB HBM3, from the
same release candidate as the performance table above. Each backend ran the
same 87 conditioned designs. n is how many of those 87 the metric is defined
on.
Metric |
Better |
|
|
|
|
|---|---|---|---|---|---|
|
higher |
87 |
1.000 |
1.000 |
1.000 |
|
lower |
87 |
0.0134 |
0.0105 |
0.0106 |
|
lower |
87 |
0.917 |
0.833 |
0.917 |
|
lower |
87 |
0.00023 |
0.00025 |
0.00031 |
|
lower |
87 |
0.0082 |
0.0108 |
0.0041 |
|
higher |
57 |
1.000 |
1.000 |
0.994 |
|
higher |
1 |
0.663 |
0.807 |
0.659 |
Every design completed on every backend, and the structural-quality means sit within a few thousandths of one another. Results are comparable in kind across backends but not bit-identical — compilation and BioIR’s own kernels both change floating-point numerics — so small differences between the three columns are expected and none of the gaps here is larger than that.
Warning
These are means over a small evaluation set, not a published accuracy
benchmark, and two rows are weaker than the rest. sequence_adherence is
defined on 57 of the 87 designs, and ss_conditioning_accuracy on exactly
one — the spread from 0.659 to 0.807 in that last row is a single design and
carries no information about backend quality. Do not select a backend on
these numbers.
Cold-Start Behavior#
The NIM reports ready as soon as its weights are loaded; it does not pre-warm
every token-count bucket the oss_torch_compile backend compiles for. This
means the first request against a given token bucket after startup pays a
Triton JIT compilation cost, once per bucket per container start, and later requests against the same (or
a smaller, same-bucket) size run at steady-state speed. This is a deliberate
trade: a NIM that is servable in well under a minute, with the compile cost
spread across a handful of early requests, versus one that is uniformly fast
but takes several minutes to report ready.
Measured on one H100, thinking_medium (the checkpoint’s default recycle
strategy), one design:
Tokens |
Bucket |
Cold (first request in bucket) |
Steady state |
Penalty |
|---|---|---|---|---|
100 |
128 |
30.3 s |
3.1 s |
9.8x |
150 |
160 |
22.2 s |
3.6 s |
6.2x |
250 |
256 |
21.9 s |
4.4 s |
5.0x |
650 |
768 |
54.9 s |
28.4 s |
1.9x |
694 |
768 (already warm from 650) |
28.5 s |
28.4 s |
1.0x |
Two things follow from this table. First, the penalty is per bucket, not “the first request ever”: a deployment serving a wide range of input sizes pays this cost once per distinct bucket its clients exercise, scattered across whichever requests happen to land first. Second, after a bucket is warm, a smaller request that rounds up into that same bucket (694 tokens padding to the 768 bucket already warmed by a 650-token request) pays nothing extra.
If a deployment or a demo needs a specific input size to respond quickly from the first request, send one request of that size to warm its bucket before the size matters.
Memory Footprint#
This NIM ships with no refolding oracle built in (refer to
Oracles), so the design model’s own memory
footprint is what a deployment plans capacity around. That footprint grows
with the square of the token count and with the batch size, and
sampling.batch_size defaults to sampling.num_replicates: a multi-replicate
request is a single large forward pass unless it says otherwise. Refer to
Replicates and
Batching. A mounted oracle that
calls out to an independently hosted structure-prediction NIM (the
reference implementations this NIM ships against OpenFold3 and Boltz-2 both
work this way) adds no GPU memory cost of its own here, since the model it
calls lives entirely in that other service.