Skip to navigation

Release Notes

September 3, 2026

Release 1.6.0

Summary

This release substantially reduces prediction latency, particularly for short sequences, and adds the ability to serve requests from several GPUs in one container.

Key Features

  • Faster predictions through CUDA graph capture: The BioNeMo Inference Runtime now captures the diffusion module as a CUDA graph. What this removes is per-request kernel launch overhead, which is a fixed cost rather than a proportional one: across the benchmark suite on H100 it takes roughly 7-8 seconds off every prediction, whatever the input size.

    The speedup therefore looks very different at different sizes, from the same change. A 186-residue prediction drops from 10.2 to 2.9 seconds, because about 7 of those 10 seconds were launch overhead. A 1869-residue prediction drops from 82.5 to 72.5 seconds, the same 10 seconds but now a small fraction of the total. Expect a large improvement on short inputs, a modest one on long inputs, and on the longest inputs on some SKUs no measurable change. Refer to Performance for measured per-SKU results.

  • Multi-GPU serving: A container given more than one GPU starts one inference process per GPU and serves requests concurrently. Each prediction still runs on a single GPU, so this raises throughput rather than lowering single-request latency. Refer to Configuration Alternatives.

  • Bounded request queue: When every GPU is busy and the queue is full, the NIM returns HTTP 503 rather than accepting work it cannot start. Clients should retry after a short delay.

  • Serving your own checkpoint is documented and enforced: MODEL_PATH and MODEL_FILE_NAME select a checkpoint of the same architecture, fine-tuned weights for example. A path that does not resolve is now fatal at startup instead of falling back to the shipped checkpoint, which previously made a misconfigured mount look like a working one. Refer to Configuration Alternatives.

  • Reproducible predictions: The sampler is seeded per request, so the same input returns the same structures on every call. Asking for several diffusion samples still returns several different structures within that response. It is repeat calls that now agree. NIM_INFERENCE_SEED= (empty) restores the previous behaviour, where repeat calls varied.

  • Abandoned requests are discarded: A queued request whose client has disconnected is dropped rather than run, and reported as HTTP 499. Previously one abandoned long prediction could hold the GPU while every request behind it waited for a result nobody would read.

  • Out-of-memory requests are reported, not hidden: A prediction too large for the GPU now returns HTTP 413 naming the residue count, the sample count and how much memory was wanted. It previously returned HTTP 500 with “Internal server error.”, which was indistinguishable from a genuine fault.

  • Updated base image: The container now builds on CUDA 13.2 and PyTorch 2.12. This also reduced the container’s CVE exposure relative to 1.5.0.

  • Smaller container: Roughly 11GB to download, down from the previous release.

Notes and Limitations

  • Requests are not batched. Concurrency comes from serving one request per GPU; a single request never uses more than one GPU.
  • All features from release 1.5.0 remain supported.