> This page is for version 0.2.1.
> For other versions, use one of these documentation indexes:
> - 0.2.2 (latest) (default): https://docs.nvidia.com/bionemo/inference-runtime/latest/llms.txt
> - 0.2.1: https://docs.nvidia.com/bionemo/inference-runtime/0.2.1/llms.txt
> - 0.2.0: https://docs.nvidia.com/bionemo/inference-runtime/0.2.0/llms.txt
> - 0.1.1: https://docs.nvidia.com/bionemo/inference-runtime/0.1.1/llms.txt
> - 0.1.0: https://docs.nvidia.com/bionemo/inference-runtime/0.1.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/bionemo/inference-runtime/llms.txt.

# Development Workflow

How to build, run, test, and lint BioNeMo Inference Runtime (BioIR). For coding
style and conventions, refer to [Coding Guidelines](/bionemo/inference-runtime/0.2.1/references/coding); for the public
API, refer to [Python API](/bionemo/inference-runtime/0.2.1/references/api).

## Prerequisites for BioIR Development Workflow

- **GPU**: An NVIDIA GPU listed in the [BioIR Support Matrix][support-matrix].
  - BioIR's CuTeDSL kernels, CUDA graphs, and CUDA runtime paths use NVIDIA GPU
    capabilities; they do not run on CPU-only or non-NVIDIA systems. Which
    architectures the fused kernels cover, and what the rest fall back to, is in
    [Support Matrix][support-matrix].
  - H200, H100, A100, L40S, GB200, and GB300 are release-qualified with measured
    speed, peak memory, and accuracy in the [benchmarks][benchmarks].

- **Driver**: minimum version [580][driver-580].
  - The NVIDIA Container Toolkit (below) reads the CUDA version requirement off
    the image label and enforces it at `docker run`.

- **Docker**: [Docker Engine][docker-engine] (minimum version 23.0.1), and the
  [NVIDIA Container Toolkit][container-toolkit] (minimum version 1.13.5).
  - They are needed for `docker/dev.sh` and `make -C docker wheel`.
  - Ignore them if the host already has Python and the toolchain below, and
    docker is not used.

- **Python**: 3.12 for source development. `.python-version` selects the latest
  3.12 patch through uv, and the extension targets `cp312-abi3`. The package
  metadata declares `requires-python >=3.12,<3.15`; `uv.lock` resolves Python
  3.12, 3.13 and 3.14 on x86_64 and aarch64, and CI installs and tests the
  release wheel on each. Lift the cap only after the dependencies publish wheels
  for the next Python and CI tests it.

- **uv**: 0.13.x on a configured host. [Install uv][uv-install] if it is not
  already available. The development image includes the repository's minimum
  accepted version.

- **Toolchain**: a C++17 compiler and CUDA toolkit headers, only to build the
  extension from source. CMake and nanobind are declared build dependencies, so
  the build frontend supplies them.

Verify these requirements with the commands in
[Collecting System Information][system-information]. If the requirements are
not met, follow the [GPU Stack][gpu-stack] guide.

The kernels are **precompiled**. Neither building the wheel nor running it needs
`nvcc`. Running needs the driver's `libcuda.so.1`, plus a C compiler and
`Python.h` for Triton's launcher.

Host and development-container commands use uv and the committed lock. The
container reuses its preinstalled locked dependencies for native rebuilds.

For convenience, `docker/dev.sh` builds the dev image and opens a shell in it,
with the checkout and the caches — weights, ccache, uv/pip, compiled kernels —
mounted from the host; refer to [Docker Images][docker-images].

[system-information]: /bionemo/inference-runtime/0.2.1/references/system-information
[gpu-stack]: /bionemo/inference-runtime/0.2.1/references/gpu-stack
[docker-images]: /bionemo/inference-runtime/0.2.1/references/docker-images
[support-matrix]: /bionemo/inference-runtime/0.2.1/references/support-matrix#gpus
[benchmarks]: /bionemo/inference-runtime/0.2.1/references/benchmark
[driver-580]: https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-580-65-06
[docker-engine]: https://docs.docker.com/engine/install
[container-toolkit]: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html
[uv-install]: https://docs.astral.sh/uv/getting-started/installation/

## Clone the Repository

Configure [SSH authentication with GitHub][github-ssh], then clone the
repository and fetch its Git LFS objects and submodules:

```bash
git lfs install &&
  GIT_LFS_SKIP_SMUDGE=0 \
    git clone --recurse-submodules \
      git@github.com:NVIDIA-BioNeMo/BioNeMo-Inference-Runtime.git &&
  cd BioNeMo-Inference-Runtime
```

That leaves you on the host checkout.

## Create the development environment

### Configured host

From the repository root, create the locked Python 3.12 environment and build
the editable native extension:

```bash
uv sync --locked
uv run --locked python -c "import bionemo_ir.libs._cutedsl_kernels; print('ok')"
uv run --locked prek install
```

uv downloads a compatible Python when the host has no Python 3.12 interpreter.
It installs the `dev` dependency group by default. If `nvcc` is not on
`PATH`, set the wheel's CUDA tag explicitly:

```bash
CUDA_TAG=cu132 uv sync --locked
```

### Development container

Build the image and open its shell from the host:

```bash
docker/dev.sh
```

The image already contains the third-party dependencies. Verify the checked-out
lock, register the editable project in its writable environment, and install the
hooks inside the shell:

```bash
uv lock --check
uv pip install --no-deps -e .
prek install
```

## Build

### Build a Wheel

From the host, invoke the containerized wheel build. The source is copied into
the image rather than mounted, so your checkout is untouched and only `dist/` is
written.

```bash
# Run from the host; this command builds the wheel in a container.
make -C docker wheel
ls dist
```

The artifact is `bionemo_ir-<version>+cu<xyz>-cp312-abi3-linux_<arch>.whl`.

To build directly on a configured host or inside the development container, use
the lock-constrained repository wrapper:

```bash
scripts/build_wheel.sh --out-dir dist
```

The wrapper exports the exact `build` group from `uv.lock`, constrains the
isolated PEP 517 environment with its hashes, and invokes `uv build`. It shares
`build/` with the [editable install](#editable-install), and it deletes the
extension from the source tree on its way through — every real build does, so a
stale copy cannot shadow the fresh one — but only an editable install puts one
back. Until you reinstall, `pytest` refuses to collect: the session starts by
requiring the extension and stops with the reason it could not load it.

Restore the host editable build with:

```bash
uv sync --locked --reinstall-package bionemo-ir
```

### Editable Install

On a configured host, `uv sync` creates `.venv`, installs the locked `dev`
group and installs BioIR in editable mode. Activate that environment for plain
`python`, `pip`, and `prek` commands:

```bash
uv sync --locked
source .venv/bin/activate
```

Inside the development container, verify the lock and register the mounted
checkout without resolving dependencies again:

```bash
uv lock --check
uv pip install --no-deps -e .
```

Both paths compile `bionemo_ir.libs._cutedsl_kernels`, a CPython 3.12 stable ABI
(`.abi3.so`) nanobind extension embedding verified CUBIN build artifacts.
ClaudeKit links the CUDA runtime statically; the host supplies `libcuda.so.1`.
Generated packs and records are excluded from Git. Before an isolated build,
obtain the matching complete `build/cubin_corpus` artifact from CI and select
its extracted directory:

```bash
export BIOIR_CUBIN_ARTIFACT_ROOT="$PWD/build/cubin_corpus"
```

A source-free checkout uses the matching corpus from CI with the same
`BIOIR_CUBIN_ARTIFACT_ROOT` setting. Source distributions stage verified
payloads inside the archive, so their native build needs no private compiler
code. Verify the active environment:

```bash
python -c "import bionemo_ir.libs._cutedsl_kernels; print('ok')"
```

### CuTeDSL Compiler

The compiler and its TVM-FFI launcher live in the `cutedsl` dependency group,
which `dev` includes. The package itself does not depend on them. Without the
compiler, supported variants launch their shipped CUBINs, as a source-free
build does. Configurations without a shipped CUBIN use the operator's fallback
where available, such as AdaLN's vanilla PyTorch implementation. Only functions
that compile from source import the compiler.

`nvidia-cutlass-dsl[cu13]` installs both `nvidia-cutlass-dsl-libs-base` and
`nvidia-cutlass-dsl-libs-cu13`. The two wheels ship the same native compiler
module, each embedding its own ptxas (CUDA 12.9 and 13.1), and whichever unpacks
last compiles every kernel. The image reinstalls the CUDA 13 wheel last, and the
published CUBIN packs come from it. After any `uv sync`, do the same:

```bash
uv pip install --no-deps --reinstall-package nvidia-cutlass-dsl-libs-cu13 \
  'nvidia-cutlass-dsl-libs-cu13==4.5.2'
```

Use the `nvidia-cutlass-dsl` version `pyproject.toml` pins. Without this step,
source mode and locally built packs run different machine code from the shipped
packs, so their timings do not transfer. The CUBIN build tooling warns about a
foreign compiler and refuses to publish packs from it.

uv's host project workflow builds in an isolated environment constrained by the
`build` group in `uv.lock`. The image installs the same build and development
groups on top of the NGC Python environment so it keeps the tested torch,
Triton, and CUDA stack; `--no-deps` adds only the mounted checkout to its
writable development venv. Do not use the container command in a general
interpreter because it suppresses dependencies the package requires.

One thing the isolated build environment does not have is `torch`. The wheel's
CUDA tag comes from `nvcc` first and only falls back to `torch.version.cuda`, so
this is invisible wherever `nvcc` is on `PATH`. Where it is not, set the tag
rather than reaching for the flag:

```bash
# Configured host.
CUDA_TAG=cu132 uv sync --locked

# Development container.
CUDA_TAG=cu132 uv pip install --no-deps -e .
```

On a configured host, uv tracks `setup.py`, the CMake and C++ sources,
committed CUBIN indexes and packs, `build.env`, and build-affecting environment
variables in the editable project's cache key. A later `uv run` or `uv sync`
rebuilds the extension when one of those inputs changes. Python-only edits
remain immediately visible through the editable install.

### Change dependencies

Consumer dependency ranges live in `[project.dependencies]` in `pyproject.toml`.
Development-only requirements live in `[dependency-groups]`. The focused `build`
group pins PEP 517 requirements; `cutedsl` holds the kernel compiler; `release`
adds artifact inspection tools and is not installed in the default environment.

Regenerate and verify the exact development resolution after either dependency
source changes:

```bash
uv lock
uv sync --locked
```

The lock covers Python 3.12 on Linux x86_64 and aarch64. Resolution fails when a
wheel-only dependency does not support either architecture. Upgrade
deliberately with `uv lock --upgrade`, inspect the lock diff, then run the same
locked sync and test gates; routine `uv lock` preserves versions already
selected.

## Run

### Data Dependencies

The commands below use the configured-host form. Inside the development
container, omit the `uv run --locked` prefix. In a release-wheel environment,
also omit it and run the script from a checkout with the environment
activated, for example `scripts/fetch_weights.sh --model boltz-2`:
`uv run --locked` builds BioIR from source. The script needs `bash` and `curl`.

Two things are needed:

- **Inputs**: `examples/data/samples/` carries small, real inputs — monomers,
  heterooligomers, `RNA/DNA/ligand` complexes, MSAs, and templates — so both
  routes below have something to fold without assembling your own data.

- **Weights**: pull the Boltz-2 and OpenFold2 checkpoints into the shared
  cache.

  ```bash
  uv run --locked scripts/fetch_weights.sh --model boltz-2
  uv run --locked scripts/fetch_weights.sh --model openfold2
  ```

  OpenFold3 uses a gated Hugging Face checkpoint. Create a Hugging Face account,
  request access and accept the terms on [`OpenFold/OpenFold3`][openfold3-hf],
  then export a token and fetch the checkpoint:

  ```bash
  export HF_TOKEN=hf_...
  uv run --locked scripts/fetch_weights.sh --model openfold3
  ```

  The script reads only `HF_TOKEN`. A token stored by `hf auth login` covers
  only the automatic download on the first prediction.

  Refer to [Model Weights](/bionemo/inference-runtime/0.2.1/references/model-weights) for the other
  families, cache layout, token alternatives, and manual checkpoint staging.

### Run From the Wheel

The wheel is self-contained: the CUBINs are compiled into the extension, so
running needs no source checkout, no CMake, and no CUDA toolkit — only a driver
providing `libcuda.so.1` and the [build tools][install-build-tools] Triton uses
for its launcher.

[install-build-tools]: /bionemo/inference-runtime/0.2.1/install#install-build-tools

In the dev image every dependency is already installed, so build the venv on top
of them and install the wheel alone:

```bash
python -m venv --system-site-packages /tmp/venv
/tmp/venv/bin/pip install --no-deps dist/bionemo_ir-*.whl
/tmp/venv/bin/python examples/folding/run_demo.py --output-dir /tmp/bioir-demo
```

An empty environment instead makes pip resolve the whole dependency closure —
torch, ray, and the rest — from PyPI, which takes a while on a cold pip cache:

```bash
python -m venv /tmp/venv
/tmp/venv/bin/pip install dist/bionemo_ir-*.whl
```

That is the slower path and the stricter one: `--no-deps` above says nothing
about whether the wheel's dependencies are declared correctly.

Python puts the script's own directory on `sys.path`, not the repo root, so this
imports the installed package even when run from the checkout. That makes it the
check that a change survives packaging: a module that only imports because the
source tree happened to be on `sys.path` fails here. The
[minimal runtime image](/bionemo/inference-runtime/0.2.1/references/docker-images#minimal-runtime-image) applies both
checks at once, on a CUDA base carrying nothing else.

### Run From the Code

With the editable install from [Build](#build):

```bash
uv run --locked python examples/folding/run_demo.py            # Boltz-2 on the bundled T1031 sample
uv run --locked python examples/folding/run_demo.py \
    --model-source openfold3 \
    --input examples/data/samples/monomers/T1031.json \
    --output-dir output
uv run --locked python examples/folding/run_demo.py --help
```

The default run writes `T1031.cif` and `T1031_scores.json` to the output
directory, and takes a few minutes on an A100. Which model sources the demo can
drive is in [the folding demo README](https://github.com/NVIDIA-BioNeMo/BioNeMo-Inference-Runtime/blob/v0.2.1/examples/folding/README.md) —
it is narrower than the support matrix.

## Test

```bash
uv run --locked scripts/run_tests.sh
```

This stages missing model weights, then runs pytest in two phases: an
xdist-parallel bulk phase, then the trees that must run serially.

Expect a green run with a large number of skips, from missing checkpoints or
inputs — what actually runs depends on what you staged.

```bash
uv run --locked scripts/run_tests.sh --no-weights
uv run --locked scripts/run_tests.sh --help
```

For one tree, file, or case, call `pytest` directly:

```bash
uv run --locked pytest -q tests/ops
uv run --locked pytest -q tests/_torch
uv run --locked pytest -q tests/ops/test_gated_sigmoid.py::test_gated_sigmoid_config_selection_is_source_free
```

`BIOIR_DEBUG_ASSERTS=1` turns on device-side input validation (for example
the left-aligned pair-mask check of the CuTeDSL attention backends). Each
check reads a device tensor back to the host, which stalls the launch queue
and is illegal inside CUDA-graph capture, so it is off by default.

## Lint and Format

Style is enforced by [prek](https://github.com/j178/prek), which runs the hooks
pinned in `prek.toml`. The default `dev` group installs it; wire up the git
hooks once:

```bash
uv run --locked prek install
```

They then run on `git commit`. To run them by hand:

```bash
uv run --locked prek run
uv run --locked prek run --all-files
```

A clean `uv run --locked prek run --all-files` is required for every PR, and
CI runs the same hooks. `prek.toml` lists them: a formatter and a linter per
language in the tree, plus the SPDX license header every source file carries.
Inside the development container, use the equivalent bare `prek` commands.

Both CI systems run their CPU gates through `scripts/ci/`: `style.sh` wraps the
hooks above, and `contract.sh` runs the build-contract and CUBIN suites with no
GPU and no built extension.

Release CI checks the installed wheel's dependencies without modifying their
metadata. It treats the known cuSPARSELt ARM64 wheel-tag defect as a warning
because PyTorch's exact dependency pin excludes the corrected post-release.
Every other dependency-check failure blocks wheel acceptance. See the
[installation note][wheel-installation].

[wheel-installation]: /bionemo/inference-runtime/0.2.1/install#install-the-package

## Open a Pull Request

Fork, then sign commits with `git commit -s`; refer to
[Contributing](/bionemo/inference-runtime/0.2.1/community/contributing). Ensure all three pass:

- **Style** — `uv run --locked prek run --all-files`
- **Build** — `uv sync --locked`, then import the extension
- **Tests** — `uv run --locked scripts/run_tests.sh`

Write the MR/PR title as the commit you want in history; squash is the default.
Refer to [Commits](/bionemo/inference-runtime/0.2.1/references/coding#commits).

[github-ssh]: https://docs.github.com/en/authentication/connecting-to-github-with-ssh
[openfold3-hf]: https://huggingface.co/OpenFold/OpenFold3