Benchmarking

Accuracy

Accuracy benchmarking for RFdiffusion involves generating long sequences. However, given a fixed scaffolding input, there still exist hundreds of possible configurations at the same length. Therefore, we currently only test whether the sequences generated under fixed random number conditions in the NIM container match those generated by the public version on GitHub.

The sequences generated by the GitHub version have undergone manual verification beforehand. This test assesses NIM’s ability to faithfully reproduce these sequences. It is important to note that accuracy can vary depending on the GPU microarchitecture. When executed on the same architecture, the outputs match exactly. When evaluated across different GPU architectures, the mean RMSE (Root Mean Square Error) between atoms in the reference dataset and the generated dataset is below 0.64 Ångströms. Here, mean is the average RMSE across all benchmark runs; individual runs can vary within the min–max range.

Performance

The run time of RFdiffusion depends on several factors, including the number of atoms in the inputs, the number and length of chains, and how many chains need to be generated.

To represent overall performance effectively, we measure RFdiffusion’s main performance characteristic as the average number of generated amino acids per second per diffusion step. We calculate this by multiplying the length of the generated sequence (number of amino acids) by the number of diffusion steps, and dividing by the measured run time in seconds. For instance, when considering the generation of a protein binder of a specific length using a certain number of diffusion steps, one can estimate the time by dividing the product of the protein length and the number of steps by the performance metric:

time (seconds) ≈ (protein length × number of diffusion steps) ÷ performance (aa/s/step)

This calculation gives an estimate of the time needed to generate one protein structure. For example, generating a 100-amino-acid binder with 50 diffusion steps on an H100 (up to 969.1 aa/s/step) takes an estimated (100 × 50) ÷ 969.1, or about 5.2 seconds.

Starting with 2.0 release of RFdiffusion NIM, the model is optimized using NVIDIA Warp and NVIDIA TensorRT frameworks.

With optimizations, RFdiffusion NIM runs up to two times faster than the non-optimized version. Performance varies by GPU architecture. The following measurements (amino acids per second per step) are from v2.3.0:

Run the Sample Benchmarking Script

This NIM comes with a benchmarking script that can measure both accuracy and performance. It is useful to make sure that the neural network provides the same results on some known proteins.

The script is already packaged in NIM’s Docker image. You can view and study the benchmark using the following command:

docker run --entrypoint cat nvcr.io/nim/ipd/rfdiffusion:2 /opt/nim/benchmarking.py

To execute the benchmark, follow this sequence:

1. Make sure NIM is running as described in the Quickstart Guide.

2. Benchmark script automatically downloads test dataset. To save time and bandwidth it is recommended to provide a local cache directory. This way the script will be able to reuse already downloaded data. Execute the following command to set up the cache directory.

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
sudo chmod 0777 -R "$LOCAL_NIM_CACHE"

3. Execute the benchmark.

docker run -it --net host -v "$LOCAL_NIM_CACHE":/opt/nim/.cache --entrypoint "" \
    nvcr.io/nim/ipd/rfdiffusion:2 \
    /opt/nim/benchmarking.py --benchmark-type both