Benchmarking

Accuracy

Accuracy benchmarking measures how well ProteinMPNN can predict amino acid sequences from 3D structures of known proteins. The benchmark is also called sequence recovery rate.

The sequence recovery rate is a critical metric for evaluating the performance of ProteinMPNN, as it quantifies the neural network’s ability to generate protein sequences that closely resemble the native sequences found in nature. This metric is calculated by comparing the designed amino acid sequences produced by ProteinMPNN to the corresponding native sequences of the target proteins.

The sequence recovery rate determines the percentage of residues in the designed sequence that are identical to those in the native sequence at each position. A higher sequence recovery rate indicates that the model is more successful at capturing the intrinsic sequence features and evolutionary constraints that define the native protein, indicating that the designed protein is more likely to adopt the correct structure and function.

For example, ProteinMPNN has demonstrated an average sequence recovery rate of 52.4%, compared with the 32.9% recovery rate achieved by traditional methods, such as Rosetta.¹²

Performance

ProteinMPNN run time depends on many factors, such as number of atoms in the inputs, number of chains, provided list of temperatures, and so forth.

As a representative measure of overall performance, the main performance characteristic of ProteinMPNN is the average number of amino acids per second.

Run the Sample Benchmarking Script

This NIM comes with a benchmarking script that can measure both accuracy and performance. It is useful to make sure that the neural network provides reasonable results on some known proteins.

The script is already packaged in the NIM’s Docker image. You can view and study the benchmark using the following command:

docker run --entrypoint cat nvcr.io/nim/ipd/proteinmpnn:latest /opt/benchmark.py

To run the benchmark, follow this sequence:

1. Make sure the NIM is running as described in the Quickstart Guide.

2. The benchmark script automatically downloads a test dataset. To save time and bandwidth, provide a local cache directory. This way the script can reuse already downloaded data. Run the following commands to set up the cache directory.

export LOCAL_NIM_CACHE=~/.cache/nim

mkdir -p "$LOCAL_NIM_CACHE"
sudo chmod 0777 -R "$LOCAL_NIM_CACHE"

3. Run the benchmark.

docker run -it --net host -v "$LOCAL_NIM_CACHE":/home/nvs/.cache/nim \
       nvcr.io/nim/ipd/proteinmpnn:latest /opt/benchmark.py