> This page is for ProteinMPNN, version 1.0.0.
> For other versions, use one of these documentation indexes:
> - 1.2.0 (Latest) (default): https://docs.nvidia.com/nim/bionemo/proteinmpnn/1.2.0/llms.txt
> - 1.1.0: https://docs.nvidia.com/nim/bionemo/proteinmpnn/1.1.0/llms.txt
> - 1.0.2: https://docs.nvidia.com/nim/bionemo/proteinmpnn/1.0.2/llms.txt
> - 1.0.0: https://docs.nvidia.com/nim/bionemo/proteinmpnn/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/bionemo/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/bionemo/_mcp/server.

# Benchmarking

## Accuracy

Accuracy benchmarking measures how well ProteinMPNN can predict amino acids sequences from 3D structures of known proteins. The benchmark is also called sequence recovery rate.

The sequence recovery rate is a critical metric for evaluating the performance of ProteinMPNN, as it quantifies the neural network's ability to generate protein sequences that closely resemble the native sequences found in nature. This metric is calculated by comparing the designed amino acid sequences produced by ProteinMPNN to the corresponding native sequences of the target proteins.

The sequence recovery rate determines the percentage of residues in the designed sequence that are identical to those in the native sequence at each position. A higher sequence recovery rate indicates that the model is more successful at capturing the intrinsic sequence features and evolutionary constraints that define the native protein, indicating that the designed protein is more likely to adopt the correct structure and function.

For example, ProteinMPNN has demonstrated an average sequence recovery rate of 52.4%, which significantly exceeds the 32.9% recovery rate achieved by traditional methods, such as Rosetta, highlighting its superior capability in accurately designing protein sequences that mimic natural proteins.[^1][^2]

## Performance

ProteinMPNN run time depends on many factors, such as number of atoms in the
inputs, number of chains, number of sequences that needs to be generated,
provided list if temperatures, and so forth.

As a good representative of overall performance, we measure main performance
characteristic of ProteinMPNN as average number of amino acids per seconds.

## Sample benchmarking scripts

This NIM comes with a simple benchmarking script that can measure both accuracy
and performance. It is useful to make sure that the neural network provides sane
results on some known proteins.

The script is already packaged in NIM's docker image. You can view and study the
benchmark using following command:

```bash
docker run --entrypoint cat nvcr.io/nim/ipd/proteinmpnn:1.0.0 /opt/benchmark.py
```

To execute the benchmark, follow this sequence:

1. Make sure NIM is running as described in Quickstart Guide.

2. Benchmark script automatically downloads test dataset. To save time and
   bandwidth it is recommended to provide local cache directory. This way the
   script will be able to reuse already downloaded data. Execute following
   command to setup cache directory.

```bash
export LOCAL_NIM_CACHE=~/.cache/nim
```


3. Execute the benchmark.

```bash
docker run -it --net host -v "$LOCAL_NIM_CACHE":/home/nvs/.cache/nim \
       nvcr.io/nim/ipd/proteinmpnn:1.0.0 /opt/benchmark.py
```


[^1]: [Dauparas J, Anishchenko I, Bennett N, Bai H, Ragotte RJ, Milles LF, Wicky BIM, Courbet A, de Haas RJ, Bethel N, Leung PJY, Huddy TF, Pellock S, Tischer D, Chan F, Koepnick B, Nguyen H, Kang A, Sankaran B, Bera AK, King NP, Baker D. Robust deep learning-based protein sequence design using ProteinMPNN. Science. 2022 Oct 7;378(6615):49-56. doi: 10.1126/science.add2187. Epub 2022 Sep 15. PMID: 36108050; PMCID: PMC9997061.](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9997061/)

[^2]: [Wang T, Jin X, Lu X, Min X, Ge S, Li S. Empirical validation of ProteinMPNN's efficiency in enhancing protein fitness. Front Genet. 2024 Jan 11;14:1347667. doi: 10.3389/fgene.2023.1347667. PMID: 38274106; PMCID: PMC10808456.](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10808456/)