Benchmarks
With BioNeMo GenMol, a benchmark for both performance and accuracy is built on two fragment completion tasks, motif-extension and scaffold-decoration, with 10 tests per task drawn from the SAFE-DRUGS dataset. These are derived by following tests from SAFE-GPT. For more information about these tasks, refer to Gotta be SAFE: A New Framework for Molecular Design.
For the task of motif-extension, parameters are chosen as:
- mask_length = 17
- temperature = 1.2
- noise = 1.6
For the task of scaffold-decoration, parameters are chosen as:
- mask_length = 17
- temperature = 1.2
- noise = 2.0
Performance
Average wall-time (in seconds) for generating 1000 molecules across 10 tests on each supported GPU.
Accuracy
Accuracy is evaluated from generating 100 molecules and computing the following metrics:
- validity: fraction of generated SMILES that are valid.
- uniqueness: ratio of unique molecules from all valid molecules.
- diversity: average pair-wise distances in molecular fingerprints of generated molecules.
- novelty: average distances of molecular fingerprints from the input molecule to generated molecules.
- quality: fraction of generated molecules with a QED score greater than 0.6 and an SA score less than 4.
Reference values measured on H100:
Values are single-run measurements and standard deviations across runs are not reported. Accuracy is broadly consistent across the validated GPU matrix, while validity stays within approximately ±2 percentage points and quality within approximately ±4 percentage points across all 13 supported SKUs.
v1 Compared to v2 on H100
The following table compares v1 and v2 measurements on the same H100 reference SKU, using the same benchmark harness and methodology in both runs.
Motif-extension quality is sensitive to hyperparameter selection (temperature,
noise, and mask-length). The values reported above use the defaults listed in the
test parameters at the top of this page; customer applications can achieve different
accuracy/quality trade-offs by tuning these parameters.