Benchmarks
Following with the tests from SAFE-GPT, a benchmark for both performance and accuracy is built on two fragment completion tasks: motif-extension and scaffold-decoration, for each of them 10 tests from the SAFE-DRUGS dataset were used. More information about these tasks can be found in this article.
For the task of motif-extension, parameters are chosen as:
- mask_length = 17
- temperature = 1.2
- noise = 1.6
- step_size = 1
For the task of scaffold-decoration, parameters are chosen as:
- mask_length = 17
- temperature = 1.2
- noise = 2.0
- step_size = 1
Performance
Average of wall-time (in seconds) of generating 1000 molecules for tests of the two tasks are below, showing relative performances of the model on different GPU models.
The end-to-end time of the molecular generation requests (on RTX6000) depends on following parameters:
- num_molecule: number of molecules to be generated.
- steps: number of steps to recover all masks.
- context_length: number of tokens, including the ending and masking tokens, converted from the input molecular sequence (SMILES).
Comparison of wall-times on num_molecule:
Comparison of wall-times on steps:
Comparison of wall-times on context_length:
Accuracy
Accuracy of the model have been evaluated from generation of 100 molecules on following metrics:
- validity: fraction of generated SMILES that are valid.
- uniqueness: ratio of unique molecules from all valid molecules.
- diversity: average pair-wise distances in molecular fingerprints of generated molecules.
- novelty: average distances of molecular fingerprints from the input molecule to generated molecules.
- quality: fraction of generated molecules that are with QED_score > 0.6 and SA_score < 4.
A summary of accuracy evaluations is below