Performance#

NIM Accuracy#

The MSA Search NIM is based on GPU MMSeqs2, a highly-sensitive, highly-performant, GPU-accelerated package for a variety of sequence-alignment tasks. The NIM’s accuracy should match that of the open-source version of MMSeqs2 in most settings.

Note

Running on hardware that is not listed as supported in the prerequisites section may produce results that deviate from the expected accuracy.

The accuracy of the NIM is measured by the comparison of the NIM’s results against those of MMSeqs2. In general, we expect the NIM to produce identical results to GPU MMSeqs2, barring random seed differences and differences in numerical stability across architectures.

Factors Affecting NIM Performance#

In addition to runtime factors, such as the number of concurrent users, the performance of the MSA NIM is mostly determined by three hardware factors:

  • The number and type of GPUs available to the NIM

  • Whether the databases in the NIM are being served via the NIM’s GPU Server

  • The speed of the underlying storage for the NIM’s model files.

In general, the NIM will achieve optimal latency at four GPUs, when each database can be loaded on a GPU Server. The NIM will generally achieve optimal throughput at the maximum number of GPUs in your compute instance (usually 8), especially when those GPUs are of Compute Capability 9.0 or greater. Finally, the NIM requires an NVMe SSD or other fast storage with at least 3,500 MB/s throughput for good performance. Faster storage may not lead to significant improvements in performance.

We measure the performance of the MSA Search NIM in sequences-per-second using a fixed set of input sequences derived from the CAMEO Protein Folding Benchmark.

Tools and Testing Practices#

The MSA Search NIM is tested using a Python benchmarking script. This script is available below:

import argparse
import csv
import os
import sys
from typing import List, Literal
import yaml
import time
import requests

HEADERS = {
    "content-type": "application/json"
}

def parse_csv_to_dict(csv_file):
    result = {}
    with open(csv_file, mode='r') as infile:
        reader = csv.reader(infile)
        for row in reader:
            if len(row) >= 2:
                result[row[0]] = row[1]
    return result

def parse_fasta(content: str):
    """
    Parses FASTA formatted string into a dictionary of sequence ID to sequence.
    """
    sequences = {}
    sequence_lines = []
    seq_id = None

    for line in content.splitlines():
        line = line.strip()
        if line.startswith('>'):
            if seq_id:
                sequences[seq_id] = ''.join(sequence_lines)
                sequence_lines = []
            seq_id = line[1:].split()[0]  # Take the first word after '>'
        else:
            sequence_lines.append(line)

    if seq_id:
        sequences[seq_id] = ''.join(sequence_lines)

    return sequences

def read_fasta_file(file_path: str):
    """
    Reads a FASTA file from the given file path.
    """
    with open(file_path, 'r') as f:
        content = f.read()
    return parse_fasta(content)

def compute_similarity(predicted_alignment: dict, expected_alignment: dict):
    """
    Computes the percent similarity between two alignments.

    Parameters:
        predicted_alignment (dict): Dictionary of sequence ID to sequence from the predicted alignment.
        expected_alignment (dict): Dictionary of sequence ID to sequence from the expected alignment.

    Returns:
        float: Percent similarity between the predicted and expected alignments.
    """

    predicted_sequences = list(predicted_alignment.values())
    expected_sequences = list(expected_alignment.values())

    # Ensure the alignments have the same number of sequences
    if len(predicted_sequences) != len(expected_sequences):
        print("Number of sequences in predicted and expected alignments do not match")
        return 0.0

    alignment_length = len(predicted_sequences[0])
    # Ensure all sequences have the same length
    for seq in predicted_sequences + expected_sequences:
        if len(seq) != alignment_length:
            print("Sequences in the alignments do not have the same length")
            return 0.0

    total_positions = alignment_length * len(predicted_sequences)
    matching_positions = 0

    for pred_seq, exp_seq in zip(predicted_sequences, expected_sequences):
        for pred_res, exp_res in zip(pred_seq, exp_seq):
            if pred_res == exp_res:
                matching_positions += 1

    percent_similarity = (matching_positions / total_positions) * 100
    return percent_similarity

def run_prediction(identifier: str,
                   sequence: str,
                   iterations: int = 1,
                   max_msa_sequences: int = 500,
                   databases: List[str] = ["Uniref30_2302", "colabfold_envdb_202108", "PDB70_220313"],
                   search_type: Literal["alphafold2", "colabfold"] = "alphafold2",
                   output_formats: List[str] = ["a3m"],
                   nim_url_endpoint: str = "http://0.0.0.0:8000/biology/colabfold/msa-search/predict"):
    print(f"Running prediction for {identifier} ...")
    data = {
        "sequence": sequence,
        "e_value": 0.0001,
        "iterations": iterations,
        "databases": databases,
        "search_type": search_type,
        "max_msa_sequences": max_msa_sequences,
        "output_alignment_formats": output_formats
    }

    start = time.perf_counter()
    response = requests.post(nim_url_endpoint, json=data, headers=HEADERS)
    end = time.perf_counter()

    elapsed_time = end - start

    return response, elapsed_time

def main():
    parser = argparse.ArgumentParser(description='Script to test performance and accuracy.')
    parser.add_argument('--input-csv',
                        required=True,
                        help='Input CSV file containing sequence IDs and sequences')
    parser.add_argument('--pdbs',
                        default=None,
                        type=str,
                        help='Optional comma-separated list of sequence IDs to select from the input CSV file\'s first column. If specified, only those in [pdbs] will be analyzed. [Default: None, run all]')
    parser.add_argument('--expected-alignments',
                        default=None,
                        help='YAML file mapping sequence IDs to expected alignment filenames')
    parser.add_argument('--url',
                        default="http://0.0.0.0:8000/biology/colabfold/msa-search/predict",
                        type=str,
                        help='NIM URL endpoint for prediction requests')
    parser.add_argument('--output-yaml-name',
                        default="results.yaml",
                        help='Output YAML file (default: stdout)')
    parser.add_argument("--result-dir",
                        type=str,
                        default=os.curdir,
                        help="The output directory of the results.")
    parser.add_argument("--databases",
                        type=str,
                        default="Uniref30_2302,colabfold_envdb_202108,PDB70_220313",
                        help="A comma-separated list of databases to benchmark.")
    parser.add_argument("--benchmark-type", default="performance", type=str, help="Either accuracy or performance.")

    args = parser.parse_args()

    data_dict = parse_csv_to_dict(args.input_csv)
    if args.pdbs:
        keys = args.pdbs.split(",")
        print(f"Restricting analysis to sequence IDs: {keys}")
        data_dict = {key: val for key, val in data_dict.items() if key in keys}
    if len(data_dict) == 0:
        raise ValueError("Input data was empty")

    if args.expected_alignments:
        with open(args.expected_alignments, 'r') as f:
            expected_alignments_mapping = yaml.safe_load(f)
    else:
        expected_alignments_mapping = None

    ## Set up runtime args
    databases = args.databases.split(",")

    results = {}
    times = {}
    total_amino_acids = 0
    for key, sequence in data_dict.items():
        response, elapsed_time = run_prediction(key,
                                                sequence,
                                                nim_url_endpoint=args.url,
                                                databases=databases)
        times[key] = elapsed_time
        total_amino_acids += len(sequence)

        if response.status_code != 200:
            print(f"Prediction failed for {key} with status code {response.status_code}")
            continue

        try:
            response_json = response.json()
            ## TODO: make this get alignments[format]
            predicted_alignment_str = response_json.get('alignments').get('Uniref30_2302').get('fasta')
            if not predicted_alignment_str:
                print(f"No 'alignments' field in response for sequence ID {key}")
                continue

            # Parse predicted alignment
            predicted_alignment = parse_fasta(predicted_alignment_str)

            percent_similarity = None
            if expected_alignments_mapping and key in expected_alignments_mapping:
                expected_alignment_file = expected_alignments_mapping.get(key)
                try:
                    expected_alignment = read_fasta_file(expected_alignment_file)
                    percent_similarity = compute_similarity(predicted_alignment, expected_alignment)
                    print(f"Sequence ID {key}: Percent similarity = {percent_similarity:.2f}%")
                except Exception as e:
                    print(f"Error reading or parsing expected alignment for sequence ID {key}: {e}")
                    percent_similarity = None
            else:
                if expected_alignments_mapping:
                    print(f"No expected alignment file specified for sequence ID {key} in expected alignments mapping")
                percent_similarity = None

            results[key] = {
                'predicted_alignment': predicted_alignment_str,
                'elapsed_time': elapsed_time,
                'percent_similarity': percent_similarity
            }

        except Exception as e:
            print(f"Error processing response for sequence ID {key}: {e}")
            continue

    total_sequences = len(times)
    total_seconds = sum(times.values())
    sequences_per_second = total_sequences / total_seconds if total_seconds > 0 else None
    amino_acids_per_second = total_amino_acids / total_seconds if total_seconds > 0 else None

    bench_results = {
            "benchmark_config" : {"type" : args.benchmark_type},
            "sequences_per_second" : sequences_per_second,
            "raw_times" : times,
            "amino_acids_per_second" : amino_acids_per_second,
            "PDBs" : args.pdbs
        }

    output = {
        "schema_type" : "measurement",
        "benchmark_type" : args.pdbs,
        "nim_model_name" : "colabfold/msa-search",
        "benchmark_container" : os.environ.get("DOCKER_TAG", "msa-search-benchmark-default"),
        "benchmark_config_global" : {},
        "benchmark_results" : [bench_results]
    }

    telemetry_dir = os.path.join(args.result_dir, "nim_telemetry_data")
    os.makedirs(telemetry_dir, exist_ok=True)
    os.chmod(telemetry_dir, 0o777)


    if args.output_yaml:
        output_path = os.path.join(telemetry_dir, args.output_yaml)
        with open(output_path, 'w') as outfile:
            yaml.dump(output, outfile)
        os.chmod(output_path, 0o777)
        print(f"Output YAML file has been saved to: {os.path.abspath(output_path)}")
    else:
        yaml.dump(output, sys.stdout)

if __name__ == "__main__":
    main()

This script reports performance data in sequences_per_second and amino_acids_per_second, as well as the raw times per sequence.

The sequences for benchmarking are taken from PDB sequences. The sequences used for benchmarking are below, in CSV format:

PDB_ID,SEQUENCE,Command,MSA_UNIREF_MD5SUM,MSA_MGNIFY_B64,MSA_SMALLBFD_MD5SUM,PDB_STRUCTURE_B64,RUNTIME_S,SEQ_LEN
7WJ0_A,SGSMKTAISLPDETFDRVSRRASELGMSRSEFFTKAAQRYLHELDAQLLTGQ,baseline,,,,,,52
7WBN_A,GGSKENEISHHAKEIERLQKEIERHKQSIKKLKQSEQSNPPPNPEGTRQARRNRRRRWRERQRQKENEISHHAKEIERLQKEIERHKQSIKKLKQSEC,baseline,538f34b1abbfece3ee6027718f9f9484,,fe13552069a8e3a7848ea213b5aeb59a,,,98
7OX1_Z,GSHMQGCPTLAGILDINFLINKMQEDPASKCHCSANVTSCLCLGIPSDNCTRPCFSERLSQMTNTTMQTRYPLIFSRVKKSVEVLKNNKCPYFSCEQPCNQTTAGNALTFLKSLLEIFQKEKMRGMRGKI,baseline,64a0fd3cf17cafe98fad3da8a9cf36f0,,b22f91784e83c50a731431be0dbcc64d,,,130
8BFH_A,GPDSGVEIKRIMAKAFKSPLSSPQQTQLLGELEKDPKLVYHIGLTPAKLPDLVENNPLVAIEMLLKLMQSSQITEYFSVLVNMDMSLHSMEVVNRLTTAVDLPPEFIHLYISNCISTCEQIKDKYMQNRLVRLVCVFLQSLIRNKIINVQDLFIEVQAFCIEFSRIREAAGLFRLLKTL,baseline,f9b71089e3503e6b9868c92173b134f0,,d8611cede616e4992807e5119ea1785a,,,179
7SKH_B,MNAPVFQQPHYEVVLDEGPDTINTSLITVQALDLDEGPNGTVTYAIVAGNIINTFRINKHTGVITAAKELDYEISHGRYTLIVTATDQCPILSHRLTSTTTVLVNVNDINDNVPTFPRDYEGPFDVTEGQPGPRVWTFLAHDRDSGPNGQVEYSVVDGDPLGEFVISPVEGVLRVRKDVELDRETIAFYNLTICARDRGVPPLSSTMLVGIRVLDINDNLEHHHHHH,baseline,c854cba39b6d80dd678acf30de99fc05,,088b99b8e0e6337c9c46255cb6820e00,,,227
7WG3_K,PEPTIQCGSEPGPSPEWMVRHTLTPGDLRDLRVETIKSNVDLEDSPILMNISWILRADASIRLLKATKICVMGKSHFQSYSCIRCNYTQAFQTQTRPSGGKWTFSYVGFPVELNTVYFIGAHNIPNANMNEDGPSMAVNFTSPGCLDHVMKYKKKCIEAGSLWKPNITACKRSANTVEVNFTTSPLGDRYMALIQSTAVIGTSYVSEKELTRTSVVVHVTGESEGAVVQLTPYFHTCGNDCIRQRGTVVQCPQTG,baseline,29c24babf2814d27dce416c2f6692720,,b13552ca04564493b99765fd2aec757a,,,255
7QJO_B,MSNPYERGPAPTESSVTAVRGYFDTDTDTVSSLVSGFGGGTIYYPTDTSEGTFGGVVIAPGYTASQSSMAWMGHRIASQGFVVFTIDTITRYDQPDSRGRQIEAALDYLVEDSDVADRVDGNRLAVMGHSMGGGGTLAAAENRPELRAAIPLTPWHLQKNWSDVEVPTMIIGAENDTVASVRTHSIPFYESLDEDLERAYLELDGASHFAPNISNTVIAKYSISWLKRFVDEDERYEQFLCPPPDTGLFSDFSDYRDSCPHTTLEHHHHHH,baseline,6e5ddcb0da98615bdaf3e1affc50c5a3,,9cfdcba4b339f0e0a009866ecd6bb2b3,,,271
7ONG_A,GSHMNGLIYAVGGYDGTGYNTHLNSVEAYDPERNEWSLVAPLSTRRSGVGVAVLNGLIYAVGGYDGTGYNTHLNSVEAYDPERNEWSLVAPLSTRRSGVGVAVLNGLIYAVGGYDGTGYNTHLNSVEAYDPERNEWSLVAPLSTRRSGVGVAVLNGLIYAVGGYDGTGYNTHLNSVEAYDPERNEWSLVAPLSTRRSGVGVAVLNGLIYAVGGYDGTGYNTHLNSVEAYDPERNEWSLVAPLSTRRSGVGVAVLNGLIYAVGGYDGTGYNTHLNSVEAYDPERNEWSLVAPLSTRRSGVGVAVL,baseline,6295751208d63afaba3971eb86a3a4f9,,3a098f31570b1c1b21d2347a91301fcf,,,304
8B26_A,MAGKSPEEEHPVKAYGWAVKDRTTGILSPFKFSRRATGDNDIRIKILYCGICHTDLTSVKNEYEFLSYPLVPGMEIVGIATEVGSKVTKIKVGEKVAVAAYLGTCGKCYNCVNDLENYCPEVIIGYGTPYHDGTINYGGLSNETVVNERFVLRFPEKLSPAGGAPLLSAGITAYSAMRNHGLDKPGIHLGVVGLGGLGHLAVKFAKAFGVRVTVISTTPSKKDEAINNLGADAFLFSRDDKQMRAAIGTFDAIIDTLAVVHPIAPLLDLLRSHGKLVLVGAPSKPLELPTIPLLSGGKSLIGSAAGNVKQTQEMLDFAAEHDITANIEVIPIDYINTAMERLDKGDIRFRFVVDIENTLTPPPEP,baseline,2a581365cce2f8ff10bf582ac56b4a9b,,e99ecacebb4cd1eb68dc04e61252ab29,,,365
7TWL_D,SNAMALKKPGSLTIAGSGIASIGHITLETLALIKEADKIFYAVTDPATECYIQENSRGDHFDLTTFYDTNKKRYESYVQMSEVMLRDVRAGRNVLGIFYGHPGVFVAPSHRAIAIAREEGFQAKMLPGISAEDYMFADLGFDPSTYGCMTQEATELLVRNKKLDPSIHNIIWQVGSVGVDTMVFDNGKFHLLVERLEKDFGLDHKIQHYIGAILPQSVTVKDTFAIRDLRKEEVLKQFTTTSTFYVPPRTPAPIDPKAVQALGLPATVTKGAQDWTGFQSVSPAYGPDEMRAVAALDSFVPSQEKAVVHASRAMQSLMVDLALRPALLEQYKADPVAFANTRNGLTAQEKFALGLKKPGPIFVVMRQLPSAIASGQEPSQEEIARADDATALALVVVIVQ,baseline,6365b0454a9bbdc4353f22d9e60dfb50,,e8e4a04ac07fe1df1e6512657d8a7a13,,,400
7YP4_B,MGSSHHHHHHSSGLVPRGSHMRILFATVSEKSHLFTMVPLAWSLAAAGHEVHVASNPALTASIKSTGLTAVPVGKDHNLHEMLTENRDSLENPLSDWSTPELDRHSWEQVLMKFKVSVMFAYQTYNDCMVHELVDYARHWQPDLVIWDPVTYAGPVAARVVGAAHARLLWCIDIYAKMREVFLARLAEQPEERREDPMADWLGGILGRYGHTFDEEVVVGQWTIDQIPTSLQLPLSLRRVPVRYLPHNGPSEIPDWLREAPGRPRVVLTSGVSARAALGGTFMPVADMINTLGSMDIDVVAALPPEEVEALEKVPANTRIVDFVPLHALLPGASVLIHHGGFGSWGTALVNGVPQFIPTIRYADWWNKGTSLHEAGAGLVVHASELTAEVLRESVERLVEDASYREAAERLREENQRTPTPHDVVPVIEELTAEHGR,baseline,13f0c3dc29f089c34151f8bd26178b83,,10413e00a81b0a56000ba5a9a73de00a,,,437
7QSX_A,MARAQYEAGVRPYRETYYDPDYEPKDTDLLCAFRITPKPGVPMEEAAAAVAAESSTGTWTEVWSNLLTDLERYKARCYRIEGDVAYIAYPLDLFEEGSIVNIMSSIVGNVFGFKAVQALRLEDMRIPVAYLKTFPGPPTGIQVERDRLNKYGRPLLGGTIKPKLGLSAKEYARVVYECLRGGLDTTKDDENLNSQPFNRWRDRFLYVMEAVRKAEAETGERKGHWLNVTAGSTEEMLKRAEFAAELGSRYIMVDFLTAGFAAFASVRRRAEELGLMLHCHRAMHAVFDRQPNHGIHFRVLAKWLRMVGGDHVHTGTVVGKLEGDRAETLGIADLLREDYVPADPGRGLFFDQDWAGLKPVFPVASGGIHVWHVPDLVSIFGDDAFFLFGGGTHGHPRGSRAGATANRVAVEAVVQARNEGRDILAEGREILEEAARSCPELREAMELWGDVKFEVEA,baseline, 4392a9d8bfeb2a3e59c41a0e1c980c59,,201eb1341df10bf8c2a6079853f0910e,,,457
8FBD_B,MQTTTLNWDTVYAVPINIVNEAIKLKHPTPENFELLNGKYGNCSGSFEEWQITNGGDGSNIRLKIPIKNFKATIIGNRLNGKGGFAFANLEVQVKLKYLPHFPQSKNKDIELVDLKIRTQSDNPEDPAIIVISSYKNIQGFYFEDEYKLTEDDEFVVSYFYRLIKEWLEKNLHFFNYIFNTVNLNLYISDKEKWEWTKPSYVDYAYSEIEGDLSRSALGVLCMTGGRTGSKNQQQKIDPYAIPAASQSGFLISEERLLRNILLPTIPKKFPKSKGDEFEVINESSQGGGYSYILKLKKGKKIDLENIQAVGYTCTPYIQEMKIYLLGSYLKLETTTRVDLPLGVASICETTCEYKFKLSTNNKGEQTIAYEQIGSPVNIQYSENTGNVGLNIVVSFLSATLSFALTFVPGFGTFLAVGLIGGCLIGSVALIPTFIESYNSDTAPSIDLSLENSVSEITWNSSDVFNLDYVALAGPLQLGGTLQVQNS,baseline,0efe9edc802b7aaccecab4bb9f961419,,7d5b3933e45a294022763dc000416187,,,487
8E83_A,MAKKTSSKGKLPPSPKPRLPFIGHLHLLDNPLLHHTLIKLGKRYGPLYTLYFGSMPTVVASTPDLFKLFLQTHEATSFNTRFQTSAISRLTYDNSVAMVPFAPYWKFIRKLIMNDLLNATTVNKLRPLRSREILKVLKVMANSAETQQPLDVTEELLKWTNSTISTMMLGEAEEVRDIARDVLKIFGEYSVTNFIWPLNKFKFGNYDKRTEEIFNKYDPIIEKVIKKRQEIVNKRKNGEIVEGEQNVVFLDTLLEFAQDETMEIKITKEQIKGLVVDFFSAGTDSTAVSTEWTLSELINNPRVLKKAREEIDSVVGKDRLVDESDVQNLPYIKAIVKEAFRLHPPLPVVKRKCTQECEIDGYVVPEGALILFNVWAVGRDPKYWVKPLEFRPERFIENVGEGEAASIDLRGQHFTLLPFGSGRRMCPGVNLATAGMATMIASIIQCFDLQVPGQHGEILNGDYAKVSMEERPGLTVPRAHNLMCVPLARAGVADKLLSSHHHH,baseline,af7cc1ef8eeee6224e50dffd559774c2,,559e378c946b0126a8cad0890c495aaf,,,503
7ZHT_A,MSEEQSHADQDAYVADVDGILDVLRAQVLERKPDDIFQFISKSALSLQKDRGAESCDRINCKVKDEQKSRALTIIVFGASGDLAKKKTFPALFDLYCGGLLPPEVNIIGYARTKVDDVEKWKHETLMKYFSNLSERGCHAEDFLKHISYFCGAYDSVDDFKRLDAVIREKENAFKGPEKGGNRLFYLALPPSVFASVCESIHKGAMPQEVGGWVRVIIEKPFGRDTKSSAELSQALEPFFDESQLYRIDHYLGKEMVQNIITTRFANRIFSAVWNASNIACVQITFKETIGTEGRGGYFDNIGIIRDVMQNHLTQILALLAMEKPRSLDAECIRDEKVSVLKCIEPITKENCVLGQYTASADGSIPGYLEDVTVPEGSTCPTFAVMRLNINNDRWAGVPFILKAGKAVEQKYVAIRIQFRDEVHPYGEATQRNELVIRAQPSEAMYVKITTKVPGLSGDLRQTHQTELDLTYHTRYDVRLPDAYESLINDALLGNSTNFVRKDELDVAWRIFTPLLHQIDSGEIKPIPYQAGTRGPKEADEFIANNGFKHQKGYHWLPSNKL,baseline,654e404bc4e95e3baaa2030cfd21eaab,,7d084d8476f18d2e16fa884a456e06d2,,,562
7VMA_A,MRTILAGNGAFVLSDERGDMPSHYDGFYFLDTRFVRKARLEVSPEPDFIGASSTFTRAVSHFSLGERGILVRLRTLDGVYEEKLSFYNTSEESLGVKVRYSYEAPIEDIFQVRGFMGLKSGKAIAPAGGTHVKESPSGRRSLSIETNMEREGSLLRAELEIPPLGKAVLYVRFIPKIEGSISEILGEKRKTIKNVAFTGSPAIDGIFERAVENINALTLFTRFGPVPLAGIPYFACPFGRDAIIASLFLLPYYPEYAAGTLRLFGRLQGKRTNPKNEEEPGKIPHEFRLGELAQSGKVPFAPYYGTVDATPLYVALAGEYLRWTGDRKLIEELRPNLTAAVEWILKKLDDGYITYVPGILGNKGWKDSRDGIIDEEGKIPKPPIALVEVQGYTYWALKLAGELSLTDLDEKTLLAEAEKLKKRFNRDFWLGSYYALALDGEGRPLRVVSSNMGHLLLTGIAEHEEELAERLFRPDMFSRYGIRTLSAKEKAYNPFSYHRGSVWPHDNALIALGLARIGRTDMAKALMDAVFDAAKLLPERELPELYSGLNELVPVPRANSPQAWSSASVFAFVTASLGMEAGDELTVRPAEGTSIVLRGVSFGGRRYVVVVNGGVSVEPL,baseline,56abe32deb88a5c595ddcf0be5ad1505,,4992a06bdc591d588d21d29e19c77e2c,,,620
8BFI_B,GPDSMAADKPADQGAEKHEGTGQSSGITDQEKELSTNAFQAFTSGNYDACLQHLACLQDINKDDYKIILNTAVAEFFKSNQTTTDNLRQTLNQLKNQVHSAVEEMDGLDDVENSMLYYNQAVILYHLRQYTEAISVGEKLYQFIEPFEEKFAQAVCFLLVDLYILTYQAEKALHLLAVLEKMISQGNNNKNGKNETGNNNNKDGSNHKAESGALIEAAKSKIHQYKVRAYIQMKSLKACKREIKSVMNTAGNSAPSLFLKSNFEYLRGNYRKAVKLLNSSNIAEHPGFMKTGECLRCMFWNNLGCIHFAMSKHNLGIFYFKKALQENDNVCAQLSAGSTDPGKKFSGRPMCTLLTNKRYELLYNCGIQLLHIGRPLAAFECLIEAVQVYHANPRLWLRLAECCIAANKGTSEQETKGLPSKKGIVQSIVGQGYHRKIVLASQSIQNTVYNDGQSSAIPVASMEFAAICLRNALLLLPEEQQDPKQENGAKNSNQLGGNTESSESSETCSSKSHDGDKFIPAPPSSPLRKQELENLKCSILACSAYVALALGDNLMALNHADKLLQQPKLSGSLKFLGHLYAAEALISLDRISDAITHLNPENVTDVSLGISSNEQDQGSDKGENEAMESSGKRAPQCYPSSVNSARTVMLFNLGSAYCLRSEYDKARKCLHQAASMIHPKEVPPEAILLAVYLELQNGNTQLALQIIKRNQLLPAVKT,baseline,a5f0d5cd7b3494eae5482ffaf6201097,,14a1d86e5570f8a2cb65580ee0f82db4,,,718
7QE1_A,SATTPPGDLEQPELEARVKEIIEVDGYQFRDLNDNGELDPYEDWRLPTPERVADLVGQMSLVEKSGLMLINTLNAACDPQTGEFGVLPAQADNYINTQHMHRFVFRNVVDVRAEGVECTGTGTPVVSPAEAATFTNAVQEMSEATRLGIPSLFKSNARNHIDPDARVGINEAAGAFSAFPKEAGIAAAALGEQARRTGEATTGDMSVVADFADVMGEEWASIGLRGMYGYMADLSTEPRWYRTHETFTEDAYLAAEIMETLVQTLQGEELTDNGLALSPQTRVALTLKHFPGGGPQELGLDPHYAFGKAQVYPAGRFEEHFLPFQAAIDAGVSSIMPYYGVPVDVPVVGGEPGETYPHTGFAFSDSIVNGLLRDQLGFTGYVNSDTGIINDRAWGLEGNTVPERVAAAINGGTDTLSGFSDVSVITDLYEADLISEERIDLAAERLLEPLFDMGLFENPYVDPDVATATVGADDHRAVGLDLQRKSLVLLQNEETDEGPVLPLKEGGDVYILGDFTEETVESYGYEVTNGNVAEGEERPSAAGSDYVLISMTAKTNAGDYVSDDPSLGLNPDHGTNPSVIIGDDGEPLPGLDGQSLWGAADVCVHKEGHEENPSCTDNRLRFGGAYPWESSILDFTGMEAAESWEVVPSLETIQEVMAEVEDPSKVILHVYFRQPYVLDEESGLRDAGAILAGFGMTDTALMDVLTGAYAPQGKLPFALAGTREAIIEQDSDRPGYDETEDGALYPFGYGLTYEDDTEE,baseline,935e1c35b301b09f98ae656657290933,,dae145658fd12c72518faf09240f4989,,,759
8HHE_B,MGSSHHHHHHSSGLVPRGSHMASMTGGQQMGRDPNSMATINELYPVPYNVLAHPIKEVDDPYSWSNLLKGIQEGWEEWGKTGQKKLFEDHLTIAWNLYKTGKLDYFALTKASISLIGFIPGAEAAVPFINMFVDFVWPKLFGANTEGKDQQLFNAIMDAVNKMVDNKFLSYNLSTLNKTIEGLQGNLGLFQNAIQVAICQGSTPERVIFDQNCTPCNPNQPCKDDLDRVASRFDTANSQFTQHLPEFKNPWSDENSTQEFKRTSVELTLPMYTTVATLHLLLYKGYIEFMTKWNFHNEQYLNNLKVELQQLIHSYSETVRTSFLQFLPTLNNRSKSSVNAYNRYVRNMTVNCLDIAATWPTFDTHNYHQGGKLDLTRIILSDTAGPIEEYTTGDKTSGPEHSNITPNNILDTPSPTYQHSFVSVDSIVYSRKELQQLDIATYSTNNSNNCHPYGLRLSYTDGSRYDYGDNQPDFTTSNNNYCHNSYTAPITLVNARHLYNAKGSLQNVESLVVSTVNGGSGSCICDAWINYLRPPQTSKNESRPDQKINVLYPITETVNKGTGGNLGVISAYVPMELVPENVIGDVNADTKLPLTQLKGFPFEKYGSEYNNRGISLVREWINGNNAVKLSNSQSVGIQITNQTKQKYEIRCRYASKGDNNVYFNVDLSENPFRNSISFGSTESSVVGVQGENGKYILKSITTVEIPAGSFYVHITNQGSSDLFLDRIEFVPKIQFQFCDNNNLHCDCNNPVDTDCTFCCVCTSLTDCDCNNPRGLDCTLCCQVENQLPSFVTLTDLQNITTQVNALVA,baseline,137c7893dd3dad298a5bf1f60f195642,,d6598c7d000a21d9ad052ed74f7725e4,,,808
7Y4I_A,MVGLEDDTERERSPVVENGFSNGSRSSSSSAGVLSPSRKVTQGNDTLSYANILRARNKFADALALYEAMLEKDSKNVEAHIGKGICLQTQNKGNLAFDCFSEAIRLDPHNACALTHCGILHKEEGRLVEAAESYQKALMADASYKPAAECLAIVLTDLGTSLKLAGNTQEGIQKYYEALKIDPHYAPAYYNLGVVYSEMMQYDNALSCYEKAALERPMYAEAYCNMGVIYKNRGDLEMAITCYERCLAVSPNFEIAKNNMAIALTDLGTKVKLEGDVTQGVAYYKKALYYNWHYADAMYNLGVAYGEMLKFDMAIVFYELAFHFNPHCAEACNNLGVLYKDRDNLDKAVECYQMALSIKPNFAQSLNNLGVVYTVQGKMDAAASMIEKAILANPTYAEAFNNLGVLYRDAGNITMAIDAYEECLKIDPDSRNAGQNRLLAMNYINEGLDDKLFEAHRDWGWRFTRLHPQYTSWDNLKDPERPITIGYISPDFFTHSVSYFIEAPLTHHDYTKYKVVVYSAVVKADAKTYRFRDKVLKKGGVWKDIYGIDEKKIASMVREDKIDILVELTGHTANNKLGTMACRPAPVQVTWIGYPNTTGLPTVDYRITDSLADPPDTKQKQVEELVRLPDCFLCYTPSPEAGPVSPTPALSNGFVTFGSFNNLAKITPKVLQVWARILCAVPNSRLVVKCKPFCCDSIRQRFLTTLEQLGLESKRVDLLPLILFNHDHMQAYSLMDISLDTFPYAGTTTTCESLYMGVPCVTMAGSVHAHNVGVSLLTKVGLGHLVAKNEDEYVQLSVDLASDVTALSKLRMSLRDLMAGSPVCNGPSFAVGLESAYRNMWKKYCKGEVPSLRRMEMLQKEVHDDPLISKDLGPSRVSVTGEATPSLKANGSAPVPSSLPTQSPQLSKRMDSTS,baseline,9ee6226bdc547feb6b2c7573e049b9ad,,101a13cc66266e62caf2f2337503c557,,,914

The benchmarking script can be run using the following example command line, assuming that the script is saved as msa-benchmark.py, the input data is saved as msa-sequences.csv, and the --url argument to the script is the full path to the endpoint of the running NIM.

python msa-benchmark.py --input-csv msa-sequences.csv --url http://0.0.0.0:8000/biology/colabfold/msa-search/predict --out-dir results/ --output-yaml-name results.yaml

In general, a short test is sufficient to ensure baseline performance expectations are met. You can run a subset of the benchmark set by passing the --pdbs argument with a list of PDBs to benchmark. Below is a command running the first four PDB sequence entries in the file:

python msa-benchmark.py --input-csv msa-sequences.csv --url http://0.0.0.0:8000/biology/colabfold/msa-search/predict --out-dir results/ --output-yaml-name results.yaml --pdbs 7WJ0_A,7WBN_A,7OX1_Z,8BFH_A

In general, each of these four sequences should complete search against the three colabfold databases (Uniref30_2302, colabfold_envdb_202108, and PDB70_220313) in roughly 20 seconds on a single NVIDIA A100 GPU (total runtime for four sequences: approximately 80 seconds). This is considered a baseline for performance and accuracy because it does the following:

  1. Demonstrates that the NIM is meeting performance expectations of MMSeqs2

  2. Produces resulting alignments that are identical to those that could generate from GPU-accelerated MMSeqs2 from outside the NIM.

On this page