# How to Benchmark Protein Structure Prediction Tools After working through this guide you will be able to design and run a fair, reproducible comparison of [protein structure](/blog/guides/protein-structure) prediction tools on your own targets, score every model with standard accuracy metrics, and report results with [confidence intervals](/blog/data-analysis/confidence-interval-formula-examples) that survive review. The walkthrough compares AlphaFold2 (run through ColabFold) against ESMFold, and the same harness accepts Chai-1, Boltz, or RoseTTAFold variants without changing the scoring code. You need a Linux machine with an NVIDIA GPU (16 GB of memory covers most single-chain targets), Conda or Mamba, Python 3.10, MMseqs2, a TM-score program, and enough disk for the MSA databases, which range from several hundred gigabytes to more than a terabyte depending on what you install. The reason to benchmark rather than trust vendor demos is simple. Predicted structures now feed directly into downstream pipelines such as structural similarity search and function annotation, and one such pipeline built on AlphaFold-predicted structures reached above 90 percent annotation accuracy on a curated bacterial test set while showing that sequence-based and structure-based evidence complement each other [1]. Structure-guided design workflows likewise assume the starting model is appropriate for the task [2]. Which predictor deserves that trust depends on your protein families, your alignment depth, and your definition of success. ## What a Fair Comparison Requires A benchmark is a controlled experiment. You hold four things fixed: the test structures, the inputs, the metrics, and the hardware. You then change one tool at a time and attribute every difference in score to that tool. Break any of the four and the comparison measures your protocol instead of the software. The tools fall into three broad families. [Multiple sequence alignment](/blog/guides/multiple-sequence-alignment) (MSA) based methods such as AlphaFold2 and RoseTTAFold infer residue contacts from co-evolutionary signals across homologous sequences. Protein language model (PLM) methods such as ESMFold fold from a single sequence using representations learned across millions of sequences. Newer diffusion-based systems such as AlphaFold3, Chai-1, and Boltz model complexes that can include ligands and [nucleic acids](/blog/guides/nucleic-acids). A unified evaluation of 13 models spanning MSA-based, PLM-based, and docking-based approaches on TCR-pMHC structure prediction found MSA-based methods, AlphaFold3 in particular, ahead on that task [3]. Category leadership does not transfer automatically between tasks, which is exactly why you benchmark on your own targets rather than trusting someone else's headline number. Downstream fit matters as much as raw accuracy. Studies that integrated multiple specialized prediction tools found that a well-designed combination beats every individual tool, with integration strategy itself a decisive factor [4], and that combining standard with alternative analysis tools can reveal biology that any single method misses [5]. A bioinformatics tools comparison is only as good as its test set, so most of the effort below goes into building one. ## Choose Scope and Metrics Before Running Anything ![Calculation card showing RMSD equals the square root of 57.5 over 76, giving 0.87 angstroms for the 76 matched C-alpha pairs of ubiquitin.](/article-images/illustrations/benchmark-protein-structure-prediction-tools-rmsd-worked-example.webp) Decide the scope first, because it determines everything downstream: - Single chains or complexes. Complexes need multimer-capable tools and interface metrics. - Full chains or domains. Flexible full chains punish global metrics, domain-level scoring is kinder and often more informative. - A broad sample or your family of interest. Family-specific results are cheaper and more relevant to your pipeline. - Templates on or off. Structural templates are known structures fed to the predictor as hints. They help accuracy but can leak your test set back into the tool. Speed, accuracy, and scale always trade against each other, so mirror your production conditions in the benchmark rather than an idealized setting [6]. The standard metrics, all computed on Cα atoms unless stated otherwise: | Metric | Scale | What it tells you | Formula or notes | |--|--|--|--| | RMSD | Å, lower is better | Mean deviation after optimal rigid superposition | Square root of the mean squared distance between matched atoms. Outlier sensitive, so always state which residues were matched | | TM-score | 0 to 1, higher is better | Global fold agreement, length normalized | TM = (1/L) Σ 1/(1+(di/d0)²) with d0 = 1.24(L-15)^(1/3) - 1.8. Above 0.5 usually indicates the same fold | | GDT_TS | 0 to 100 | Fraction of Cα atoms within 1, 2, 4, and 8 Å | Average of the GD1, GD2, GD4, GD8 percentages. The CASP assessment standard | | lDDT | 0 to 1 | Local distance agreement without superposition | Fraction of atom-pair distances preserved within tolerances. Preferred for flexible or multi-domain targets | | pLDDT | 0 to 100 | Per-residue model confidence emitted by the tool | A confidence estimate, not accuracy. Calibrate it against real error on your own set | | PAE / ipTM | Å / 0 to 1 | Expected error between residue pairs, interface confidence | Read these before trusting domain or chain placement in complexes | | DockQ | 0 to 1 | Protein-protein interface quality | Combines interface RMSD, ligand RMSD, and contact recovery | A worked RMSD calculation makes the numbers concrete. If the 76 matched Cα pairs of ubiquitin contribute a total squared deviation of 57.5 Ų, then RMSD = sqrt(57.5 / 76) = 0.87 Å, which is excellent for a 76-residue domain. Where an established benchmark matches your task, reuse it. Affinity modeling leans on the versioned PDBbind sets v2007, v2013, and v2016 [7]. Binding-pocket prediction uses LIGYSIS and CryptoBench [8]. Lipid-protein pose prediction now has LiPP, a curated set of 311 nonredundant nonannular lipid-protein complex structures [9]. Mutation-stability modeling uses strictly non-redundant blind sets such as S669 alongside training data [10]. For structure prediction on your own targets you will usually build a custom set, but copy the design principles these sets embody: nonredundancy, unseen status relative to the tools, and documented selection rules. ## Build a Leakage-Free Test Set ![Decision path checking a candidate PDB entry against the tool training cutoff, then clustering at 30 percent identity, ending in keep or drop.](/article-images/illustrations/benchmark-protein-structure-prediction-tools-leakage-cutoff-check.webp) Data leakage means any overlap between what a tool saw during training and what you test on. It is the single most common reason published numbers fail to reproduce. A study of mRNA-protein interaction prediction showed that overlapping proteins between training and test sets substantially inflate performance metrics, and compared a random interaction-level split against an RBP-aware split that removes all test proteins from training [11]. The effect is not hypothetical for structure prediction either. On the LiPP lipid benchmark, AlphaFold 3 succeeded on 76.1 percent of poses under the RMSD < 2 Å criterion overall, but only 47.2 percent on structures not seen during training [9]. That gap is memorization showing up as accuracy. Benchmark builders respond by testing only on unseen targets, as the TCR-pMHC evaluation did with 70 previously unseen complexes [3]. Build your set like this: 1. List candidate PDB entries for your targets. PDBminer automates the search across the PDB and the AlphaFold Database and returns a quality-ranked table with coverage aligned to the UniProt sequence, plus annotations for complexes, ligands, and nucleic acid chains [12]. 2. Record the PDB release date for every entry and drop anything released after the training cutoff of the tool you are testing. AlphaFold2, for example, trained on structures released before May 2018. Newer tools have later cutoffs, so check each tool's publication or model card. 3. Cluster the target sequences at 30 percent identity or lower so no two targets are close homologs. 4. Keep one representative per cluster. 5. Save a manifest with PDB ID, chain, UniProt accession, release date, experimental method, and resolution. Every number in your final report should trace back to a row in this file. ## Benchmark Protein Structure Prediction Tools Step by Step The following walkthrough produces a two-tool comparison on a small single-chain set. Scale the same steps to hundreds of targets by wrapping them in a script. ### Step 1: Set Up the Environment Keep the benchmark in its own environment so library versions cannot drift between runs. ```bash mamba create -n bench -c conda-forge -c bioconda python=3.10 mmseqs2 mamba activate bench pip install numpy scipy biopython ``` Install ColabFold following the installation instructions in its repository, then verify: ```bash colabfold_batch --help | head -n 20 ``` Expected output resembles: ``` usage: colabfold_batch [--num-models] [--num-recycle] ... positional arguments: input Path to directory containing MSA files, or a FASTA file output Output directory ``` For ESMFold: ```bash pip install "fair-esm[esmfold]" ``` ESMFold also requires OpenFold and a logging dependency, so follow the esm repository's model installation notes exactly. Verify the import path without triggering a download: ```bash python -c "import esm; print(esm.pretrained.esmfold_v1)" ``` Download the TMscore program from the Zhang Group TM-score page and place it in `tools/`. ### Step 2: Assemble Targets and Native Structures Create the directory layout and a list of PDB IDs: ```bash mkdir -p bench/{fasta,native,predictions,results,tools,tmp} cd bench printf "1UBQ\n1TEN\n" > pdb_ids.txt ``` Download the experimental structures that serve as ground truth, meaning the experimentally determined reference you score against: ```bash while read -r pdb; do curl -s "https://files.rcsb.org/download/${pdb}.pdb" -o "native/${pdb}.pdb" done < pdb_ids.txt ls -lh native | head ``` Expected output resembles: ``` -rw-r--r-- 1 user user 98K native/1UBQ.pdb -rw-r--r-- 1 user user 61K native/1TEN.pdb ``` Build the input FASTA from the sequences in those entries. One record per target, with the header matching the PDB ID: ``` >1UBQ MQIFVKTLTGKTITLEVEPSDTIENVKAKIQDKEGIPPDQQRLIFAGKQLEDGRTLSDYNIQKESTLHLVLRLRGG >1TEN ``` Before committing to a structure, check it. PDBminer ranks available structures by quality and reports coverage against the UniProt sequence, which prevents you from benchmarking against a fragment by accident [12]. For a quick interactive look at contacts, interfaces, or sequence-to-structure residue mapping in a candidate PDB file, the iMolTalk server accepts a four-character PDB identifier and returns graphical and text summaries [13]. ### Step 3: Cluster the Targets for Nonredundancy Nonredundant means no two targets are close homologs, so each one contributes independent information. Cluster at 30 percent sequence identity with 80 percent coverage of both sequences: ```bash mmseqs easy-linclust fasta/targets.fasta result tmp \ --min-seq-id 0.3 -c 0.8 --cov-mode 0 head -n 3 result_cluster.tsv grep -c '>' result_rep_seq.fasta ``` Expected output resembles: ``` 1UBQ 1UBQ 1UBQ 1QES 1TEN 1TEN 42 ``` The cluster file lists each representative next to its members. The representative FASTA contains one sequence per cluster, and that file becomes the benchmark input. A common mistake here is forgetting that the representative appears as a member of its own cluster, which is normal. ### Step 4: Fix the Protocol and Its Parameters Write the protocol down before running anything, then apply it identically to every tool. For ColabFold, these flags matter most: | ColabFold flag | Effect | Benchmark guidance | |--|--|--| | `-model-type` | Selects the weight set | `alphafold2_ptm` for single chains, `alphafold2_multimer_v3` for complexes | | `-num-models` | How many parameter sets run | Keep 5 so the ranking is not one-seed luck | | `-num-recycle` | Refinement passes through the network | Fix at 3 for every tool in the comparison | | `-msa-mode` | Where the MSA comes from | `mmseqs2_uniref_env` for the standard pipeline | | `-templates` | Use structural templates from the PDB | Off for headline numbers, on in a separate run to measure template benefit | | `-random-seed` | Fixes stochastic choices | Same seed for every target and tool | | `-amber` | Relax the raw model with Amber energy minimization | Optional, but apply the same choice to all tools and say which you used | The template flag deserves emphasis. Templates are drawn from the PDB, so a template database can contain your exact test structure. Run the headline comparison with templates disabled, then a second pass with templates enabled if template benefit is a question you care about. ### Step 5: Run the MSA-Based Tool ```bash colabfold_batch \ --model-type alphafold2_ptm \ --num-models 5 \ --num-recycle 3 \ --msa-mode mmseqs2_uniref_env \ --random-seed 42 \ fasta/nr_targets.fasta predictions/colabfold ``` Expected log lines resemble: ``` rank_001 alphafold2_ptm_model_1_seed_000 pLDDT=94.7 ptm=0.962 rank_002 alphafold2_ptm_model_2_seed_000 pLDDT=93.9 ptm=0.958 ``` Expected files: ``` predictions/colabfold/1UBQ_unrelaxed_rank_001_alphafold2_ptm_model_1_seed_000.pdb predictions/colabfold/1UBQ_scores_rank_001_alphafold2_ptm_model_1_seed_000.json predictions/colabfold/1UBQ.a3m ``` The JSON carries per-residue pLDDT and the predicted aligned error (PAE), the expected positional error between residue pairs. The a3m file holds the MSA and becomes useful in Step 8. Score the rank_001 model for every target, or score all five models and report the best and the median, but pick one rule and apply it everywhere. ### Step 6: Run the Sequence-Model Tool ESMFold needs no MSA, which is the property you are testing when you include it: ```python import esm import torch model = esm.pretrained.esmfold_v1() model = model.eval().cuda() records = [] with open("fasta/nr_targets.fasta") as fh: lines = [l.strip() for l in fh if l.strip()] for header, seq in zip(lines[0::2], lines[1::2]): records.append((header[1:].split(), seq)) for name, seq in records: with torch.no_grad(): pdb_string = model.infer_pdb(seq) with open(f"predictions/esmfold/{name}.pdb", "w") as out: out.write(pdb_string) print(name, len(seq), "done") ``` Expected output: ``` 1UBQ 76 done 1TEN 89 done ``` This snippet assumes a strict two-line FASTA. In the output PDB, the B-factor column carries the pLDDT value, same convention as AlphaFold2, which makes downstream calibration straightforward. ### Step 7: Score Every Model Score with TM-score, running the native structure first so the score is normalized by native length, the convention used in CASP assessments: ```bash tools/TMscore \ native/1UBQ.pdb \ predictions/colabfold/1UBQ_unrelaxed_rank_001_alphafold2_ptm_model_1_seed_000.pdb ``` Expected output resembles: ``` TM-score = 0.9452 (normalized by length of the Structure_1: P1_length= 76, d0= 4.32) RMSD of the common residues= 0.871 ``` Compute RMSD yourself with the Kabsch algorithm, the standard optimal rigid superposition: ```python import csv import glob import numpy as np def ca_coords(path, chain="A"): coords = [] with open(path) as fh: for line in fh: if line.startswith("ATOM") and line[12:16].strip() == "CA" and line == chain: coords.append([float(line[30:38]), float(line[38:46]), float(line[46:54])]) return np.asarray(coords) def kabsch_rmsd(P, Q): P = P - P.mean(axis=0) Q = Q - Q.mean(axis=0) V, S, Wt = np.linalg.svd(P.T @ Q) d = np.sign(np.linalg.det(V @ Wt)) R = V @ np.diag([1.0, 1.0, d]) @ Wt return float(np.sqrt(((P @ R - Q) ** 2).sum(axis=1).mean())) rows = [] for pred in sorted(glob.glob("predictions/colabfold/*rank_001*.pdb")): name = pred.split("/")[-1].split("_") P = ca_coords(pred) Q = ca_coords(f"native/{name}.pdb") n = min(len(P), len(Q)) rows.append((name, len(Q), n, kabsch_rmsd(P[:n], Q[:n]))) with open("results/colabfold_rmsd.csv", "w", newline="") as fh: writer = csv.writer(fh) writer.writerow(["target", "native_len", "matched", "rmsd"]) writer.writerows(rows) ``` Expected CSV: ``` target,native_len,matched,rmsd 1UBQ,76,76,0.87 1TEN,89,89,1.42 ``` The truncation to the shorter length is a quick-look shortcut. For the real benchmark, match residues by number and chain, or use the UniProt-aligned coverage table from PDBminer, because index truncation silently misaligns targets with gaps or tag residues [12]. Add lDDT for flexible targets using an established implementation rather than a homegrown one, since the tolerance definitions are easy to get subtly wrong. Three checks confirm the scoring code is right. Score any prediction against itself, which must return a TM-score of 1.0000 and RMSD of 0.000. Rerun one target with the same seed and confirm identical pLDDT. Open one model next to its native in a molecular viewer and confirm the visual impression matches the numbers. ### Step 8: Compare With Paired Statistics Because both tools predict the same targets, the comparison is paired, and paired tests are far more sensitive than comparing two sets of averages: ```python import numpy as np from scipy.stats import wilcoxon tm_af2 = np.array([0.95, 0.88, 0.61, 0.42, 0.90]) tm_esm = np.array([0.93, 0.90, 0.55, 0.38, 0.89]) stat, p = wilcoxon(tm_af2, tm_esm) print("Wilcoxon signed-rank p =", p) rng = np.random.default_rng(42) diffs = tm_af2 - tm_esm boot = np.array([rng.choice(diffs, size=len(diffs)).mean() for _ in range(10000)]) print("95% CI for mean TM-score difference:", np.percentile(boot, [2.5, 97.5])) ``` Expected output with these illustrative five-target arrays: ``` Wilcoxon signed-rank p = 0.0625 95% CI for mean TM-score difference: [-0.004 0.056] ``` Replace the arrays with your real per-target values. With five targets the smallest possible Wilcoxon p-value is 0.0625, which is itself a lesson: small test sets cannot produce decisive statistics, so grow the set until confidence intervals are narrow enough to act on. Stratify before you conclude. MSA depth is the biggest confounder between MSA-based and language-model tools, and ColabFold saves the alignment for every target: ```bash grep -c '^>' predictions/colabfold/1UBQ.a3m ``` Expected output: ``` 21453 ``` Bin targets into shallow, medium, and deep alignments and compare tools within bins. A tool that wins only on deep-MSA targets will lose on the orphan proteins you probably care about. ### Step 9: Log Runtime and Write the Report Time every tool on identical hardware: ```bash /usr/bin/time -v colabfold_batch --num-models 1 --num-recycle 3 \ --msa-mode mmseqs2_uniref_env fasta/nr_targets.fasta predictions/colabfold_timed \ 2>&1 | tee results/colabfold_time.log nvidia-smi --query-gpu=name,memory.total --format=csv ``` Expected output resembles: ``` Command being timed: "colabfold_batch ..." Elapsed (wall clock) time (m:ss): 4:12 ``` ``` name, memory.total [MiB] NVIDIA A100-SXM4-40GB, 40960 ``` The report should contain: tool names with version numbers or commit hashes, database versions, the template and MSA policy, hardware, run dates, the manifest, per-target results, medians with interquartile ranges, success rates at fixed thresholds, the count of failed runs for each tool, and the exact commands. A reviewer should be able to rerun your benchmark from the report alone. ### When Something Breaks - `torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB` means the model or the batch does not fit on the GPU. Run one model at a time with `-num-models 1`, reduce recycles, split very long targets, or move to a larger GPU. Multimer jobs need substantially more memory than monomers. - MMseqs2 exits with a message like `File does not seem to be valid fasta` when the input has blank lines or malformed headers. Ensure every record starts with `>` and remove duplicate sequence names. - `curl: (6) Could not resolve host: files.rcsb.org` is a DNS or proxy failure. Retry, or configure proxy environment variables on institutional clusters. - `FileNotFoundError: [Errno 2] No such file or directory: 'native/1UBQ.pdb'` means a download was skipped or a path is misspelled. Check `pdb_ids.txt` against the files on disk. - A TM-score near zero with a tiny common-residue count means a chain ID or residue numbering mismatch. Extract the matching chain from both files before scoring. - ColabFold stops immediately complaining about a missing database when the UniRef30 or template databases are not configured. Redo the database setup from the ColabFold documentation and record the database version in your report. ## Compare Results Without Fooling Yourself Report medians and interquartile ranges, not just means, and report success rates at fixed thresholds so results are comparable across papers. The LiPP benchmark used the RMSD < 2 Å criterion for lipid poses [9], and the PMGen evaluation reported median peptide-core Cα RMSD per MHC class, 0.62 Å for class I on its benchmark [14]. Both conventions are fine, just state yours explicitly. Calibrate confidence scores before trusting them. A two-part benchmark simulating validation of GPCR-peptide and GPCR-protein complexes with AlphaFold2 Initial Guess, Boltz-2, and RoseTTAFold3 found that current pipelines suffer significant confidence overestimation for misplaced peptides across all three tools [15]. Within the AlphaFold family, confidence can still be genuinely informative when used carefully, as CDR3-region pLDDT enabled reranking gains of up to 4.3 percent Top-1 success in TCR-pMHC prediction [3]. The rule for your benchmark: plot pLDDT against actual per-residue error for each tool separately, and never rank tools by their self-reported confidence. Look for complementarity, because it changes the recommendation. Integrating multiple subcellular localization tools reached up to 92 percent accuracy, far above any individual tool, with the integration method itself decisive [4]. Consensus between standard and alternative analyses has repeatedly surfaced biology that single methods miss [5], and sequence-based plus structure-based annotation outperforms either alone [1]. Report per-target wins, not only averages, and consider whether an ensemble serves your pipeline better than a single winner. When a wet-lab readout exists, prefer it. TCR-pHLA binding models gained 19.2 percent over state-of-the-art tools when evaluated on an external wet-lab test set rather than held-out benchmark data alone [16]. Broader machine-learning practice in proteomics stresses the same triad of uncertainty quantification, calibration, and experimental validation [17]. If you benchmark a derived screening score rather than a predictor, build decoys carefully and test on complexes unseen during development, the design that let PPIscreenML demonstrate generalization beyond its training data when compared against pDockQ and iPTM [18]. ## Runtime, Hardware, and Practical Trade-Offs MSA generation dominates the wall-clock time of MSA-based tools, so generate alignments once, cache them, and reuse them across tools that accept a shared MSA. Language-model tools fold a target in seconds to minutes on a GPU, but their memory appetite grows with sequence length. The speed-accuracy-scale trade-off is structural, not a bug, so the best bioinformatics software for your pipeline is the one that wins under your production constraints, not on someone else's leaderboard [6]. Wrap every tool behind one harness that takes the same FASTA and emits the same results table. This pattern, a unified interface that lets you swap methods while holding inputs constant, has a long history in bioinformatics tool integration and is what turns a pile of one-off runs into a benchmark. Pin local versions for the benchmark itself, because web servers change versions without notice and queue effects distort timing. Check the license terms of model weights before using any tool in production, since some newer models restrict commercial use. ## Common Mistakes and Limitations 1. Training-set contamination. If test structures were released after your tool's cutoff or share close homologs with its training data, your numbers measure memory. The random-versus-aware split comparison in interaction prediction [11] and the drop from 76.1 to 47.2 percent on unseen structures in the LiPP benchmark [9] show how large the effect is. 2. Template leakage. Templates come from the PDB and can include your test structure itself. Disable templates for headline numbers. 3. Comparing pLDDT across tools as if interchangeable. Each model calibrates its own confidence, and overestimation has been documented across multiple tools [15]. 4. MSA depth confounding. A language-model tool beaten overall may win on shallow-MSA targets. Stratify or the conclusion is wrong for exactly the proteins that matter most. 5. Tiny or cherry-picked test sets. Five targets cannot produce a decisive p-value. Report confidence intervals and grow the set until they are narrow. 6. Sloppy residue matching. Truncating both coordinate arrays to the shorter length misaligns targets with gaps, tags, or alternate chains. Match by residue number, or use UniProt-aligned coverage mapping [12]. 7. Single-metric conclusions. RMSD punishes one flexible tail across the whole chain, TM-score describes the fold, lDDT describes local geometry. Use at least two and say which drove your decision. 8. Ignoring ground-truth quality. X-ray structures below 2.5 Å resolution and cryo-EM reconstructions carry their own errors, and AI-assisted cryo-EM model building is itself an active, benchmarked research area [19]. A prediction cannot be scored more precisely than the reference. 9. Silent failures. A tool that crashes on 20 percent of targets and wins on the rest is worse than its average suggests. Count and report every failure. 10. Overgeneralizing from family-specific benchmarks. Results on pMHC [14], TCR-pMHC [3], and lipid complexes [9] describe those problems, not protein folding in general. 11. Mistaking agreement with experiment for biological truth. An X-ray structure is one frozen conformation, disordered regions score poorly for every tool, and a low score on a flexible loop may describe the protein, not the predictor. ## Frequently Asked Questions ### What metrics should I report when I benchmark structure prediction tools? Report at least one global fold metric such as TM-score, one superposition-free metric such as lDDT, and RMSD over an explicitly stated matched residue set, plus success rates at fixed thresholds. If the tools emit confidence scores, add a calibration plot of predicted confidence against actual error. ### How do I prevent data leakage in my benchmark set? Check each tool's training cutoff, keep only structures released before it, cluster targets at 30 percent sequence identity or lower, and keep one representative per cluster. Record release dates in your manifest so the exclusion rule is auditable. ### How many test structures do I need for a meaningful comparison? Enough that medians and bootstrap confidence intervals are stable, which in practice means dozens to a few hundred targets stratified across lengths, fold classes, and MSA depths. Very small sets produce rankings that flip on rerun. ### Is pLDDT comparable across different prediction tools? No. pLDDT is calibrated per model, and confidence overestimation has been documented across several tools. Use it within a tool, calibrate it against real error on your own set, and never rank tools by confidence alone. ### Do all structure prediction tools need an MSA? No. Language-model methods such as ESMFold fold from a single sequence, while MSA-based methods generally win when deep alignments exist. Record MSA depth for every target so you can see which regime each tool owns. ### How do I benchmark tools on protein complexes instead of single chains? Use multimer-capable tools, keep paired inputs and MSA policy identical across tools, and add interface metrics such as DockQ or ipTM to chain-level metrics. A chain-level score can look perfect while the interface is wrong. ### Can I benchmark web servers the same way as local installations? Yes, with caveats. Pin the server version and access date, expect queue effects in any timing measurement, and confirm key results on a local installation before drawing conclusions you will act on. ### Should I reuse an existing benchmark set or build my own? Reuse an established set when it matches your task, because community scrutiny makes those results comparable across papers. Build your own when your targets have special properties, and follow the same design rules either way: nonredundancy, unseen status, and documented selection. ## Sources 1. [ProtPen Combines Sequence- and Structure-based Approaches to Facilitate Protein Function Predictions on a Proteome-wide Scale.](https://pubmed.ncbi.nlm.nih.gov/42695892/) 2. [In silico protein design: fitting sequence onto structure.](https://pubmed.ncbi.nlm.nih.gov/16671409/) 3. [Benchmarking TCR-pMHC structure prediction: a unified evaluation and CDR3-based functional insights.](https://pubmed.ncbi.nlm.nih.gov/42248582/) 4. ['Unite and conquer': enhanced prediction of protein subcellular localization by integrating multiple specialized tools.](https://pubmed.ncbi.nlm.nih.gov/17967180/) 5. [A comparative computational analysis of protein sequences and literature mining classify 'orphan' neurotransmitter transporters.](https://pubmed.ncbi.nlm.nih.gov/18621060/) 6. [In silico protein motif discovery and structural analysis.](https://pubmed.ncbi.nlm.nih.gov/21780007/) 7. [Persistent Local Laplacian Prediction of Protein-Ligand Binding Affinities.](https://pubmed.ncbi.nlm.nih.gov/42906032/) 8. [Seq2Pocket: augmenting protein language models for spatially consistent binding site prediction.](https://pubmed.ncbi.nlm.nih.gov/42886502/) 9. [The LiPP Benchmark Set for Modeling Lipid-Protein Complexes: Comparison of Co-Folding and Docking Methods.](https://pubmed.ncbi.nlm.nih.gov/42268396/) 10. [Persistent sheaf Laplacian analysis of protein stability and solubility changes upon mutation.](https://pubmed.ncbi.nlm.nih.gov/42428996/) 11. [Generalizable deep-learning-based mRNA-protein interaction prediction strongly depends on protein diversity.](https://pubmed.ncbi.nlm.nih.gov/42015301/) 12. [PDBminer to Find and Annotate Protein Structures for Computational Analysis.](https://pubmed.ncbi.nlm.nih.gov/37977136/) 13. [iMolTalk: an interactive, internet-based protein structure analysis server.](https://pubmed.ncbi.nlm.nih.gov/15215439/) 14. [PMGen: from peptide-MHC structure prediction to peptide generation.](https://pubmed.ncbi.nlm.nih.gov/42289978/) 15. [Assessment of generative de novo peptide design methods for G protein-coupled receptors.](https://pubmed.ncbi.nlm.nih.gov/42658811/) 16. [Machine learning dataset and benchmark for accurate T cell receptor-pHLA binding prediction.](https://pubmed.ncbi.nlm.nih.gov/42934405/) 17. [AI and Machine Learning for Proteomics-Driven Drug Discovery: Methods, Tools, and Best Practices.](https://pubmed.ncbi.nlm.nih.gov/42193136/) 18. [PPIscreenML is a method for structure-based screening of protein-protein interactions using AlphaFold.](https://pubmed.ncbi.nlm.nih.gov/42012336/) 19. [AI tools for cryo-EM: Protein particle picking, density map enhancement, and atomic model building.](https://pubmed.ncbi.nlm.nih.gov/42463260/)