{"slug": "from-in-silico-to-wet-lab-evaluating-ai-protein-design-performance", "title": "From In-Silico to Wet-Lab: Evaluating AI Protein Design Performance", "summary": "Anthropic's claude-protein-binder-design dataset, containing 1,440 AI-designed miniprotein binders tested against 16 targets, was used in a tutorial to evaluate AI protein design performance, finding that structure predictors and combined predictions can help identify successful binders, with wet-lab results from two independent labs providing ground truth.", "body_md": "In this tutorial, we use [ Anthropic’s claude-protein-binder-design](https://huggingface.co/datasets/Anthropic/claude-protein-binder-design) dataset, which contains 1,440 AI-designed miniprotein binders tested against 16 targets. Because the release includes both computational predictions and real wet-lab results from two independent labs, we can go beyond simply studying the designs. We evaluate how well structure predictors identify successful binders, whether combining predictions improves performance, how rankings translate into practical testing budgets, and how much disagreement comes from the assays themselves. Also, we train a target-aware classifier to test whether these signals can reliably predict experimental success.\n\n``` python\nimport subprocess, sys, warnings, itertools, math\nwarnings.filterwarnings(\"ignore\")\nimport importlib.util\n_needed = {\"huggingface_hub\": \"huggingface_hub>=0.24\", \"pyarrow\": \"pyarrow\",\n          \"pandas\": \"pandas\", \"sklearn\": \"scikit-learn\",\n          \"matplotlib\": \"matplotlib\", \"scipy\": \"scipy\"}\n_missing = [pkg for mod, pkg in _needed.items() if importlib.util.find_spec(mod) is None]\nif _missing:\n   print(\"installing:\", \", \".join(_missing))\n   subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", *_missing], check=False)\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom scipy import stats\nfrom huggingface_hub import HfApi, hf_hub_download\nfrom sklearn.metrics import roc_auc_score, cohen_kappa_score, average_precision_score\nfrom sklearn.model_selection import GroupKFold, StratifiedKFold\nfrom sklearn.ensemble import HistGradientBoostingClassifier\nfrom sklearn.inspection import permutation_importance\nSEED = 0\nrng_global = np.random.default_rng(SEED)\npd.set_option(\"display.width\", 200)\npd.set_option(\"display.max_columns\", 100)\nplt.rcParams.update({\"figure.dpi\": 110, \"font.size\": 9, \"axes.grid\": True,\n                    \"grid.alpha\": 0.25, \"axes.spines.top\": False, \"axes.spines.right\": False})\nREPO = \"Anthropic/claude-protein-binder-design\"\nBAR = \"=\" * 78\ndef head(n, title):\n   prefix = f\"{n}. \" if str(n) else \"\"\n   print(f\"\\n{BAR}\\n  {prefix}{title}\\n{BAR}\")\nhead(1, \"TABLE DISCOVERY\")\napi = HfApi()\nrepo_files = api.list_repo_files(REPO, repo_type=\"dataset\")\nTABLES = {}\nfor f in repo_files:\n   if f.startswith(\"data/tables/\") and f.endswith(\".parquet\"):\n       key = f[len(\"data/tables/\"): -len(\".parquet\")].replace(\"/\", \"_\")\n       TABLES[key] = f\nprint(f\"Found {len(TABLES)} Parquet tables:\")\nfor k in sorted(TABLES):\n   print(f\"   - {k:38s} {TABLES[k]}\")\ndef load_table(name: str) -> pd.DataFrame:\n   \"\"\"Load a subset by its viewer name, with a datasets-library fallback.\"\"\"\n   if name in TABLES:\n       return pd.read_parquet(hf_hub_download(REPO, TABLES[name], repo_type=\"dataset\"))\n   from datasets import load_dataset\n   return load_dataset(REPO, name, split=\"full\").to_pandas()\nds = load_table(\"design_summary\")\nprint(f\"\\ndesign_summary: {ds.shape[0]:,} rows x {ds.shape[1]} columns\")\n```\n\nWe start by installing only what the runtime is actually missing, then enumerate the repository once and build a {subset to path} map instead of hard-coding file locations. This matters because the naming is not uniform; the subset wetlab_summary lives at data/tables/wetlab/summary.parquet, and a guessed path would fail silently. With the map in place we pull design_summary, one row per design, 1,440 rows wide enough to carry every join we need downstream.\n\n```\nhead(2, \"SCHEMA + EVALUABLE SET\")\nCALLS = {\"binder\", \"non_binder\"}\ntested = ds[\"adaptyv_binding\"].isin(CALLS) | ds[\"twist_binding\"].isin(CALLS)\nev = ds[tested].copy()\nev[\"y\"] = ev[\"binder_final\"].astype(int)\nprint(f\"All designs               : {len(ds):,}\")\nprint(f\"Evaluable (>=1 vendor call): {len(ev):,}\")\nprint(f\"Confirmed binders          : {int(ev['y'].sum()):,}  \"\n     f\"({100 * ev['y'].mean():.1f}% base rate)\")\nprint(f\"Never measured             : {len(ds) - len(ev):,}\")\nprint(\"\\nCategorical levels:\")\nfor c in [\"design_model\", \"campaign\", \"generator\", \"sequence_design_method\", \"vendor_agreement\"]:\n   vals = ds[c].astype(str).value_counts()\n   print(f\"  {c:24s} ({len(vals)}): {', '.join(vals.index[:6])}\"\n         + (\" ...\" if len(vals) > 6 else \"\"))\nprint(f\"\\nTargets ({ds['target'].nunique()}): {', '.join(sorted(ds['target'].unique()))}\")\nprint(f\"Binder length: {ds.binder_length.min()}-{ds.binder_length.max()} aa \"\n     f\"(median {ds.binder_length.median():.0f})\")\nhead(3, \"HIT-RATE LANDSCAPE\")\ndef wilson(k, n, z=1.96):\n   if n == 0:\n       return (np.nan, np.nan, np.nan)\n   p = k / n\n   d = 1 + z**2 / n\n   c = (p + z**2 / (2 * n)) / d\n   h = z * math.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / d\n   return p, max(0.0, c - h), min(1.0, c + h)\ndef rate_table(df, by):\n   rows = []\n   for key, g in df.groupby(by, dropna=False):\n       p, lo, hi = wilson(int(g.y.sum()), len(g))\n       rows.append({by: key, \"n\": len(g), \"hits\": int(g.y.sum()),\n                    \"rate\": p, \"lo\": lo, \"hi\": hi})\n   return pd.DataFrame(rows).sort_values(\"rate\", ascending=False).reset_index(drop=True)\nfor dim in [\"design_model\", \"campaign\", \"generator\", \"sequence_design_method\"]:\n   t = rate_table(ev, dim)\n   print(f\"\\n--- hit rate by {dim} ---\")\n   print(t.to_string(index=False,\n                     formatters={\"rate\": \"{:.3f}\".format, \"lo\": \"{:.3f}\".format, \"hi\": \"{:.3f}\".format}))\ntt = rate_table(ev, \"target\")\nfig, ax = plt.subplots(figsize=(9, 4.2))\nax.bar(tt.target, tt.rate, color=\"#4C72B0\")\nax.errorbar(tt.target, tt.rate,\n           yerr=[(tt.rate - tt.lo).clip(lower=0), (tt.hi - tt.rate).clip(lower=0)],\n           fmt=\"none\", ecolor=\"0.25\", capsize=3, lw=1)\nax.axhline(ev.y.mean(), ls=\"--\", c=\"crimson\", lw=1, label=f\"pooled {ev.y.mean():.2f}\")\nax.set_ylabel(\"experimental hit rate\"); ax.set_title(\"Hit rate by target (Wilson 95% CI)\")\nax.tick_params(axis=\"x\", rotation=55); ax.legend(); plt.tight_layout(); plt.show()\nprint(\"\\nRead this plot as the dominant effect size in the dataset: target choice \"\n     \"swamps generator choice. Any model comparison that does not stratify by \"\n     \"target is mostly measuring which targets that model was pointed at.\")\n```\n\nWe define the evaluable set by filtering on actual vendor calls rather than on binder_final, because that column is a bool and so records the 120 never-measured designs as False rather than missing. From there we compute hit rates by model, campaign, generator, and target, wrapping each in a Wilson interval since several subgroups sit in the small-n regime where the normal approximation misbehaves. The target plot is the one to read first: it shows antigen choice swamping every other factor we compare.\n\n```\nhead(4, \"PER-PREDICTOR DISCRIMINATIVE POWER\")\nPREDICTORS = sorted({c[len(\"ipsae_min_\"):] for c in ds.columns if c.startswith(\"ipsae_min_\")})\nprint(f\"Predictors ({len(PREDICTORS)}): {', '.join(PREDICTORS)}\")\ndef auc_ci(y, s, n_boot=300, seed=SEED):\n   s = np.asarray(s, dtype=float); y = np.asarray(y, dtype=int)\n   m = ~np.isnan(s)\n   y, s = y[m], s[m]\n   if len(y) < 30 or len(np.unique(y)) < 2:\n       return dict(auc=np.nan, lo=np.nan, hi=np.nan, n=len(y), ap=np.nan)\n   base = roc_auc_score(y, s)\n   ap = average_precision_score(y, s)\n   rng = np.random.default_rng(seed)\n   idx, boots = np.arange(len(y)), []\n   for _ in range(n_boot):\n       b = rng.choice(idx, len(idx), replace=True)\n       if len(np.unique(y[b])) > 1:\n           boots.append(roc_auc_score(y[b], s[b]))\n   lo, hi = (np.percentile(boots, [2.5, 97.5]) if boots else (np.nan, np.nan))\n   return dict(auc=base, lo=lo, hi=hi, n=len(y), ap=ap)\nrows = []\nfor p in PREDICTORS:\n   for metric in [\"ipsae_min\", \"sc_dockq\"]:\n       col = f\"{metric}_{p}\"\n       if col in ev.columns:\n           r = auc_ci(ev.y, ev[col])\n           rows.append({\"predictor\": p, \"metric\": metric, **r})\nperf = pd.DataFrame(rows)\npiv = perf.pivot(index=\"predictor\", columns=\"metric\", values=\"auc\").sort_values(\"ipsae_min\", ascending=False)\nprint(\"\\nAUC vs experimental binder_final:\")\nprint(perf.sort_values(\"auc\", ascending=False).to_string(\n   index=False, formatters={c: \"{:.3f}\".format for c in [\"auc\", \"lo\", \"hi\", \"ap\"]}))\nfig, ax = plt.subplots(figsize=(9, 4.2))\nx = np.arange(len(piv)); w = 0.38\nfor i, (metric, colr) in enumerate([(\"ipsae_min\", \"#4C72B0\"), (\"sc_dockq\", \"#DD8452\")]):\n   sub = perf[perf.metric == metric].set_index(\"predictor\").reindex(piv.index)\n   lo_err = (sub.auc - sub.lo).clip(lower=0).fillna(0)\n   hi_err = (sub.hi - sub.auc).clip(lower=0).fillna(0)\n   ax.bar(x + (i - 0.5) * w, sub.auc, w, label=metric, color=colr)\n   ax.errorbar(x + (i - 0.5) * w, sub.auc,\n               yerr=[lo_err, hi_err], fmt=\"none\", ecolor=\"0.3\", capsize=2, lw=0.9)\nax.axhline(0.5, ls=\"--\", c=\"crimson\", lw=1)\nax.set_xticks(x); ax.set_xticklabels(piv.index, rotation=45, ha=\"right\")\nax.set_ylabel(\"AUC\"); ax.set_ylim(0.35, None)\nax.set_title(\"In-silico score vs wet-lab binding, by structure predictor\")\nax.legend(); plt.tight_layout(); plt.show()\nprint(\"Interpretation: AUCs land well above chance but far below the ~0.9 you \"\n     \"would need to trust a single filter. That gap is the entire practical \"\n     \"reason this dataset exists.\")\nhead(5, \"CONSENSUS SCORING\")\nips_cols = [f\"ipsae_min_{p}\" for p in PREDICTORS if f\"ipsae_min_{p}\" in ev.columns]\ndq_cols = [f\"sc_dockq_{p}\" for p in PREDICTORS if f\"sc_dockq_{p}\" in ev.columns]\ndef pct_rank(df, cols):\n   return df[cols].rank(pct=True, na_option=\"keep\")\nR_ips, R_dq = pct_rank(ev, ips_cols), pct_rank(ev, dq_cols)\nev[\"cons_ipsae\"] = R_ips.mean(axis=1)\nev[\"cons_dockq\"] = R_dq.mean(axis=1)\nev[\"cons_all\"] = pd.concat([R_ips, R_dq], axis=1).mean(axis=1)\nev[\"cons_median\"] = pd.concat([R_ips, R_dq], axis=1).median(axis=1)\nev[\"cons_min\"] = pd.concat([R_ips, R_dq], axis=1).min(axis=1)\nev[\"cons_disagree\"] = pd.concat([R_ips, R_dq], axis=1).std(axis=1)\nbest_single = perf.loc[perf.auc.idxmax()]\nprint(f\"Best single column: {best_single.metric}_{best_single.predictor}  AUC={best_single.auc:.3f}\")\nprint()\nfor name in [\"cons_ipsae\", \"cons_dockq\", \"cons_all\", \"cons_median\", \"cons_min\", \"cons_disagree\"]:\n   r = auc_ci(ev.y, ev[name])\n   print(f\"  {name:16s} AUC={r['auc']:.3f}  [{r['lo']:.3f}, {r['hi']:.3f}]  AP={r['ap']:.3f}\")\ncorr = ev[ips_cols].corr(method=\"spearman\")\nfig, ax = plt.subplots(figsize=(6.2, 5.2))\nim = ax.imshow(corr.values, cmap=\"viridis\", vmin=0, vmax=1)\nlbl = [c.replace(\"ipsae_min_\", \"\") for c in ips_cols]\nax.set_xticks(range(len(lbl))); ax.set_xticklabels(lbl, rotation=90)\nax.set_yticks(range(len(lbl))); ax.set_yticklabels(lbl)\nax.set_title(\"Spearman correlation between predictors (ipSAE)\")\nax.grid(False); fig.colorbar(im, shrink=0.8); plt.tight_layout(); plt.show()\nprint(\"\\nIf every off-diagonal cell were ~1.0 there would be no ensemble gain to \"\n     \"harvest. The moderate correlations are why cons_all typically edges out \"\n     \"the best single predictor — and why disagreement itself carries signal.\")\n```\n\nWe score all ten predictors against the wet-lab label, on both ipSAE and self-consistency DockQ, with bootstrapped confidence intervals so we can see which differences are real. We then rank-normalize each column to percentiles and aggregate them, which keeps the comparison scale-free across metrics that live on different ranges and pile up differently at zero. The Spearman heatmap explains why the ensemble helps at all; if the predictors agreed perfectly there would be nothing left to harvest.\n\n``` python\nhead(6, \"BUDGET CURVES (precision@N)\")\ndef budget_curve(df, score_col, max_n=400):\n   d = df[[score_col, \"y\"]].dropna().sort_values(score_col, ascending=False)\n   hits = d.y.values.cumsum()\n   n = np.arange(1, len(d) + 1)\n   k = min(max_n, len(d))\n   return n[:k], (hits / n)[:k]\nfig, ax = plt.subplots(figsize=(8, 4.4))\nbest_col = f\"{best_single.metric}_{best_single.predictor}\"\nfor col, lab, style in [(best_col, f\"best single ({best_col})\", \"-\"),\n                       (\"cons_all\", \"consensus (rank-avg, all)\", \"-\"),\n                       (\"cons_min\", \"consensus (unanimity/min)\", \"--\")]:\n   n, prec = budget_curve(ev, col)\n   ax.plot(n, prec, style, lw=1.8, label=lab)\nax.axhline(ev.y.mean(), ls=\":\", c=\"crimson\", lw=1.4, label=f\"random baseline ({ev.y.mean():.2f})\")\nax.set_xlabel(\"designs ordered for wet-lab testing (N, best-first)\")\nax.set_ylabel(\"hit rate among top N\"); ax.set_title(\"How much does in-silico triage buy you?\")\nax.legend(); plt.tight_layout(); plt.show()\nprint(\"Enrichment at small budgets:\")\nfor N in [25, 50, 100, 200]:\n   line = f\"  N={N:4d} | random {ev.y.mean():.3f}\"\n   for col, lab in [(best_col, \"best-single\"), (\"cons_all\", \"consensus\")]:\n       n, prec = budget_curve(ev, col, max_n=N)\n       line += f\" | {lab} {prec[-1]:.3f} ({prec[-1] / ev.y.mean():.2f}x)\"\n   print(line)\nhead(7, \"VENDOR CONCORDANCE\")\nboth = ev[ev.adaptyv_binding.isin(CALLS) & ev.twist_binding.isin(CALLS)]\nct = pd.crosstab(both.adaptyv_binding, both.twist_binding)\nprint(f\"Designs with calls from BOTH vendors: {len(both):,}\\n\")\nprint(ct.to_string())\nif len(both) > 10:\n   kappa = cohen_kappa_score(both.adaptyv_binding, both.twist_binding)\n   agree = (both.adaptyv_binding == both.twist_binding).mean()\n   print(f\"\\nRaw agreement: {agree:.3f}   Cohen's kappa: {kappa:.3f}\")\n   print(\"Kappa well under 1.0 means part of the 'unpredictable' variance above \"\n         \"is assay disagreement, not model failure.\")\nkd = ev[[\"adaptyv_kd_nM\", \"twist_kd_nM\"]].dropna()\nkd = kd[(kd > 0).all(axis=1)]\nif len(kd) > 10:\n   rho, pv = stats.spearmanr(kd.adaptyv_kd_nM, kd.twist_kd_nM)\n   fig, ax = plt.subplots(figsize=(4.8, 4.6))\n   ax.scatter(kd.adaptyv_kd_nM, kd.twist_kd_nM, s=16, alpha=0.6, c=\"#4C72B0\", edgecolor=\"none\")\n   lims = [min(kd.min()) * 0.5, max(kd.max()) * 2]\n   ax.plot(lims, lims, \"k--\", lw=1)\n   ax.set_xscale(\"log\"); ax.set_yscale(\"log\")\n   ax.set_xlabel(\"Adaptyv KD (nM)\"); ax.set_ylabel(\"Twist KD (nM)\")\n   ax.set_title(f\"Cross-vendor KD, n={len(kd)}, Spearman rho={rho:.2f}\")\n   plt.tight_layout(); plt.show()\n   med_ratio = np.median(kd.twist_kd_nM / kd.adaptyv_kd_nM)\n   print(f\"Median KD ratio (Twist/Adaptyv): {med_ratio:.2f}x  -> systematic format offset, \"\n         \"so treat absolute KD across vendors as ordinal, not interchangeable.\")\n```\n\nWe convert ranking performance into precision@N, because no lab orders 1,300 constructs and AUC quietly hides how a score behaves at the top of the list. The enrichment table then tells us what triage actually buys at budgets of 25, 50, 100, and 200. We follow it with Cohen’s κ and a log-log KD comparison between vendors, which sets the ceiling: label noise bounds how high any AUC above can honestly climb.\n\n```\nhead(8, \"EXPRESSION CONFOUND\")\nif \"twist_expression_mg_per_mL\" in ev.columns:\n   g = ev.dropna(subset=[\"twist_expression_mg_per_mL\"])\n   a = g.loc[g.y == 1, \"twist_expression_mg_per_mL\"]\n   b = g.loc[g.y == 0, \"twist_expression_mg_per_mL\"]\n   if len(a) > 5 and len(b) > 5:\n       u, pv = stats.mannwhitneyu(a, b)\n       print(f\"Titer (mg/mL)  binders median {a.median():.2f} (n={len(a)})  |  \"\n             f\"non-binders {b.median():.2f} (n={len(b)})   Mann-Whitney p={pv:.2e}\")\n   r = auc_ci(g.y, g.twist_expression_mg_per_mL)\n   print(f\"AUC of raw expression titer alone as a 'binder' predictor: {r['auc']:.3f}\")\n   fig, axes = plt.subplots(1, 2, figsize=(9, 3.6))\n   axes[0].hist([b, a], bins=25, label=[\"non-binder\", \"binder\"],\n                color=[\"#BBBBBB\", \"#4C72B0\"], density=True)\n   axes[0].set_xlabel(\"Twist titer (mg/mL)\"); axes[0].set_ylabel(\"density\"); axes[0].legend()\n   axes[0].set_title(\"Expression by outcome\")\n   if \"adaptyv_expression\" in ev.columns:\n       ex = ev.groupby(ev.adaptyv_expression.astype(str)).y.agg([\"mean\", \"size\"])\n       ex = ex[ex[\"size\"] >= 10].sort_values(\"mean\")\n       axes[1].barh(ex.index, ex[\"mean\"], color=\"#DD8452\")\n       axes[1].set_xlabel(\"hit rate\"); axes[1].set_title(\"Hit rate by Adaptyv expression class\")\n   plt.tight_layout(); plt.show()\nprint(\"\\nTakeaway: if expression alone scores meaningfully above 0.5, then part of \"\n     \"every AUC in section 4 is a solubility signal riding along. To isolate \"\n     \"interface quality, re-run section 4 restricted to designs that expressed.\")\nexpressed = ev[ev.adaptyv_expression.astype(str).isin([\"medium\", \"high\"])] if \"adaptyv_expression\" in ev.columns else ev\nif len(expressed) > 100:\n   r_all = auc_ci(ev.y, ev.cons_all)\n   r_exp = auc_ci(expressed.y, expressed.cons_all)\n   print(f\"  consensus AUC, all evaluable   : {r_all['auc']:.3f} (n={r_all['n']})\")\n   print(f\"  consensus AUC, expressed only  : {r_exp['auc']:.3f} (n={r_exp['n']})\")\nhead(9, \"EPITOPE CONVERGENCE\")\ndef parse_epitope(s):\n   if not isinstance(s, str) or not s.strip():\n       return frozenset()\n   out = set()\n   for tok in s.split(\";\"):\n       tok = tok.strip()\n       if not tok:\n           continue\n       out.add(tok.split(\":\")[-1])\n   return frozenset(out)\nev[\"epi\"] = ev[\"epitope_residues\"].apply(parse_epitope)\ndef mean_pairwise_jaccard(sets, max_pairs=4000, seed=SEED):\n   sets = [s for s in sets if len(s) > 0]\n   if len(sets) < 2:\n       return np.nan\n   pairs = list(itertools.combinations(range(len(sets)), 2))\n   rng = np.random.default_rng(seed)\n   if len(pairs) > max_pairs:\n       pairs = [pairs[i] for i in rng.choice(len(pairs), max_pairs, replace=False)]\n   vals = []\n   for i, j in pairs:\n       u = len(sets[i] | sets[j])\n       vals.append(len(sets[i] & sets[j]) / u if u else 0.0)\n   return float(np.mean(vals))\nrows = []\nfor tgt, g in ev.groupby(\"target\"):\n   B = g.loc[g.y == 1, \"epi\"].tolist()\n   N = g.loc[g.y == 0, \"epi\"].tolist()\n   if len(B) >= 3 and len(N) >= 3:\n       rows.append({\"target\": tgt, \"n_bind\": len(B), \"n_non\": len(N),\n                    \"J_binders\": mean_pairwise_jaccard(B),\n                    \"J_nonbinders\": mean_pairwise_jaccard(N)})\nepi = pd.DataFrame(rows)\nif len(epi):\n   epi[\"delta\"] = epi.J_binders - epi.J_nonbinders\n   print(epi.sort_values(\"delta\", ascending=False).to_string(\n       index=False, formatters={c: \"{:.3f}\".format for c in [\"J_binders\", \"J_nonbinders\", \"delta\"]}))\n   w = stats.wilcoxon(epi.J_binders, epi.J_nonbinders) if len(epi) >= 6 else None\n   if w:\n       print(f\"\\nPaired Wilcoxon across targets: p={w.pvalue:.4f}  \"\n             f\"(binders more epitope-convergent than failures?)\")\n   tgt = epi.sort_values(\"n_bind\", ascending=False).target.iloc[0]\n   sub = ev[ev.target == tgt]\n   freq_b = pd.Series([r for s in sub[sub.y == 1].epi for r in s]).value_counts()\n   freq_n = pd.Series([r for s in sub[sub.y == 0].epi for r in s]).value_counts()\n   top = freq_b.head(18).index\n   fig, ax = plt.subplots(figsize=(9, 3.8))\n   xx = np.arange(len(top))\n   ax.bar(xx - 0.2, (freq_b.reindex(top).fillna(0) / max(1, (sub.y == 1).sum())), 0.4,\n          label=\"binders\", color=\"#4C72B0\")\n   ax.bar(xx + 0.2, (freq_n.reindex(top).fillna(0) / max(1, (sub.y == 0).sum())), 0.4,\n          label=\"non-binders\", color=\"#BBBBBB\")\n   ax.set_xticks(xx); ax.set_xticklabels(top, rotation=70, ha=\"right\")\n   ax.set_ylabel(\"fraction of designs contacting\"); ax.set_title(f\"Epitope usage on {tgt}\")\n   ax.legend(); plt.tight_layout(); plt.show()\n```\n\nWe test whether expression titer alone discriminates binders, and if it does, we know part of every score from above is solubility riding along under another name. Re-running consensus on expressed-only designs isolates interface quality from biophysics. We then parse the epitope contact lists into residue sets and ask, per target and paired across targets, whether confirmed binders converge on a shared patch more than the failures do.\n\n```\nhead(10, \"MODELLING WITH HONEST CROSS-VALIDATION\")\nAAS = \"ACDEFGHIKLMNPQRSTVWY\"\nKD_HYDRO = dict(zip(AAS, [1.8, 2.5, -3.5, -3.5, 2.8, -0.4, -3.2, 4.5, -3.9, 3.8,\n                         1.9, -3.5, -1.6, -3.5, -4.5, -0.8, -0.7, 4.2, -0.9, -1.3]))\nCHARGE = {\"K\": 1, \"R\": 1, \"H\": 0.1, \"D\": -1, \"E\": -1}\ndef seq_features(seq):\n   seq = \"\".join(ch for ch in str(seq).upper() if ch in AAS)\n   L = max(1, len(seq))\n   counts = {a: seq.count(a) / L for a in AAS}\n   f = {f\"aa_{a}\": counts[a] for a in AAS}\n   f[\"length\"] = len(seq)\n   f[\"net_charge\"] = sum(CHARGE.get(c, 0) for c in seq)\n   f[\"charge_density\"] = f[\"net_charge\"] / L\n   f[\"gravy\"] = float(np.mean([KD_HYDRO[c] for c in seq])) if seq else 0.0\n   f[\"aromatic\"] = sum(counts[a] for a in \"FWY\")\n   f[\"helix_prone\"] = sum(counts[a] for a in \"AELM\")\n   f[\"beta_prone\"] = sum(counts[a] for a in \"VIYFT\")\n   f[\"gly_pro\"] = counts[\"G\"] + counts[\"P\"]\n   p = np.array([counts[a] for a in AAS]); p = p[p > 0]\n   f[\"entropy\"] = float(-(p * np.log2(p)).sum())\n   run, best = 0, 0\n   for c in seq:\n       run = run + 1 if KD_HYDRO[c] > 1.5 else 0\n       best = max(best, run)\n   f[\"max_hydrophobic_run\"] = best\n   return f\nSF = pd.DataFrame([seq_features(s) for s in ev.sequence], index=ev.index)\nseq_cols = list(SF.columns)\nsil_cols = [c for c in ev.columns if c.startswith((\"ipsae_min_\", \"sc_dockq_\"))] + \\\n          [\"cons_all\", \"cons_min\", \"cons_disagree\"]\nmeta_cols = [c for c in [\"rank\", \"n_optimization_rounds\", \"epitope_n_residues\"] if c in ev.columns]\nX_all = pd.concat([ev[sil_cols + meta_cols], SF], axis=1)\ny = ev.y.values\ngroups = ev.target.values\nFEATURE_SETS = {\n   \"in-silico only\": sil_cols + meta_cols,\n   \"sequence only\": seq_cols,\n   \"in-silico + sequence\": sil_cols + meta_cols + seq_cols,\n}\ndef cv_auc(X, y, splitter, groups=None):\n   aucs = []\n   it = splitter.split(X, y, groups) if groups is not None else splitter.split(X, y)\n   for tr, te in it:\n       if len(np.unique(y[te])) < 2:\n           continue\n       m = HistGradientBoostingClassifier(max_depth=4, max_iter=250,\n                                          learning_rate=0.06, random_state=SEED)\n       m.fit(X.iloc[tr], y[tr])\n       aucs.append(roc_auc_score(y[te], m.predict_proba(X.iloc[te])[:, 1]))\n   return float(np.mean(aucs)), float(np.std(aucs)), len(aucs)\nprint(f\"{'feature set':24s} {'random 5-fold':>18s} {'grouped-by-target':>20s}\")\nprint(\"-\" * 66)\nresults = {}\nfor name, cols in FEATURE_SETS.items():\n   X = X_all[cols]\n   r_mean, r_sd, _ = cv_auc(X, y, StratifiedKFold(5, shuffle=True, random_state=SEED))\n   g_mean, g_sd, nf = cv_auc(X, y, GroupKFold(n_splits=5), groups=groups)\n   results[name] = (r_mean, g_mean)\n   print(f\"{name:24s} {r_mean:.3f} +/- {r_sd:.3f}   {g_mean:.3f} +/- {g_sd:.3f}\")\ngap = results[\"in-silico + sequence\"][0] - results[\"in-silico + sequence\"][1]\nprint(f\"\\nRandom-CV minus grouped-CV for the full feature set: {gap:+.3f}\")\nprint(\"That gap is leakage: features that encode target identity (epitope size, \"\n     \"length priors, generator habits) let a randomly-split model recover the \"\n     \"per-target base rate instead of learning what makes a binder. Report the \"\n     \"grouped number; the random one is what a target-blind reviewer will catch.\")\nXt = pd.get_dummies(pd.Series(groups, index=ev.index), prefix=\"tgt\")\nr_mean, _, _ = cv_auc(Xt, y, StratifiedKFold(5, shuffle=True, random_state=SEED))\nprint(f\"\\nControl - target one-hot ONLY, random CV: AUC={r_mean:.3f} \"\n     \"(pure base-rate memorisation, zero design signal).\")\ngkf = GroupKFold(n_splits=5)\ntr, te = next(iter(gkf.split(X_all, y, groups)))\nmodel = HistGradientBoostingClassifier(max_depth=4, max_iter=250,\n                                      learning_rate=0.06, random_state=SEED).fit(\n   X_all[FEATURE_SETS[\"in-silico + sequence\"]].iloc[tr], y[tr])\nimp = permutation_importance(model, X_all[FEATURE_SETS[\"in-silico + sequence\"]].iloc[te],\n                            y[te], n_repeats=12, random_state=SEED, scoring=\"roc_auc\")\norder = np.argsort(imp.importances_mean)[-18:]\nnames = np.array(FEATURE_SETS[\"in-silico + sequence\"])[order]\nfig, ax = plt.subplots(figsize=(7, 5))\nax.barh(names, imp.importances_mean[order],\n       xerr=imp.importances_std[order], color=\"#55A868\")\nax.set_xlabel(\"drop in AUC when permuted\")\nax.set_title(\"Permutation importance (held-out target block)\")\nplt.tight_layout(); plt.show()\nhead(\"\", \"SUMMARY\")\nprint(f\"\"\"\nEvaluable designs        : {len(ev):,}   base hit rate {ev.y.mean():.3f}\nBest single in-silico    : {best_col}  AUC {best_single.auc:.3f}\nRank-average consensus   : AUC {auc_ci(ev.y, ev.cons_all)['auc']:.3f}\nHonest ML (grouped CV)   : AUC {results['in-silico + sequence'][1]:.3f}   <- the one to report\nSame model, random CV    : AUC {results['in-silico + sequence'][0]:.3f}   (gap = {gap:+.3f} leakage)\nFive things this dataset teaches that a design paper usually cannot:\n  1. Target identity dominates every other factor; always stratify.\n  2. Structure-predictor confidence is real but weak signal (AUC ~0.6-0.75),\n     nowhere near a standalone go/no-go filter.\n  3. Ensembling across predictors is a cheap, reliable few-points-of-AUC win.\n  4. Cross-vendor label noise caps how high any AUC here can honestly go.\n  5. Expression failure masquerades as binding failure. Condition on it.\nExtensions worth trying:\n  - load_table('insilico_cofold_predictions') for all 5 seeds/predictor, and\n    test whether seed VARIANCE beats seed-best as a confidence signal\n  - load_table('adaptyv_fit_curves') to refit kinetics yourself and flag\n    designs whose reported KD rests on a poorly-conditioned fit\n  - load_table('insilico_provenance_steps') to relate optimisation-round count\n    to eventual success\n  - snapshot_download(..., allow_patterns='data/designs/EGFR/<name>/*') for\n    mmCIF structures + PAE matrices on a single design\n\"\"\")\n```\n\nWe featurize sequences by composition, charge, hydropathy, entropy, and hydrophobic run length, then fit gradient boosting under two schemes: random folds and target-grouped folds. The gap between them is the leakage, since designs nest inside targets with very different base rates and a random split lets the model memorize which antigens are easy. The one-hot control makes that explicit, and permutation importance on a held-out target block shows what survives when we remove the shortcut.\n\nIn conclusion, in-silico scoring was helpful, but it did not tell the whole story. The target strongly influenced the results, so comparing models without accounting for it could easily give us a misleading picture. The structure predictors showed useful signals, and combining them gave a modest improvement, but they were still not reliable enough to use on their own. We also found that differences between experiments and protein expression could make a design look like a binding failure even when the real issue was poor expression. Overall, we learned that careful evaluation mattered more than chasing impressive individual metrics. By grouping our cross-validation by target, we got a more realistic view of how well the models could generalize to new targets.\n\nCheck out the ** FULL CODES here.** Also, feel free to follow us on\n\n**and don’t forget to join our**[Twitter](https://x.com/intent/follow?screen_name=marktechpost)\n\n**and Subscribe to**\n\n[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)**. Wait! are you on telegram?**\n\n[our Newsletter](https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}})\n\n[now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)\n\nSana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.", "url": "https://wpnews.pro/news/from-in-silico-to-wet-lab-evaluating-ai-protein-design-performance", "canonical_source": "https://www.marktechpost.com/2026/08/27/from-in-silico-to-wet-lab-evaluating-ai-protein-design-performance/", "published_at": "2026-08-27 15:36:29+00:00", "updated_at": "2026-08-27 15:49:29.355630+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-tools"], "entities": ["Anthropic", "claude-protein-binder-design"], "alternates": {"html": "https://wpnews.pro/news/from-in-silico-to-wet-lab-evaluating-ai-protein-design-performance", "markdown": "https://wpnews.pro/news/from-in-silico-to-wet-lab-evaluating-ai-protein-design-performance.md", "text": "https://wpnews.pro/news/from-in-silico-to-wet-lab-evaluating-ai-protein-design-performance.txt", "jsonld": "https://wpnews.pro/news/from-in-silico-to-wet-lab-evaluating-ai-protein-design-performance.jsonld"}}