{"slug": "show-hn-i-benchmarked-llms-on-predicting-knife-steel-properties", "title": "Show HN: I benchmarked LLMs on predicting knife steel properties", "summary": "A new open benchmark from the Steel-predictor project shows that Anthropic's Claude Sonnet 5 leads large language models in predicting knife-steel properties from chemical composition, achieving a mean Spearman rank correlation of 0.869 against laboratory measurements, while a reference machine-learning model trained on the same data scored 0.969. The benchmark, which tested 48 steels for edge retention via CATRA testing and 12 for toughness via Charpy impact energy, found LLMs rank edge retention well (ρ 0.85–0.92) but struggle with toughness (ρ 0.38–0.84).", "body_md": "**How well can large language models predict knife-steel properties from chemical composition alone?**\n\nAn open, reproducible benchmark that gives an LLM only a steel's composition (e.g. `C=1.45%, Cr=20%, V=4%, Mo=1%, powder-metallurgy: yes`\n\n) and asks it to rate two properties on a 1–10 scale, then scores those ratings against **objective laboratory measurements**:\n\n**Edge retention**← CATRA standardized machine-cutting test (total card stock cut, mm) — 48 steels** Toughness**← Charpy impact energy (ft-lbs) — 12 steels\n\nScoring is **scale-free** (rank correlation + pairwise ranking accuracy), so a model is judged purely on whether it orders steels correctly, not on how it calibrates the 1–10 scale.\n\n**📊 Site with charts & analysis:** [https://steel-predictor-project.github.io/steel-llm-eval/](https://steel-predictor-project.github.io/steel-llm-eval/) · **Docs:** [methodology](/Steel-predictor-project/steel-llm-eval/blob/main/docs/methodology.md) · [results & analysis](/Steel-predictor-project/steel-llm-eval/blob/main/docs/results-analysis.md)\n\nRanked by mean Spearman rank correlation (ρ) vs. the measurements. Higher is better; 1.0 = perfect ordering, 0.0 = random.\n\n| Model | Edge ρ (n) | Edge pairwise | Tough ρ (n) | Tough pairwise | Mean ρ |\n|---|---|---|---|---|---|\n| steel-predictor (reference ML) † | 0.992 (48) | 0.98 | 0.946 (12) | 0.938 | 0.969 |\n| anthropic/claude-sonnet-5 | 0.894 (48) | 0.918 | 0.844 (12) | 0.881 | 0.869 |\n| google/gemini-3.6-flash | 0.918 (48) | 0.913 | 0.698 (12) | 0.797 | 0.808 |\n| openai/gpt-4o | 0.868 (48) | 0.907 | 0.600 (12) | 0.746 | 0.734 |\n| meta-llama/llama-3.3-70b-instruct | 0.864 (47) | 0.964 | 0.514 (12) | 0.780 | 0.689 |\n| deepseek/deepseek-chat-v3.1 | 0.869 (48) | 0.910 | 0.380 (12) | 0.661 | 0.625 |\n| openai/gpt-4o-mini | 0.850 (48) | 0.984 | 0.385 (12) | 0.689 | 0.617 |\n\n*Edge retention n=48 (CATRA), toughness n=12 (Charpy). Zero-shot, temperature 0, one sample per steel.*\n\n**† Important fairness caveat:** the reference ML model ([Steel-predictor](https://github.com/Steel-predictor-project/Steel-predictor)) was **trained on these same CATRA/Charpy measurements**, so its scores here are largely *in-sample* and are shown as an upper-reference bar, **not** as a fair head-to-head with the zero-shot LLMs. The model's honest out-of-sample performance is its LOOCV MAE (0.391), reported in that repo. The LLMs, by contrast, have never seen this labeled set.\n\n**LLMs are genuinely good at ranking edge retention**(ρ ≈ 0.85–0.92). Wear resistance is strongly and legibly encoded in composition (carbide-forming elements — C, V, Cr, W, Mo), and frontier models clearly \"know\" that chemistry.**Toughness is where they struggle**(ρ 0.38–0.84). It depends on subtler factors (carbide size/distribution, powder-metallurgy processing, matrix state) that aren't obvious from a composition string, and the spread across models is large.**Frontier > small.** Claude Sonnet and Gemini lead; the smaller/cheaper models drop off sharply on toughness while staying competitive on edge retention.\n\n```\ngit clone https://github.com/Steel-predictor-project/steel-llm-eval.git\ncd steel-llm-eval\n\nexport OPENROUTER_API_KEY=sk-or-...   # one key → OpenAI, Anthropic, Google, Meta, DeepSeek, ...\n./run_benchmark.sh                    # runs every model and rebuilds the leaderboard\n```\n\nRun a single model, or a quick offline sanity check with no API key:\n\n```\npython harness/run_eval.py --model anthropic/claude-sonnet-5\npython harness/run_eval.py --provider mock     # deterministic heuristic, no key needed\npython harness/score.py\n```\n\nRaw per-steel responses are written to `results/raw_<model>.csv`\n\n; scores to `results/scores.csv`\n\nand `results/leaderboard.md`\n\n.\n\n**Prompt**(`harness/prompts.py`\n\n) — a fixed system + user prompt gives the model the composition, PM flag, and test hardness (when known) and asks for JSON:`{\"edge_retention\": n, \"toughness\": n}`\n\n. Identical for every model.**Run**(`harness/run_eval.py`\n\n) — queries a model for all 51 steels via OpenRouter and parses the JSON.**Score**(`harness/score.py`\n\n) — vs. the measurements:**Spearman ρ** and**Kendall τ** rank correlation (headline; scale-free).**Pairwise accuracy**— over all steel pairs, how often the model orders them the same way the measurement does (ties excluded).** Normalized MAE**— a calibration sanity check after min-max scaling the measurement to 1–10 (secondary; see`scores.csv`\n\n).\n\n**Baselines**— the purpose-built ML model (upper reference, in-sample caveat above) and a constant predictor (floor).\n\n**Ground truth is objective measurement only**(CATRA, Charpy). No subjective 1–10 expert ratings are used anywhere in scoring.** Composition-only.**Models are not told heat-treat protocol or geometry (only hardness where recorded), so this measures what chemistry*alone*implies — the same constraint the reference model operates under.**Small toughness set (n=12).** Treat toughness ρ as indicative, not definitive; single sample per steel at temperature 0 (no self-consistency / multi-sample averaging yet).**Rank metrics are primary** precisely because different models calibrate the 1–10 scale differently; ranking is what's comparable and decision-relevant.\n\nThe benchmark (`data/benchmark.csv`\n\n) is derived from the processed dataset of the [Steel-predictor](https://github.com/Steel-predictor-project/Steel-predictor) project. The two ground-truth measurements come from:\n\n**Edge retention (CATRA, 48 steels)**— Larrin Thomas,*\"Testing the Edge Retention of 48 Knife Steels\"*(2020),[KnifeSteelNerds.com](https://knifesteelnerds.com/2020/05/01/testing-the-edge-retention-of-48-knife-steels/). This benchmark's entire edge-retention ground truth is built on Larrin Thomas's published CATRA measurements — full credit to him.**Toughness (Charpy, 12 steels)**—[Crucible Industries](https://www.crucible.com/)published datasheets (CPM series).** Compositions & test hardness**— manufacturer datasheets (Crucible, Böhler-Uddeholm/voestalpine, Carpenter, Alleima, Hitachi/Proterial) plus published literature.\n\nEvery individual source is enumerated with links in the Steel-predictor repo's [ DATA_SOURCES.md](https://github.com/Steel-predictor-project/Steel-predictor/blob/main/DATA_SOURCES.md). Underlying factual measurements remain the property of their original publishers; this repo redistributes only its own normalized compilation and derived features.\n\nCode: **Apache-2.0** (`LICENSE`\n\n). Curated benchmark data + reference model outputs: **CC BY 4.0** (`data/LICENSE`\n\n), covering only this project's compilation/derived features. Attribution requested: \"Steel Property Predictor Project\" with a link to this repo.\n\nPRs welcome to add models (extend the list in `run_benchmark.sh`\n\n), prompt variants (few-shot, chain-of-thought, self-consistency), or additional measured steels (with cited public sources). Please don't add subjective-rating datasets as ground truth.", "url": "https://wpnews.pro/news/show-hn-i-benchmarked-llms-on-predicting-knife-steel-properties", "canonical_source": "https://github.com/Steel-predictor-project/steel-llm-eval", "published_at": "2026-08-18 14:49:15+00:00", "updated_at": "2026-08-18 15:12:53.286247+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models"], "entities": ["Anthropic", "Claude Sonnet 5", "Google", "Gemini 3.6 Flash", "OpenAI", "GPT-4o", "Meta", "Llama 3.3 70B Instruct"], "alternates": {"html": "https://wpnews.pro/news/show-hn-i-benchmarked-llms-on-predicting-knife-steel-properties", "markdown": "https://wpnews.pro/news/show-hn-i-benchmarked-llms-on-predicting-knife-steel-properties.md", "text": "https://wpnews.pro/news/show-hn-i-benchmarked-llms-on-predicting-knife-steel-properties.txt", "jsonld": "https://wpnews.pro/news/show-hn-i-benchmarked-llms-on-predicting-knife-steel-properties.jsonld"}}