{"slug": "one-schema-for-every-eval", "title": "One Schema for Every Eval", "summary": "The EvalEval Coalition has launched a unified, open data format and public dataset for AI evaluation results, aiming to address fragmentation and enable trust and comparability across frameworks. The schema captures full experimental context—including prompt templates, inference parameters, and system states—to make results traceable and reproducible, with a shared task for contributions and a planned ACL 2026 Workshop in San Diego.", "body_md": "A **unified, open data format** and **public dataset** for AI evaluation results.\nWe are collecting all evaluation results in a standardized schema to enable rigorous research and broader impact.\n\nDeveloped by the EvalEval Coalition\n\nwith feedback from researchers at\n\nAddressing the fragmentation in AI evaluation to enable trust and comparability.\n\nEvaluation results are currently siloed by framework. This schema creates a common interchange format, allowing results from HELM, EleutherAI, Inspect, and custom scripts to co-exist and be compared directly without complex mapping.\n\nA score without configuration is just noise. We go beyond the metric to capture the full experimental context—prompt templates, inference parameters, and system states—making every result traceable, transparent, and reproducible.\n\nLiberating evaluation results from static PDFs and closed leaderboards. We transform scattered metrics into a structured, queryable global dataset, powering the next generation of meta-analysis and automated leaderboard construction.\n\nA granular, line-by-line breakdown of the standardized format.\n\nBuilt for scale, reproducibility, and scientific rigor.\n\nWe assign every evaluation a unique UUID. This prevents filename collisions and allows multiple runs of the same model—from different dates or configurations—to coexist safely.\n\nModels change silently. We mandate `retrieved_timestamp`\n\nto capture the exact moment of inference, enabling precise studies on API drift and model versioning.\n\nPerformance depends on the runner. We explicitly separate `platform`\n\n(provider) from `engine`\n\n(inference system), isolating hardware and software variables in your analysis.\n\nFrom simple accuracy to complex LLM-as-a-Judge scores, our schema standardizes all outputs. Compare results across different evaluation libraries without writing custom parsers.\n\nBuilt for data science. Flat, structured JSON files mean you can ingest millions of results into Pandas or SQL in seconds, slicing by architecture, date, or task immediately.\n\nScience requires receipts. By linking every score to its exact prompt template and generation parameters, we ensure that every result in the dataset is fully reproducible.\n\nWe are collecting all evaluation results in our schema in a public dataset. This repository serves as a standardized metadata store for results from various leaderboards, research papers, and local evaluations. If you are an eval provider or leaderboard maintainer, we are looking for your generous data contibutions via pull requests!\n\nHow we organize and validate the data.\n\nData is split by individual model. Each evaluation consists of an **aggregate result** (JSON) and a **detailed results file** (JSONL). Both share the same **UUID** to ensure clean organization.\n\nWe provide tools to easily adapt your existing workflows.\n\nWe have ready-made [converters](https://github.com/evaleval/every_eval_ever/tree/main/eval_converters) for popular frameworks:\n\n# Run validation locally\n\nuv run pre-commit run --all-files\n\n`/data`\n\nwith the name of the benchmark.`developer_name/model_name`\n\n.\nWe are launching a **Shared Task** for practitioners to contribute public and proprietary eval data.\nParticipate for co-authorship and join us at the **ACL 2026 Workshop** in San Diego.", "url": "https://wpnews.pro/news/one-schema-for-every-eval", "canonical_source": "https://evalevalai.com/projects/every-eval-ever/", "published_at": "2026-07-23 20:40:39+00:00", "updated_at": "2026-07-23 20:52:13.449172+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["EvalEval Coalition", "HELM", "EleutherAI", "Inspect", "ACL 2026 Workshop", "San Diego"], "alternates": {"html": "https://wpnews.pro/news/one-schema-for-every-eval", "markdown": "https://wpnews.pro/news/one-schema-for-every-eval.md", "text": "https://wpnews.pro/news/one-schema-for-every-eval.txt", "jsonld": "https://wpnews.pro/news/one-schema-for-every-eval.jsonld"}}