OpenEval: Why LLM Evaluation Needs a Standard Format

OpenEval, a new open-source project, aims to standardize LLM evaluation by defining a portable JSON Schema for test cases, graders, and results. The project provides SDKs, a CLI, and converters for popular frameworks like DeepEval and Promptfoo, enabling teams to move between evaluation tools without rewriting datasets. Version 1.0.0 is now available on npm and PyPI.

Every LLM evaluation framework today invents its own test case format, its own grader definitions, and its own results schema. DeepEval, Promptfoo, Inspect AI, and lm-evaluation-harness all solve the same core problem checking whether a model's output is correct but none of them can read each other's eval datasets. That means every time a team wants to compare frameworks, or move from a notebook prototype to a production eval pipeline, they end up hand-rewriting the same test cases over and over. OpenEval is an attempt to fix that by defining a small, portable JSON Schema for eval test cases, graders, and results, plus tooling to move data between frameworks instead of retyping it. What's in the repo: A versioned JSON Schema spec for test cases, grader configs, and result records. TypeScript and Python SDKs for reading and writing OpenEval-formatted datasets. A CLI with validate, convert, init, and summarize commands. Converters for popular frameworks so existing datasets can be brought in or exported out without a manual rewrite. The project just published v1.0.0 to npm and PyPI, and issues are open on 17+ framework integrations if anyone wants to help wire up a converter for a framework not yet covered. Repo: https://github.com/adhabnr-ux/openeval https://github.com/adhabnr-ux/openeval Would love feedback from anyone who has hit this same portability problem while switching between eval tools.