{"slug": "openeval-why-llm-evaluation-needs-a-standard-format", "title": "OpenEval: Why LLM Evaluation Needs a Standard Format", "summary": "OpenEval, a new open-source project, aims to standardize LLM evaluation by defining a portable JSON Schema for test cases, graders, and results. The project provides SDKs, a CLI, and converters for popular frameworks like DeepEval and Promptfoo, enabling teams to move between evaluation tools without rewriting datasets. Version 1.0.0 is now available on npm and PyPI.", "body_md": "Every LLM evaluation framework today invents its own test case format, its own grader definitions, and its own results schema. DeepEval, Promptfoo, Inspect AI, and lm-evaluation-harness all solve the same core problem (checking whether a model's output is correct) but none of them can read each other's eval datasets.\n\nThat means every time a team wants to compare frameworks, or move from a notebook prototype to a production eval pipeline, they end up hand-rewriting the same test cases over and over.\n\nOpenEval is an attempt to fix that by defining a small, portable JSON Schema for eval test cases, graders, and results, plus tooling to move data between frameworks instead of retyping it.\n\nWhat's in the repo:\n\nA versioned JSON Schema spec for test cases, grader configs, and result records.\n\nTypeScript and Python SDKs for reading and writing OpenEval-formatted datasets.\n\nA CLI with validate, convert, init, and summarize commands.\n\nConverters for popular frameworks so existing datasets can be brought in or exported out without a manual rewrite.\n\nThe project just published v1.0.0 to npm and PyPI, and issues are open on 17+ framework integrations if anyone wants to help wire up a converter for a framework not yet covered.\n\nRepo: [https://github.com/adhabnr-ux/openeval](https://github.com/adhabnr-ux/openeval)\n\nWould love feedback from anyone who has hit this same portability problem while switching between eval tools.", "url": "https://wpnews.pro/news/openeval-why-llm-evaluation-needs-a-standard-format", "canonical_source": "https://dev.to/adha_ak_d60b39fbb66769fd1/openeval-why-llm-evaluation-needs-a-standard-format-50di", "published_at": "2026-07-30 03:40:11+00:00", "updated_at": "2026-07-30 03:59:16.656329+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["OpenEval", "DeepEval", "Promptfoo", "Inspect AI", "lm-evaluation-harness"], "alternates": {"html": "https://wpnews.pro/news/openeval-why-llm-evaluation-needs-a-standard-format", "markdown": "https://wpnews.pro/news/openeval-why-llm-evaluation-needs-a-standard-format.md", "text": "https://wpnews.pro/news/openeval-why-llm-evaluation-needs-a-standard-format.txt", "jsonld": "https://wpnews.pro/news/openeval-why-llm-evaluation-needs-a-standard-format.jsonld"}}