Feedback on LLM-as-a-Judge design for open-source library

The REFUTE dataset and leaderboard, hosted on Hugging Face by BGPT-OFFICIAL, provide a judge-free evaluation board for science paper summaries, addressing the failure mode where LLM-as-a-judge re-imports overclaiming errors. The resource splits critique skill from calibration, serving as a foil for deciding when LLM judges are useful versus when objective axes are safer.

Nice work shipping an open LLM-as-a-Judge API. One design pressure we keep hitting on science-critique evals: if the task itself is about whether a model overclaims, putting another LLM judge in the scoring loop can re-import the same failure mode. REFUTE is a judge-free board for that split on recent science paper summaries critique skill vs calibration . Useful as a foil when deciding where LLM judges help vs where objective axes are safer. Dataset: BGPT-OFFICIAL/refute · Datasets at Hugging Face https://huggingface.co/datasets/BGPT-OFFICIAL/refute Leaderboard: REFUTE Leaderboard - a Hugging Face Space by BGPT-OFFICIAL https://huggingface.co/spaces/BGPT-OFFICIAL/refute-leaderboard