{"slug": "feedback-on-llm-as-a-judge-design-for-open-source-library", "title": "Feedback on LLM-as-a-Judge design for open-source library", "summary": "The REFUTE dataset and leaderboard, hosted on Hugging Face by BGPT-OFFICIAL, provide a judge-free evaluation board for science paper summaries, addressing the failure mode where LLM-as-a-judge re-imports overclaiming errors. The resource splits critique skill from calibration, serving as a foil for deciding when LLM judges are useful versus when objective axes are safer.", "body_md": "Nice work shipping an open LLM-as-a-Judge API.\n\nOne design pressure we keep hitting on science-critique evals: if the *task itself* is about whether a model overclaims, putting another LLM judge in the scoring loop can re-import the same failure mode.\n\nREFUTE is a judge-free board for that split on recent science paper summaries (critique skill vs calibration). Useful as a foil when deciding where LLM judges help vs where objective axes are safer.\n\nDataset: [BGPT-OFFICIAL/refute · Datasets at Hugging Face](https://huggingface.co/datasets/BGPT-OFFICIAL/refute)\n\nLeaderboard: [REFUTE Leaderboard - a Hugging Face Space by BGPT-OFFICIAL](https://huggingface.co/spaces/BGPT-OFFICIAL/refute-leaderboard)", "url": "https://wpnews.pro/news/feedback-on-llm-as-a-judge-design-for-open-source-library", "canonical_source": "https://discuss.huggingface.co/t/feedback-on-llm-as-a-judge-design-for-open-source-library/166224#post_2", "published_at": "2026-07-24 20:44:43+00:00", "updated_at": "2026-07-24 21:02:27.824418+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety"], "entities": ["BGPT-OFFICIAL", "Hugging Face", "REFUTE"], "alternates": {"html": "https://wpnews.pro/news/feedback-on-llm-as-a-judge-design-for-open-source-library", "markdown": "https://wpnews.pro/news/feedback-on-llm-as-a-judge-design-for-open-source-library.md", "text": "https://wpnews.pro/news/feedback-on-llm-as-a-judge-design-for-open-source-library.txt", "jsonld": "https://wpnews.pro/news/feedback-on-llm-as-a-judge-design-for-open-source-library.jsonld"}}