How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure

A self-audit of LLM-based prompt-structure inference across eight open model variants spanning five families and 8B to 675B parameters questions how much confidence ranked evaluation tables deserve when evaluations routinely average over small prompt sets. The study examines the reproducibility of evaluation conclusions drawn from LLM-inferred prompt structure.

Evaluations of LLM systems routinely average over small prompt sets and report models as a ranked table. We ask how much confidence such a table deserves, using LLM-based prompt-structure inference as the case study: eight open model variants across five families and 8B to 675B parameters, caching d