{"slug": "knowing-what-you-know-a-trait-theory-of-reviewer-value-in-human-ai-cognitive", "title": "Knowing What You Know: A Trait Theory of Reviewer Value in Human-AI Cognitive Systems", "summary": "A paper submitted to arXiv on 18 Nov 2026 argues that a human reviewer's value alongside a capable AI model is governed by metacognitive calibration — the alignment between a reviewer's behavioral confidence and the correctness of their judgments — not by activity, speed, agreement or tenure. Grounded in a six-month production window of an AI-assisted accounting system covering 541,621 reviewer events and 483 reviewers, the authors report that a purely behavioral expertise profile inverts the ground truth, scoring an overconfident cohort (5.2 percent correction survival) above genuine experts (99.2 percent), and that reviewer quality is flat across tenure. Applying the same calibration logic to the model concentrates 92.5 percent of human effort on genuine errors, though the authors note individual causal value is unidentifiable under majority-vote aggregation.", "body_md": "# Computer Science > Artificial Intelligence\n\n  [Submitted on 18 Nov 2026]\n\n# Title:Knowing What You Know: A Trait Theory of Reviewer Value in Human-AI Cognitive Systems\n\n[View PDF](http://arxiv.org/pdf/2611.24932v1)\n\n[HTML (experimental)](https://arxiv.org/html/2611.24932v1)\n\nAbstract:When a capable model and a human reviewer cooperate on the same decision, which human still adds value? We argue this is a question about a single cognitive trait, metacognitive calibration: the alignment between a reviewer's behavioral confidence and the actual correctness of their judgments. We develop a trait theory of reviewer value to make it precise. The theory makes five claims: value is governed by calibration rather than by activity, speed, agreement, or tenure; calibration leaves observable traces in behavioral telemetry; those traces are non-diagnostic on their own, because the fast, low-revision signature of a calibrated expert is indistinguishable from that of a confident novice until it is gated by outcomes; human calibration and a model's self-knowledge are the same construct, so the joint system should defer to whichever agent is better calibrated on each item; and individual calibration is causally identifiable only above the level of the single judgment. We ground each claim in a six-month production window of an AI-assisted accounting system (541621 reviewer events, 483 reviewers), and find that a purely behavioral expertise profile inverts the ground truth, scoring an overconfident cohort (5.2 percent correction survival) higher than genuine experts (99.2 percent); that reviewer quality is flat across tenure, so the trait is not learned on the job; and that the same calibration logic that separates good from bad reviewers, applied to the model, concentrates 92.5 percent of human effort on genuine errors. We close with the boundary the theory must respect: under majority-vote aggregation, individual causal value is unidentifiable.\n    \n\n### Additional Features\n\n### Current browse context:\n\ncs.AI\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/knowing-what-you-know-a-trait-theory-of-reviewer-value-in-human-ai-cognitive", "canonical_source": "http://arxiv.org/abs/2611.24932v1", "published_at": "2026-11-19 16:24:31+00:00", "updated_at": "2026-11-19 16:45:01.263558+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-safety", "machine-learning"], "entities": ["arXiv", "cs.AI"], "also_reported_by": [], "takeaway": "Gate behavioral expertise signals on outcomes before trusting them: in a 541,621-event accounting deployment, a purely behavioral profile ranked an overconfident cohort (5.2 percent correction survival) above genuine experts (99.2 percent), while calibration-based deferral put 92.5 percent of human ", "recommendation": {"advice": "Score human reviewers by outcome-gated calibration, not speed or low revision counts, since behavioral-only expertise profiles inverted expert ranking in a 483-reviewer production system.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/knowing-what-you-know-a-trait-theory-of-reviewer-value-in-human-ai-cognitive", "markdown": "https://wpnews.pro/news/knowing-what-you-know-a-trait-theory-of-reviewer-value-in-human-ai-cognitive.md", "text": "https://wpnews.pro/news/knowing-what-you-know-a-trait-theory-of-reviewer-value-in-human-ai-cognitive.txt", "jsonld": "https://wpnews.pro/news/knowing-what-you-know-a-trait-theory-of-reviewer-value-in-human-ai-cognitive.jsonld"}}