The Consensus Trap #
The most dangerous part of this experiment was the 86% agreement rate between the four judges. High consensus usually suggests reliability, but here, the judges shared the same learned bias. They weren't making independent errors; they were converging on the same wrong answers. To make matters worse, swapping the presentation order of the text flipped the verdict in about a third of the cases—a classic position bias.
The judges' rationales revealed a weird logic: they equated detail density with humanity and ordinariness with fakeness. AI-polished text with perfect structure and precise numbers felt "lived-in" to the models, while human filler and lazy adjectives felt "generic."
Model Performance and Calibration #
I tried to fix this with a debiasing preamble, explicitly telling the models that detail density doesn't equal human origin. The results varied wildly by model family:
Jumped from 12% to 83% accuracy. The bias was at the prompt layer and could be steered.Claude-family:DeepSeek, GLM, and Qwen: Stayed stuck between 25% and 33%. For these models, the bias is baked deeper into the weights than a prompt can reach.
My Current AI Workflow for Quality Control #
After seeing how unreliable LLM-as-a-judge can be for "human feel," I've shifted to a three-layer stack for a more practical tutorial on quality gating:
-
Rule-based detector: I use phrase blacklists and sentence-pattern quotas calibrated against real user judgments.
-
Human spot checks: This remains the only true ground truth.
-
LLM blind review: Only used for relative "before vs. after" comparisons of the same text, never for absolute "human or AI" labels.
The biggest takeaway? Never use an LLM judge as an optimization target unless it has passed a gold calibration (≥80% accuracy on human-consensus anchors). Otherwise, you're just optimizing your content to sound more like an LLM's idea of a human.
Next LLM Judges: Accuracy vs Model Size →