Reverse Engineering GPTZero: Verdicts are Document-Level, Non-Additive, and Order-Dependent

The team behind Penned, a free AI humanizer and detector, benchmarked GPTZero's v3 API (version 2026-10-07-base) and found its verdicts are document-level, non-additive and order-dependent rather than averaged sentence scores. Adding a single paragraph or table caption near the 0.5 threshold flipped whole documents from human to AI, after which every sentence was flagged, and swapping one paragraph in a paper's results section dropped its AI score from 100% to 40%. The team concluded a greedy per-paragraph candidate picker cannot work against such a non-linear decision boundary and is now testing calibration pipelines in public.

Over the past month building Penned https://penned-ai.vercel.app —a free AI humanizer and detector designed to rewrite AI drafts into natural prose while strictly preserving facts, names, and numbers—we ran into a fascinating, counter-intuitive reality when evaluating our candidate picker against GPTZero v3 API version 2026-10-07-base . Most people and even many ML practitioners imagine AI text detectors as sentence-level statistical classifiers: each sentence has a perplexity or burstiness score, and the document score is simply an average or a cumulative tally. Our empirical benchmarks on research paper drafts revealed something completely different: GPTZero's verdicts are document-level, non-additive, and heavily order-dependent. Here are 7 empirical findings from our evaluation logs eval/gptzero/FINDINGS.md , and what they mean for anyone building or evaluating NLP systems today. Near the 0.5 decision threshold, GPTZero exhibits an extreme non-linear cliff. Adding just one extra paragraph or even a single table caption flipped an entire document's classification from human to ai . Once that threshold was crossed, the API highlighted every single sentence as AI-generated—including sections that scored 0% AI when evaluated in isolation. The detector is not simply aggregating independent sentence probabilities; it appears to classify the whole document context jointly, and then distributes retroactive confidence across the text. The most striking behavior was in section combinations: Even wilder was the sensitivity to ordering. We took the results section of an academic paper: Furthermore, swapping just one paragraph section 4.3 for an alternative Penned-generated candidate flipped the entire real-order paper half from 100% AI down to 40% AI human . This demonstrates that document-level classification is governed by non-linear transition dynamics rather than bag-of-words or simple token perplexity. When inspecting GPTZero's generated prob at the sentence level, the scores are not smooth. GPTZero internally segments documents into structural blocks. Even in a document overall classified as human e.g. 39% AI , individual AI segments remain clearly visible: The global document score appears to weigh these segmented blocks against each other rather than averaging raw sentence scores. Numerical specificity has an outsized effect on detection: 42.8% with N% turned a AI detectors heavily associate dense, concrete domain measurements and precise numerical observations with authentic human scientific writing. How text is pasted or extracted matters significantly: Summarizing findings is inherently dangerous under modern detectors: Because LLMs are trained to summarize and synthesize, detectors have learned that synthetic summaries look overwhelmingly "machine-like", even when written by humans. We benchmarked several in-house and open-source detection models against the exact same test suite: In short: local open-source detectors cannot see what proprietary v3 detectors see. When we started building Penned https://penned-ai.vercel.app , our initial pipeline scored rewritten candidates paragraph-by-paragraph using local DeBERTa and a logistic regression GPT-2 feature checker. These findings showed us that a greedy per-paragraph picker cannot work . Because the detector's decision boundary is non-additive and order-dependent: We are currently building open benchmarks and testing these calibration pipelines in public. Penned is free to test at penned-ai.vercel.app https://penned-ai.vercel.app . Check it out and let us know your thoughts on our findings