Over the past month building Penned—a free AI humanizer and detector designed to rewrite AI drafts into natural prose while strictly preserving facts, names, and numbers—we ran into a fascinating, counter-intuitive reality when evaluating our candidate picker against GPTZero v3 API (version 2026-10-07-base).
Most people (and even many ML practitioners) imagine AI text detectors as sentence-level statistical classifiers: each sentence has a perplexity or burstiness score, and the document score is simply an average or a cumulative tally.
Our empirical benchmarks on research paper drafts revealed something completely different: GPTZero's verdicts are document-level, non-additive, and heavily order-dependent.
Here are 7 empirical findings from our evaluation logs (eval/gptzero/FINDINGS.md), and what they mean for anyone building or evaluating NLP systems today.
Near the 0.5 decision threshold, GPTZero exhibits an extreme non-linear cliff.
Adding just one extra paragraph or even a single table caption flipped an entire document's classification from human to ai. Once that threshold was crossed, the API highlighted every single sentence as AI-generated—including sections that scored 0% AI when evaluated in isolation.
The detector is not simply aggregating independent sentence probabilities; it appears to classify the whole document context jointly, and then distributes retroactive confidence across the text.
The most striking behavior was in section combinations:
Even wilder was the sensitivity to ordering. We took the results section of an academic paper:
Furthermore, swapping just one paragraph (section 4.3) for an alternative Penned-generated candidate flipped the entire real-order paper half from 100% AI down to 40% AI (human). This demonstrates that document-level classification is governed by non-linear transition dynamics rather than bag-of-words or simple token perplexity.
When inspecting GPTZero's generated_prob at the sentence level, the scores are not smooth. GPTZero internally segments documents into structural blocks.
Even in a document overall classified as human (e.g. 39% AI), individual AI segments remain clearly visible:
The global document score appears to weigh these segmented blocks against each other rather than averaging raw sentence scores.
Numerical specificity has an outsized effect on detection:
42.8% with N%) turned a AI detectors heavily associate dense, concrete domain measurements and precise numerical observations with authentic human scientific writing.
How text is pasted or extracted matters significantly:
Summarizing findings is inherently dangerous under modern detectors:
Because LLMs are trained to summarize and synthesize, detectors have learned that synthetic summaries look overwhelmingly "machine-like", even when written by humans.
We benchmarked several in-house and open-source detection models against the exact same test suite:
In short: local open-source detectors cannot see what proprietary v3 detectors see.
When we started building Penned, our initial pipeline scored rewritten candidates paragraph-by-paragraph using local DeBERTa and a logistic regression GPT-2 feature checker.
These findings showed us that a greedy per-paragraph picker cannot work. Because the detector's decision boundary is non-additive and order-dependent:
We are currently building open benchmarks and testing these calibration pipelines in public.
Penned is free to test at penned-ai.vercel.app. Check it out and let us know your thoughts on our findings!