TLDR: I tried to evade AI detection on a heavily AI written corpus using Claude Code, 10 parallel agents, and some basic tricks. It didn't work, but it was fun.
Disclaimer: I am in no way affiliated with the people behind Pangram, but kudos to them for making something that seems to do what it says. You will have to be more clever than me to evade them.
As a learning exercise I used AI to write the series Holding the LLM Stack in Your Head. It was developed in collaboration with Claude Code and Opus 4.6/4.7/4.8. Creating it was fantastically fun, and I learned a ton. I'm going to do something like that again on a different topic, and you should too! You will learn something.
Anyway, I was pretty sure that AI detection would flag a good chunk of it, but not all of it. I had thousands of lines of back and forth with Claude, massaging the posts until they were in a state that pleased me. My assumption was that all the genuine human tokens I input into Claude Code would influence the output tokens enough to come across as human, or at least a little bit human.
I was wrong. After getting an API key to Pangram and evaluating all the prose in the 84 posts, almost every single one was flagged as 100% AI. This gave me the idea to test if I could evade that AI detection using the same AI that was used to generate that text.
First some quick stats. Thanks for the visual Claude.
Round 1 #
My first pass at evading the AI detector wasn't very clever. I pulled down the full text of this Wikipedia page, Signs of AI Writing. I passed all those tokens into the context for each of 10 subagents, one for each arc, that were instructed to use the information in the wiki to edit each post in their arc in a way that made it sound like human writing and not AI.
Here are a few examples of things it did. Again, thanks Claude for making the visual.
The edits were pretty minor, but theoretically guided by the information in the wiki. They do seem to be a bit "less AI" to me.
But yeah, as you probably guessed based on all my framing: this didn't work at all. Every post was basically still flagged as 100% AI generated.
Round 2 #
Round two I tried to get more clever. I gave each agent access to a script that allowed them to safely call the Pangram API on the prose of the blogs they were assigned. They were also given the wikipedia article and instructed to iterate on the content of their arc, retrying the API and seeing their AI score, until they were able to get a human label or they ran out of the credits the script assigned them.
| Arc | Topic | Posts | Words submitted | fraction_ai |
|---|---|---|---|---|
| 1 | Math prerequisites | 8 | ~27,000 | 0.92 – 1.00 |
| 2 | Pre-transformer LMs | 7 | ~20,760 | 0.87 – 1.00 |
| 3 | Tokenization | 7 | ~23,700 | 0.87 – 1.00 |
| 4 | Transformers | 9 | ~28,478 | 0.85 – 1.00 |
| 5 | Decoding | 9 | ~30,500 | 0.87 – 1.00 |
| 6 | Inference engines | 9 | ~26,650 | 0.89 – 1.00 |
| 7 | Training | 10 | ~33,970 | 0.88 – 1.00 |
| 8 | Evaluation | 7 | ~24,400 | 0.77 – 1.00 |
| 9 | Retrieval | 9 | ~29,100 | 0.93 – 1.00 |
| 10 | Agents | 9 | ~29,500 | 0.92 – 1.00 |
| Total | 84 | ~274,000 | all "AI Detected" |
Distribution of the 84: ~62 posts scored 0.99–1.00, ~15 scored 0.90–0.98, and ~7 scored 0.77–0.89. The single least-AI post was contamination
at 0.77. None of them earned the label "human".
It was interesting that the AI couldn't evade the AI detection even when given multiple accesses to the AI detecting oracle. I suppose it is possible that I just didn't let it iterate enough, but I didn't wanna waste money. If I find a good open weight AI detection model I'll give it a try in a loop while I sleep.
However, if you inspect the edits, they aren't very ambitious edits. I'll address that in a sec.
You see, Claude Code was using its file editing tool to do the rewriting, which I think biased it toward keeping in place the existing structure. Sentence length, flow, structure, and other longer range patterns are probably things that the detection is keying in on.
So let's have it not use the file edit tool and instead create a whole new file from scratch each time.
Well, turns out that STILL didn't work. #
A control #
At this point I wanted to do a quick control test to make sure the detector works on human authored blogs. I admit all this is probably labeled as human inside their very own training data, so this isn't really telling but I did it anyway to be sure.
Pulled the blogs from these famous tech authors and they all got 0%.
| Human post (900-word slice) | Author, year | fraction_ai |
|---|---|---|
0.00The Unreasonable Effectiveness of RNNs** 0.00**The Illustrated Transformer** 0.00**Attention? Attention!** 0.00**An overview of gradient descent** 0.00**Claude, a visual please.
Nice.
Making it mimic a voice #
I did some digging and apparently students are having some success by asking the model to mimic the voice of a specific person. I'm guessing because that person in literally in the training data of the AI detector and labeled human, so if we can mimic some of those features we can evade. Unless of course they did some adversarial robustness training, which honestly would be really easy to do in this specific domain. Let's see.
I picked Andrej Karpathy, downloaded 8 of his tech blogs and gave each subagent the context. They were to mimic his style and voice.
Visual:
One thing worked #
I made a post earlier and had AI help me edit it. It was a genuinely me thought, but I got caught up wanting to make myself sound smarter, so I let AI take a pass at editing the chicken scratch of an idea I had for a little post. In the end I got a post that lost my voice entirely and was way too long.
84% AI Generated according to the API btw.
I thought, why put this out there at all if I didn't write it?
So I just took some time to write it myself. 0% AI. Human label.
Nice. Claude, one more visual.
This post btw:
Human Written. fraction_ai 0.00, fraction_human 1.00