Detecting LLM-Generated Texts with "Classical" Machine Learning

A developer built an AI-generated text detector using classical machine learning, achieving 85% single-sentence accuracy on a test set. The tool, available as an online demo and open-source on GitHub, uses scikit-learn's SVM to distinguish LLM-generated text from human writing by exploiting statistical patterns. The project was motivated by the proliferation of low-quality AI fanfiction on platforms like Lofter.

This article is currently an experimental machine translation and may contain errors. If anything is unclear, please refer to the original Chinese version. I am continuously working to improve the translation. TL;DR & Demo As of early 2026, mainstream LLM-generated text exhibits strong statistical patterns that can be effectively distinguished from human-written content using traditional machine learning models. I suspect this is how many so-called “AI plagiarism checkers” actually work under the hood. Online Demo: https://lyc8503.github.io/AITextDetector/ https://lyc8503.github.io/AITextDetector/ The model used in this demo is not trained on general-purpose data , nor has it undergone rigorous optimization or iteration. Its single-sentence detection accuracy is approximately 85% on the test set. Please read through this article before use to understand potential limitations. The core code drafts and trained model files are available on GitHub: lyc8503/AITextDetector https://github.com/lyc8503/AITextDetector Background aka Useless Rambling Back when I was still writing my thesis at school half a year ago, rumors were already spreading about checking papers for AIGC AI-generated content . I tested several platforms—CNKI, Wanfang, and some third-party AIGC detection services—and found they could indeed distinguish between my hand-written text and LLM-generated text with decent accuracy. That sparked my curiosity about how AIGC detection actually works~~ and how to bypass it ~~. But I was juggling too many things at the time—obsessed with radio https://blog.lyc8503.net/categories/%E7%A1%AC%E4%BB%B6%E3%81%AE%E7%A0%94%E7%A9%B6/%E6%97%A0%E7%BA%BF%E5%B0%84%E9%A2%91/ , Minecraft, Touhou—and after a few failed attempts, I shelved the idea. Eventually, I faked my way through the thesis, and life moved on. But recently, while browsing Lofter, I stumbled upon entire tags flooded with low-quality, wildly out-of-character AI-generated fanfics. How can I tell they’re AI at a glance? Well, some folks or gals don’t even bother cleaning up Markdown formatting or AI-generated section headers before posting—and then they slap half the article behind a paywall 😓 Most AI-generated texts, however, are harder to spot—they’re buried among diverse writing styles, varied prompts, and not immediately obvious. By the time you realize something feels off, it’s too late. Some texts are borderline impossible to prove as AI, leaving me paranoid. After swallowing a few too many AI-generated turds, I’d finally had enough. Lofter browsing stops here—time to open VS Code Yes, that’s how I ended up reviving my weekend project idea: building an AI-generated text detector… Research Attempt – No Luck The internet is now almost entirely polluted with ads when searching for AIGC detection. Every result is just another essay-AI-rewriting service. Back then, I dug through the noise and found something called text perplexity https://zhuanlan.zhihu.com/p/114432097 . The idea is simple: use an existing LLM to estimate the probability of each word appearing in a given sentence. If nearly every word ranks high in the LLM’s predictions Top-N , the sentence is likely AI-generated. Conversely, if many words are unexpected, it’s more likely human-written. Sounds promising, right? I spent some time trying this method, but results were disappointing—plenty of false positives and false negatives, and no reasonable threshold could be set. Plus, there are practical issues: high inference cost, poor cross-model generalization, difficulty deploying large models locally, and closed-weight models being hard to integrate. Overall, this approach isn’t elegant or reliable. A Somewhat Successful Attempt – scikit-learn SVM Since online resources were useless, back to old-school alchemy. Scikit-learn, activate Following its Roadmap https://scikit-learn.org/stable/machine learning map.html , we can directly pick Linear SVC and Naive Bayes as good starting points for our classification task. Whisper: this also matched my gut feeling—LLMs have detectable word-choice patterns; even a Naive Bayes classifier should pick them up. I just didn’t expect the signal to be this strong. Data Generation ~~Old-school alchemy~~ traditional classifiers need labeled data—so we need human-written texts and confirmed LLM-generated texts for training. My approach: I pulled data I’d scraped in 2023 from a certain Ford-like and River-like platform, filtering for articles published between 2010–2022 pre-ChatGPT . I only filtered out extremely low-engagement or very short pieces, then randomly sampled nearly 10,000 multi-thousand-character texts as human-written samples. Then, I used an LLM to generate chapter summaries of these texts, fed the summaries back into the LLM, and had it regenerate full articles. This gave me a roughly equal number of LLM-generated samples, diverse in genre and closely matching the original human content. In theory, at least. But LLM APIs are expensive, and I wasn’t about to spend thousands on a weekend project. So I got creative—and ~~skirted the rules~~ leveraged multiple low-cost or free API channels: Gemini : Used CLIProxyAPI https://github.com/router-for-me/CLIProxyAPI to convert Antigravity/Gemini CLI quota into API access—just pay ~$20 for an AI Pro account Qwen : qwen-code https://github.com/QwenLM/qwen-code lets you reverse-engineer the Qwen Plus API—free GLM-5 : Lucky timing—OpenRouter was offering free GLM-5 public beta Pony Alpha Kimi, Deepseek, Doubao, GLM-4.7 : Signed up during a promotional coding plan—$8.9 first month, API access unlocked ~~Disclaimer: This is not a recommendation. These actions violate platform ToS and may get you banned. But the platforms are too busy with marketing hype to care, and I wasn’t about to pay full price.~~ Many programming-focused LLM APIs strangely charge per call, but we can ~~abuse~~ optimize this by batching tasks into massive inputs, forcing the LLM to generate more content per call. And so… Ultimately, I used gemini-3-flash to generate summaries, and seven different models gemini-3-pro , qwen-coder-plus , glm-5 , glm-4.7 , kimi-k2.5 , doubao-seed-code , deepseek-v3.2 to generate seven sets of LLM-generated samples. Training While I was halfway through data generation, I couldn’t wait and started training. I asked Claude to write the classifier code, and it naively dumped the entire raw text into the model—achieving a suspicious 99.45% accuracy… Wait, really? ~~Claude’s useless. I’ll do it myself.~~ For training, I split all texts into sentences using Chinese punctuation, cleaned non-Chinese/English characters, then applied scikit-learn’s TF-IDF → LinearSVC . After cleaning up some noise, sentence-level classification still hit ~85% accuracy Individual sentences carry limited info, but 85% accuracy per sentence means that for a longer article, we can be highly confident in judging whether it’s AI-generated. This performance far exceeded my expectations. Old-school ML still slaps—way better than those dumb online tools that just ask an LLM, “Hey, is this text AI-generated?” After finishing all data, I tried training an 8-class model human + 7 AIs , but the LLMs seem too similar—probably distilled from each other—so classification was messy, with only ~50% accuracy. Eventually, I trained seven separate binary classifiers and used majority voting: a sentence is flagged as AI if ≥2 models detect it. 1234567891011121314151617181920212223242526272829303132333435 | loaded 8536 samplestrain chapter size: 6820, test chapter size: 1716 gemini Train: 917,374 Test: 228,051 gemini full TF-IDF + SVC ... 3,336,446 features - acc=0.8809 f1=0.8082 tn=143688 fp=10650 fn=16503 tp=57210 qwen Train: 1,315,338 Test: 328,636 qwen full TF-IDF + SVC ... 3,989,603 features - acc=0.8911 f1=0.8974 tn=136293 fp=18045 fn=17739 tp=156559 pony Train: 1,128,044 Test: 278,663 pony full TF-IDF + SVC ... 3,688,143 features - acc=0.8493 f1=0.8286 tn=135085 fp=19253 fn=22755 tp=101570 kimi25 Train: 1,088,007 Test: 269,567 kimi25 full TF-IDF + SVC ... 3,976,027 features - acc=0.8721 f1=0.8473 tn=139390 fp=14948 fn=19534 tp=95695 glm47 Train: 1,124,430 Test: 279,109 glm47 full TF-IDF + SVC ... 3,980,772 features - acc=0.8436 f1=0.8222 tn=134461 fp=19877 fn=23786 tp=100985 doubao Train: 1,063,395 Test: 264,121 doubao full TF-IDF + SVC ... 4,243,728 features - acc=0.8940 f1=0.8700 tn=142420 fp=11918 fn=16089 tp=93694 deepseekv32 Train: 1,176,294 Test: 289,042 deepseekv32 full TF-IDF + SVC ... 4,361,691 features - acc=0.8529 f1=0.8403 tn=134625 fp=19713 fn=22819 tp=111885 ===================================== SUMMARY===================================== model s1 acc s1 f1 gemini 0.8809 0.8082 qwen 0.8911 0.8974 pony 0.8493 0.8286 kimi25 0.8721 0.8473 glm47 0.8436 0.8222 doubao 0.8940 0.8700 deepseekv32 0.8529 0.8403 | All models achieved over 85% accuracy and over 80% F1—pretty solid I also noticed that AI-generated texts were often flagged by multiple models, so voting made perfect sense. I tried MultinomialNB and SGDClassifier , but accuracy dropped slightly. BERT gave a minor boost but required too much GPU time—discarded. Even tested AutoGluon , which somehow managed only 53% binary accuracy. Won’t dive into those. JS Implementation for Web Demo At this point, I could’ve just published the repo and called it a day. But launching Python every time is way too inconvenient. I could’ve hosted a Python API, but that means server maintenance—violates my strict Serverless philosophy. My original plan: export model to ONNX, run inference via ONNX Web Runtime in Wasm. But when I asked my silicon servant Claude to help, I didn’t specify clearly—and it went off-script, trimming and exporting the model as a JSON… then implemented TF-IDF + SVM entirely in JavaScript for browser inference. Hmm… actually not a bad idea. I tested it on a 1-million-character text—it took about 10 seconds on my machine, acceptable. For typical few-thousand-character inputs, it’s instant. Fine, since this is just a demo, and the JS approach is more transparent, I’ll keep this ~~slightly silly~~ implementation. Blame Claude, not me. As for accuracy: I tested different feature limits. Ultimately prioritized performance and kept 500k features. Stored as JSON, it’s a bloated 107MB though gzipped server-side, it’s ~38MB . Smaller versions 50k–80k only lost 3–4% accuracy, but final AI detection rates varied significantly—especially on human texts, with ±50% relative differences, leading to false positives. So I stuck with 500k. Final accuracy drop: ~1%, as shown below: 1234567891011121314 | ============================================================ SUMMARY top-500,000 C=1.0============================================================ model s1 acc s2 acc Δacc s1 f1 s2 f1 Δf1 gemini 0.8809 0.8721 -0.0088 0.8082 0.7986 -0.0096 qwen 0.8911 0.8819 -0.0092 0.8974 0.8886 -0.0088 pony 0.8493 0.8383 -0.0109 0.8286 0.8173 -0.0113 kimi25 0.8721 0.8623 -0.0098 0.8473 0.8376 -0.0097 glm47 0.8436 0.8311 -0.0125 0.8222 0.8097 -0.0125 doubao 0.8940 0.8869 -0.0071 0.8700 0.8624 -0.0076 deepseekv32 0.8529 0.8419 -0.0110 0.8403 0.8291 -0.0112 AVG 0.8592 0.8348 MIN acc: 0.8311============================================================ | Testing Performance All tests below use the pruned web version, which should perform similarly to the full joblib models. Current logic: split input text into sentences, clean and classify using all 7 binary models. If ≥2 models flag a sentence, it’s marked as suspected AI and highlighted. Final AI score is the proportion of flagged characters. Classification: - <50%: Human - 50–70%: Maybe Human 70%: Maybe AI First, test detection rate on common models like Doubao and Deepseek—both were in training data. Prompt: write me a 3000-word story . Easily caught: Now test on unseen models—how’s generalization? I tested several other models not in training MiMo-V2, Doubao-Seed-2.0, GPT-4o —all detected at ~70%, some even 90%. Solid. Also tested more complex prompts—e.g., feeding 20 chapters of human-written text and asking the LLM to mimic style and continue. Detection rate dipped slightly to 67.8% but remember, we trained on complex prompts too . Results not shown due to space. Then I picked 10 completed web novels pre-2022 from my subscription list—diverse genres, authors, eras, and likely not in training data. Their AI detection rates: 22.7%, 24.2%, 25.0%, 24.5%, 19.0%, 13.7%, 29.1%, 4.9%, 27.3%, 19.2%—all under 30%. I also sampled random Lofter fanfics; since they’re more casual, detection rates were often below 10%. But when I fed in texts I suspected were AI-generated, detection spiked to 83.4%, strongly suggesting LLM use without disclosure. Mar 5, 2026 Update For more rigorous testing, I randomly sampled 10,000 high-engagement views 5000 , long-form word count 2000 fanfics from Lofter, all posted before 2022. Their AI detection rate distribution using 7-model voting, ≥2 votes : 1 | 0-5%:313 | 5-10%:1945 | 10-15%:3016 | 15-20%:2033 | 20-25%:1355 | 25-30%:594 | 30-35%:492 | 35-40%:123 | 40-45%:34 | 45-50%:62 | 50-55%:24 | 55-60%:5 | 60-65%:3 | 65-70%:1 | Using 60% as threshold → false positive rate: 0.04% Using 70% → false positive rate: <0.01% effectively zero The four texts above 60% were all collection indexes, not actual stories—flagged due to excessive links. Even at 50% threshold, false positive rate is only 0.33% . Then, I scraped all articles from Lofter Android’s top 20 trending tags weekly榜单 , filtered by length, and ran detection: 1 | 0-5%:27 | 5-10%:138 | 10-15%:231 | 15-20%:245 | 20-25%:238 | 25-30%:137 | 30-35%:112 | 35-40%:87 | 40-45%:116 | 45-50%:112 | 50-55%:118 | 55-60%:157 | 60-65%:118 | 65-70%:109 | 70-75%:75 | 75-80%:56 | 80-85%:28 | 85-90%:15 | 90-95%:10 | 32.22% of articles scored 50% AI —likely partially or fully AI-generated… Is there any human left?? Moreover, not a single one has proactively declared AI-generated content. “Age of Dharma’s decline, , ,” —a friend in the group Attack and Defense – Bypassing Detection ? Alright, we’ve built an AIGC detector. ~~Time to build an anti-detector now.~~ Nope, kidding. I’m not that bored. But let’s test some common anti-AIGC detection tricks: Classic Translation Method Google Translate roundtrip CN→EN→CN : 89.9% → 85.0% Youdao Translate CN→EN→CN : 89.9% → 79.2% Sogou Translate CN→EN→CN : 89.9% → 86.0% Slight drop, but still clearly flagged. LLM Prompt Method Use “magic” prompts to make LLM output less “AI-like”—sounds ridiculous from the start Tested one-line prompt: Rewrite the above article to minimize AI flavor : 89.9% → 83.0% Also tried more complex prompts https://github.com/hylarucoder/ai-flavor-remover : 89.9% → 79.3% Slight improvement, but still meaningless in practice. This detection method is way too robust ~~ flag~~ If I really wanted to bypass it, my only ideas would be fine-tuning an LLM on massive human text, or building a huge rule-based system to surgically disrupt SVM-matched features. But that’s beyond this article. Not sure if it’d even work. ~~Or maybe there are better ways—left as an exercise for the reader.~~ Epilogue Now, time for closing rambling. I honestly didn’t expect this classification task to be so easy—simple enough that a scikit-learn “Hello World” script, with minimal iteration and some hardcoded rules, could produce a fairly robust and accurate detector. Most of the effort was just waiting for LLMs to generate data… Ambitious readers could follow this approach to train detectors for other domains—say, academic paper AIGC detection. Add a flashy frontend, and you’ve got a tool you can sell to desperate college students. ~~If you make money, don’t forget to donate a little.~~ Another idea: detect AI-generated images. But with Stable Diffusion and easy LoRA fine-tuning, visual styles are far more diverse than text—this task would be much harder. And after writing this, my three-minute enthusiasm is burned out. Maybe next time. Lastly, a few words on AI-generated content: I don’t accept AI-generated entertainment as legitimate creative work. Just like AI coding tools produce bloated, unmaintainable code, AI-generated text, images, and audio may seem decent at first glance, but fall apart on closer inspection—repetitive, shallow, and so predictable that even word frequency stats can catch them. This pattern is fundamentally unsuitable for real creation, and as a reader, I’m deeply unsatisfied. I’m starting to suspect LLMs’ so-called “creative writing” is just a bunch of post-training data being endlessly recombined and regurgitated. But then again, “the world should be” has never equaled “the world is.” While LLMs bring innovation and productivity gains, misuse and abuse are spreading relentlessly across every industry. And since LLMs are fine-tuned to exploit human perception, who knows whether they “understand” anything or just memorize patterns? After patching endless bugs like “which is bigger, 3.9 or 3.11” or “should I walk or drive 50m to a car wash,” can we really say the model understands the world? Everyone’s stuck in debates: What is LLM? How will it disrupt my industry? Where will AI take humanity? No one has answers. At least I’m glad I learned to code before the AI era. Otherwise, I might not realize how stupid today’s vibe-coded software really is. As for the future? Either generative AI smashes human social order to pieces, or the AI bubble bursts and memory becomes free. Either way, sounds fine to me, doesn’t it? This article is licensed under the CC BY-NC-SA 4.0 https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en license. Author: lyc8503, Article link: https://blog.lyc8503.net/en/post/llm-classifier/ https://blog.lyc8503.net/en/post/llm-classifier/ If this article was helpful or interesting to you, consider buy me a coffee https://ko-fi.com/lyc8503 ¬ ¬ Feel free to comment in English below o/