{"slug": "show-hn-reproducing-three-ai-text-detectors-and-why-filtering-hurt-my-model", "title": "Show HN: Reproducing three AI text detectors, and why filtering hurt my model", "summary": "A GitHub repository published by abduldattijo reproduces three AI text detectors — perplexity, near-duplicate similarity, and embedding density — plus a sentiment-classifier evaluation on 600 training reviews (200 source and 400 generated) and 200 held-out test reviews, using GPT-2 and the all-MiniLM-L6-v2 sentence-embedding model downloaded from Hugging Face. The package includes an audit script for Mendeley Data dataset version 1, whose IMDb-extracted reviews are treated as human-written because the collection was published in 2019, before ChatGPT launched publicly, though the collection does not verify each review's author and the audit reports detector flags rather than confirmed AI authorship. The author reports that filtering AI-generated text hurt the downstream sentiment model, and the repository ships the exact generated reviews used in that run because regenerating them would require a model API key and the same model versions while still producing different text.", "body_md": "This package contains the exact review files and scripts used for the experiment described in the accompanying article. It lets you rerun the detector comparison, the sentiment-classifier evaluation, and (if you download the source archive) the Mendeley collection audit on your own computer.\n\n- Python 3.11 or newer\n- The packages in `requirements.txt`\n- About 2 GB of free disk space for the first run, when the pretrained models are downloaded\n\nThe first run downloads GPT-2 and the `all-MiniLM-L6-v2` sentence-embedding model from Hugging Face. The detector and classifier then run locally. The included review files preserve the exact generated examples used in the reported run, so they can be used directly in the analysis.\n\nClone the repository, enter its directory, create and activate a virtual environment, then install the dependencies:\n\n```\ngit clone https://github.com/abduldattijo/ai-slop-detection-reproduction.git\ncd ai-slop-detection-reproduction\npython3 -m venv .venv\nsource .venv/bin/activate\npython -m pip install --upgrade pip\npython -m pip install -r requirements.txt\nmkdir -p outputs\n```\n\nScore the 600 training reviews with all four detector scores (perplexity, near-duplicate similarity, embedding density, and their combined score):\n\n```\npython code/run_experiment.py \\\n  --data data/reviews_train.jsonl \\\n  --out outputs/detector_results.json\n```\n\nTrain and evaluate the sentiment classifier using the same four data treatments reported in the article:\n\n```\npython code/downstream_eval.py \\\n  --data data/reviews_train.jsonl \\\n  --results outputs/detector_results.json \\\n  --test-data data/reviews_test.jsonl \\\n  --out outputs/downstream_results.json\n```\n\nThe test file contains 200 source reviews kept separate from training. The scripts write their results under `outputs/`. Compare them with the captured run in `reference_outputs/`. Small numerical differences can occur across operating systems, hardware, or library versions.\n\nDownload version 1 of the source collection from [Mendeley Data](https://data.mendeley.com/datasets/38j8b6s2mx/1). Extract `Dataset.rar` so the extracted directory contains the ten movie folders, then run:\n\n```\npython code/audit_mendeley_collection.py \\\n  --source-dir /path/to/extracted/Dataset \\\n  --train data/reviews_train.jsonl \\\n  --test data/reviews_test.jsonl \\\n  --out outputs/source_audit.json\n```\n\nThe Mendeley reviews are treated as human-written because the collection says they were extracted from IMDb and was published in 2019, before [ChatGPT launched publicly](https://openai.com/index/chatgpt/). The collection does not verify each review's author. The audit reports detector flags, not confirmed AI authorship. Attribution and license details are in `DATASET-ATTRIBUTION.md`.\n\n- `data/reviews_train.jsonl` : 200 source reviews and 400 generated reviews used to fit and score the detectors and train the classifier.\n- `data/reviews_test.jsonl` : 200 source reviews reserved for classifier evaluation.\n- `code/` : the detector, evaluation, and archive-audit scripts.\n- `reference_outputs/` : the saved JSON results used for the article's reported figures and counts.\n\nThe exact generated reviews used in the reported run are included in the training file. This package does not regenerate them: doing so requires a model API key, access to the same model versions, and would still produce different text and potentially different results.", "url": "https://wpnews.pro/news/show-hn-reproducing-three-ai-text-detectors-and-why-filtering-hurt-my-model", "canonical_source": "https://github.com/abduldattijo/ai-slop-detection-reproduction", "published_at": "2026-10-12 10:46:45+00:00", "updated_at": "2026-10-12 11:09:01.738805+00:00", "lang": "en", "topics": ["ai-safety", "machine-learning", "natural-language-processing", "ai-research", "ai-tools"], "entities": ["GitHub", "abduldattijo", "GPT-2", "all-MiniLM-L6-v2", "Hugging Face", "Mendeley Data", "IMDb", "ChatGPT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-reproducing-three-ai-text-detectors-and-why-filtering-hurt-my-model", "markdown": "https://wpnews.pro/news/show-hn-reproducing-three-ai-text-detectors-and-why-filtering-hurt-my-model.md", "text": "https://wpnews.pro/news/show-hn-reproducing-three-ai-text-detectors-and-why-filtering-hurt-my-model.txt", "jsonld": "https://wpnews.pro/news/show-hn-reproducing-three-ai-text-detectors-and-why-filtering-hurt-my-model.jsonld"}}