Show HN: Reproducing three AI text detectors, and why filtering hurt my model

A GitHub repository published by abduldattijo reproduces three AI text detectors — perplexity, near-duplicate similarity, and embedding density — plus a sentiment-classifier evaluation on 600 training reviews (200 source and 400 generated) and 200 held-out test reviews, using GPT-2 and the all-MiniLM-L6-v2 sentence-embedding model downloaded from Hugging Face. The package includes an audit script for Mendeley Data dataset version 1, whose IMDb-extracted reviews are treated as human-written because the collection was published in 2019, before ChatGPT launched publicly, though the collection does not verify each review's author and the audit reports detector flags rather than confirmed AI authorship. The author reports that filtering AI-generated text hurt the downstream sentiment model, and the repository ships the exact generated reviews used in that run because regenerating them would require a model API key and the same model versions while still producing different text.

This package contains the exact review files and scripts used for the experiment described in the accompanying article. It lets you rerun the detector comparison, the sentiment-classifier evaluation, and if you download the source archive the Mendeley collection audit on your own computer. - Python 3.11 or newer - The packages in requirements.txt - About 2 GB of free disk space for the first run, when the pretrained models are downloaded The first run downloads GPT-2 and the all-MiniLM-L6-v2 sentence-embedding model from Hugging Face. The detector and classifier then run locally. The included review files preserve the exact generated examples used in the reported run, so they can be used directly in the analysis. Clone the repository, enter its directory, create and activate a virtual environment, then install the dependencies: git clone https://github.com/abduldattijo/ai-slop-detection-reproduction.git cd ai-slop-detection-reproduction python3 -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip python -m pip install -r requirements.txt mkdir -p outputs Score the 600 training reviews with all four detector scores perplexity, near-duplicate similarity, embedding density, and their combined score : python code/run experiment.py \ --data data/reviews train.jsonl \ --out outputs/detector results.json Train and evaluate the sentiment classifier using the same four data treatments reported in the article: python code/downstream eval.py \ --data data/reviews train.jsonl \ --results outputs/detector results.json \ --test-data data/reviews test.jsonl \ --out outputs/downstream results.json The test file contains 200 source reviews kept separate from training. The scripts write their results under outputs/ . Compare them with the captured run in reference outputs/ . Small numerical differences can occur across operating systems, hardware, or library versions. Download version 1 of the source collection from Mendeley Data https://data.mendeley.com/datasets/38j8b6s2mx/1 . Extract Dataset.rar so the extracted directory contains the ten movie folders, then run: python code/audit mendeley collection.py \ --source-dir /path/to/extracted/Dataset \ --train data/reviews train.jsonl \ --test data/reviews test.jsonl \ --out outputs/source audit.json The Mendeley reviews are treated as human-written because the collection says they were extracted from IMDb and was published in 2019, before ChatGPT launched publicly https://openai.com/index/chatgpt/ . The collection does not verify each review's author. The audit reports detector flags, not confirmed AI authorship. Attribution and license details are in DATASET-ATTRIBUTION.md . - data/reviews train.jsonl : 200 source reviews and 400 generated reviews used to fit and score the detectors and train the classifier. - data/reviews test.jsonl : 200 source reviews reserved for classifier evaluation. - code/ : the detector, evaluation, and archive-audit scripts. - reference outputs/ : the saved JSON results used for the article's reported figures and counts. The exact generated reviews used in the reported run are included in the training file. This package does not regenerate them: doing so requires a model API key, access to the same model versions, and would still produce different text and potentially different results.