tl;dr: as a weekend project, i trained a 4b model with reinforcement learning to rewrite & “unslop” text to beat ai detectors. i got claude to generate this full blogpost and then got the above finetuned model to rewrite it. here’s the training run.
ai slop
i have been doing a lot of writing recently and like many people, ai models are a regular part of my workflow. it’s useful to get past the blank page block & flesh out disconnected ideas.
the problem though is that the writing sounds like ai.
it is not necessarily bad writing. it is just generic writing. commonly referred to as ai slop.
so i wondered: could i train an ai model to unslopify text i.e., rewrite text in a way that sounded more human by removing the common “ai slop” patterns from ai models and doing this while preserving the original content, structure and tone.
this felt like a nice reinforcement learning problem (and a fun weekend project): use an ai detector as a reward function and optimize against it. here are some notes on the things i tried.
the data + setup
i first created a synthetic dataset of ai slop. the pipeline had two steps: one llm first generated a scenario (e.g., slack message, email to superior, student essay, etc.). second, another llm took that scenario and actually fleshed out the text. my goal was to train an llm that can rewrite this text in a more human way.
simple start
for my first version, i started with a single ai-detector (slop-guard) as the reward function. slop-guard is “a rule-based prose linter that scores text 0–100 for formulaic ai writing patterns.” it’s purely programmatic, with “24 configurable rules backed by 200+ literal and structural heuristics.” it returns a numeric score based on which rules were violated. so effectively, the fewer the violations, the higher the reward.
unfortunately, this reward was quite hackable → the model just collapsed to producing simple short sentences while not preserving any of the original content.
original content:
> After years of declining enrollment, the university announced that it would
> close three humanities departments, eliminate 40 faculty positions, and
> redirect funding toward engineering and computer science. Students protested
> that the decision would disproportionately harm low-income students who
> relied on those programs.
reward-hacked rewrite:
> The university is closing three humanities departments.
to fix this, i added another reward function to measure content preservation. specifically, i passed the original content and the rewritten content to an llm judge to grade how well information was preserved. that helped a lot with semantic preservation.
extending to probabilistic ai detectors
slop-guard is a good start, but it’s too dependent on programmatic rules. so i decided to add other techniques. i started with a fine-tuned bert model off huggingface (modernbert-ai-detection-raid-mage). training with it generally looked good, but i noticed that the model tended to rewrite text in a very casual style, even if the source was formal. to get around this, i added another tone preservation reward → specifically using another llm judge to detect any deviations in tone.
original content:
> The committee concluded that the proposed policy would impose substantial
> administrative costs without producing a corresponding improvement in
> public outcomes. It therefore recommended that implementation be postponed
> pending further review.
rewrite:
> so the committee was basically like, "This is going to be a huge pain to
> manage, and it probably won't even help much." So they said, "Let's hold
> off and take another look."
throwing the kitchen sink
with the above working, i decided to throw the kitchen sink at the problem. i aggregated a whole bunch of ai detectors in the open-source world and used their combined output as a reward function. this included diveye, tmr, bert-tiny-raid, etc.
while the above model is pretty decent at writing text in a way that open-source ai detectors said was human, i did notice that it wasn’t so good with closed ones like pangram, etc. (kudos to them!)
next steps
scaling up to a 35b model didn’t help with cross-detector generalization either. more work needs to be done here around improving generalizability to closed-source detectors. some ideas i have here:
- the naive option: just spend lots of $$s and train directly with closed detectors
- distill detection from closed detectors into a separate model and use that as a reward metric
- vary detector weights, thresholds, prompts, and sampling settings throughout training
- train in a gan-like loop where the rewriter learns to fool a continually updated detector, while the detector is retrained on the rewriter’s latest outputs
takeaways #
rl is very iterative. the first reward usually gets hacked, so most of the work is in inspecting outputs, finding failure modes, and improving the reward functions. better models helped less than better rewards, better data, and better evals 🙂.