Beyond Left vs. Right: Engineering Reproducible Detection of Media Bias and Propaganda

A developer built a forensic media analyzer that scores articles on rhetorical structure rather than political bias, drawing on rhetoric, framing theory, and propaganda detection. The tool, exemplified by Rhetoric Audit, aims to detect manipulation techniques such as emotional loading, omission, and logical fallacies without labeling content as left or right.

We have spent a decade building machines that generate persuasive text and almost no time building machines that audit it. That asymmetry is starting to hurt. False news spreads "farther, faster, deeper, and more broadly" than the truth — Vosoughi, Roy, and Aral demonstrated this at scale across roughly 126,000 stories on Twitter, and the effect was driven by humans, not bots 1 . If the demand side of manipulation is human psychology, then the supply side is now industrialized by language models. The tooling that helps a reader think clearly has to be at least as sophisticated as the tooling that helps a writer persuade. This essay is about how to build that auditing tool: a forensic media analyzer that treats an article the way a lab treats a tissue sample — measuring what is structurally present rather than voting on whether it feels biased. I'll cover the science it rests on, the engineering problems that make it genuinely hard LLM non-determinism chief among them , a practical build guide, and an honest account of the limitations. Some of the design decisions come from work we've done on Rhetoric Audit https://www.rhetoricaudit.com/ , a browser-based forensic media evaluator; most of the ideas are older than any of us and belong to rhetoric, argumentation theory, and cognitive science. The problem with "bias detectors" Most bias tools answer the wrong question. They place an outlet or an article on a single left–right line and call it a day. That axis is politically legible but analytically hollow. It flattens a dozen independent phenomena — emotional loading, source quality, logical structure, omission, framing — into one number that tells you almost nothing about how a piece works on a reader. Worse, a left–right label invites the reader to do the one thing forensic analysis is supposed to prevent: agree or disagree based on tribe. Wardle and Derakhshan, in their influential Council of Europe report on "information disorder," argue for separating mis-information false, no intent to harm , dis-information false, intentional , and mal-information true, weaponized precisely because lumping them together destroys our ability to respond appropriately 2 . A single spectrum can't hold those distinctions. Neither can it distinguish a factually careful opinion column from a factually reckless one that happens to share its politics. The reframe is simple to state and hard to implement: score structure, not stance. Two articles can reach opposite conclusions and be rhetorically identical; two articles can reach the same conclusion with wildly different levels of manipulation. A forensic analyzer should be able to say "this piece leans heavily on fear appeals, omits the strongest counterargument, and contains three instances of the strawman fallacy" without ever telling you whether it's right. The science underneath The intellectual scaffolding is surprisingly old. Rhetoric. Aristotle's Rhetoric gave us the three modes of persuasion — ethos credibility , pathos emotion , and logos logic — and they remain a shockingly good decomposition of how a text tries to move you 3 . A forensic system can score every passage on the mix of these three appeals and immediately surface the pieces that lean on feeling where they should be leaning on evidence. Framing. Robert Entman's canonical definition still governs the field: to frame is "to select some aspects of a perceived reality and make them more salient" so as to promote a particular interpretation 4 . Framing is often invisible because it operates through omission — what the piece declines to mention. Detecting "strategic silence" is therefore not a nice-to-have; it's central. The hardest manipulation to catch is the argument that was never made. Propaganda techniques. The most directly relevant computational work is Da San Martino and colleagues' fine-grained propaganda detection, which defined a taxonomy of specific techniques — loaded language, name-calling, appeal to fear, flag-waving, whataboutism, and more — annotated at the span level in news articles, and later formalized as SemEval-2020 Task 11 5 . This is the key methodological move: propaganda is not a property of a document, it's a property of spans of text, and it can be enumerated. Fallacies. Douglas Walton's decades of work on argumentation reframed informal fallacies not as static errors but as argumentation schemes misused in context 6 . That matters for tooling, because it means fallacy detection has to be evidence-anchored: you can't flag "ad hominem" in the abstract, you flag it at the quote that commits it. Emotion. Robert Plutchik's wheel of eight primary emotions — joy, trust, fear, surprise, sadness, disgust, anger, anticipation — gives a compact, defensible vocabulary for measuring affective loading paragraph by paragraph 7 . It's more useful than a naïve positive/negative sentiment score because manipulation lives in the specific emotion being provoked. Why this works on people. The reason any of this matters is dual-process cognition. Kahneman's System 1 / System 2 distinction explains why emotionally loaded, fluent, repetition-heavy content bypasses deliberation 8 . And crucially, Pennycook and Rand's research suggests people fall for misinformation largely because they fail to stop and think, not because they're hopelessly partisan — falling for fake news correlates with lower analytic thinking more than with ideology 9 . That's an optimistic finding for tool-builders: if the failure mode is inattention, then a tool that makes manipulation visible — that forces a System 2 pause — can actually help. Put these together and you get a measurement model: decompose a text into spans, score each span on appeals ethos/pathos/logos , emotion Plutchik , and technique a propaganda/fallacy taxonomy , anchor every judgment to an exact quote, and roll it up into document-level indices. You can even layer in a cognitive-complexity axis — Norman Webb's Depth of Knowledge framework, borrowed from education, is a reasonable proxy for whether a piece reasons or merely asserts 10 . The engineering problem nobody warns you about Here's where it stops being a literature review and starts being software. If you naively wire an LLM to this taxonomy, you will ship something that gives a different answer every time you run it. For a tool whose entire value proposition is credibility, that's fatal. Large language models sample tokens stochastically. Even at temperature zero, floating-point non-associativity across hardware and batching means outputs are not bit-for-bit stable, and any temperature above zero makes the variance visible to users. A "propaganda index" that reads 71 on Monday and 78 on Tuesday for the same article is not a measurement; it's a mood. The generative-AI reflex — "just prompt it and trust the output" — collides head-on with the requirements of an analytical instrument. There's a second, deeper hazard: hallucination. The survey literature is blunt that generative models produce fluent, confident, unsupported claims as a matter of course 11 . An analyzer that invents a fallacy that isn't in the text is worse than useless — it's itself a source of misinformation. Four engineering commitments address this. They're the difference between a demo and an instrument. Span-anchoring kills hallucination at the root. Every score must trace back to exact words — ideally to character offsets into the normalized source text, validated after the fact so that text.slice start, end actually equals the quoted span. If the model claims a fallacy but can't point to the substring that commits it, the claim is dropped. This single constraint converts "the model thinks this is manipulative" into "here are the seventeen characters that are, and here's why." It also makes the output auditable, which is the whole game. Deterministic aggregation separates the fuzzy step from the exact step. Use the LLM for what it's good at — identifying and classifying spans — and then compute all document-level numbers with plain, deterministic code: severity tables, weighted sums, fixed formulas. The manipulation index isn't asked of the model; it's derived from the spans by an algorithm that returns the same number every time. The stochastic component is quarantined to perception; the arithmetic is reproducible by construction. Calibration corrects known model failure modes. No frontier model is neutral about political content; each has systematic tics. The pragmatic answer is a post-processing layer of transparent, rule-based overrides for well-characterized errors, applied deterministically and logged. This is unglamorous and occasionally derided, but it's honest engineering: you are encoding, in reviewable code, the specific ways your model is wrong, rather than pretending it isn't. Grounding checks factual claims against the outside world. Emotional and structural analysis is intrinsic to the text, but factual claims can be cross-referenced. Extracting the central claims and verifying them against fact-check corpora and knowledge bases — while being explicit about what could and couldn't be verified — closes the loop between "how is this argued" and "is it true." Finally, stamp every result with provenance: the prompt hash, the model version, the severity-table hash, the pipeline version. Reproducibility in science means someone else can rerun your method and get your result; the same standard should apply to an automated analysis. If you can't tell a skeptical reader exactly how a score was produced, they're right not to trust it. How to build one A concrete, framework-agnostic recipe. You can implement this with any capable model and a few hundred lines of glue. Step 1 — Normalize and chunk. Strip boilerplate, normalize whitespace, and split the article into paragraphs and overlapping windows. Keep a stable mapping from every chunk back to character offsets in the normalized text; you'll need it for span validation later. Cap the analyzed window to a fixed size and tell the user when you truncate — a partial analysis silently presented as complete is its own small deception. Step 2 — Define a strict output schema. Before you touch a model, write the schema: spans with start, end, quote, technique, appeal, confidence , per-paragraph emotion vectors, extracted claims, and the fixed set of document-level fields. Validate every model response against it and reject anything that doesn't conform. A schema is your contract; it's also your first line of defense against hallucination and drift. Step 3 — Prompt for classification, not judgment. Ask the model to locate and label spans against your taxonomy, quoting exact text, and to abstain when uncertain. Do not ask it for the final indices. Pin sampling as tightly as your provider allows — low temperature, low top-p, a fixed seed — knowing this reduces but does not eliminate variance. Step 4 — Validate spans. For each returned span, confirm the quote actually appears at the claimed offsets. Discard the rest. This is the step most teams skip and the one that most separates a toy from an instrument. Step 5 — Aggregate deterministically. Feed validated spans into fixed formulas: map techniques to severity weights, combine with confidence and coverage, and compute your manipulation index, evidence-validity score, and emotion arc in ordinary code. Same input, same output, forever. Step 6 — Ground the claims. Extract the load-bearing factual claims and check them against external sources, recording verified / disputed / unverifiable status and never upgrading "unverifiable" to "false." Step 7 — Attach provenance and surface uncertainty. Return the model version, prompt hash, and any limitations truncation, low grounding coverage, failed chunks alongside the scores. Show confidence, not just verdicts. Step 8 — Evaluate against a labeled corpus. Build or borrow a benchmark with human annotations across genres — hard news, opinion, propaganda, satire, PR — and track technique-level F1, not just an aggregate. SemEval-2020 Task 11 is a natural starting point for the propaganda side 5 . Regression-test every prompt or model change against it; an "improvement" that drops your satire recall is a regression. This is the architecture behind Rhetoric Audit's analysis pipeline, and the pattern generalizes well beyond media: contract review, research-integrity screening, and moderation tooling all benefit from the same "fuzzy perception, deterministic aggregation, mandatory provenance" spine. What this approach can't do — and why saying so matters Forensic analysis measures rhetoric, not righteousness. A perfectly reasoned argument for a false conclusion will score well on structure; a clumsy argument for a true one will score badly. That's a feature — the tool deliberately refuses to adjudicate truth from style — but it means the output is an input to judgment, not a substitute for it. Three honest caveats. First, taxonomies are lenses, not laws; reasonable annotators disagree about whether a given sentence is "loaded language" or just vivid writing, and inter-annotator agreement on propaganda techniques is imperfect even among experts 5 . Second, satire and strong opinion writing use manipulation techniques intentionally and legitimately, so any system needs genre-awareness or it will libel The Onion. Third, grounding is only as good as the corpora behind it; "unverifiable" is common and must never masquerade as "refuted." State these plainly in your product. A forensic tool that overclaims becomes exactly the thing it was built to detect. FAQ Is this just sentiment analysis with extra steps? No. Sentiment analysis returns a polarity; forensic analysis returns a structured decomposition — which techniques, at which spans, with which appeals and emotions, plus deterministic document indices and provenance. Polarity tells you a text is negative; forensic analysis tells you it's negative because of an appeal-to-fear in paragraph three that omits the base rate. Doesn't using an LLM make it inherently unreliable? Only if you let the LLM produce the final numbers. The design deliberately confines the model to perception finding and labeling quoted spans and computes all scores with deterministic code. Combined with span validation and provenance stamping, the same article yields the same analysis — the reproducibility comes from the architecture, not from the model. How do you avoid political bias in the tool itself? By measuring structure rather than stance, by anchoring every judgment to a quote you can inspect, and by applying model-calibration corrections transparently and in reviewable code. You should be able to disagree with a score by reading the exact span it cites — that auditability is the accountability mechanism. Can it detect AI-generated propaganda? It detects rhetorical technique regardless of author. Whether a fear appeal was written by a human columnist or a language model is irrelevant to whether it's a fear appeal. That authorship-agnosticism is increasingly valuable as synthetic content scales. Is there real research behind this, or is it vibes? The measurement model draws on peer-reviewed work in rhetoric, framing theory, computational propaganda detection, argumentation theory, and cognitive science — several of which are cited below. The engineering draws on the hallucination and evaluation literature. It is emphatically not vibes, though like any measurement instrument it encodes choices you're free to contest. Where can I try it? Rhetoric Audit runs this pipeline in the browser: rhetoricaudit.com. More usefully for this audience, the pattern — fuzzy perception, deterministic aggregation, mandatory provenance — is yours to reimplement for any domain where you need trustworthy analysis on top of an untrustworthy generator. Closing The uncomfortable truth of the current moment is that persuasion has been automated and comprehension has not. Closing that gap is not primarily a modeling problem; it's an engineering-discipline problem. The models are good enough. What's scarce is the willingness to constrain them — to demand quotes, to compute scores deterministically, to publish provenance, and to say clearly what the tool cannot do. Build that discipline in, and you get an instrument people can actually trust. Skip it, and you've built one more confident voice in a room that already has too many. References Vosoughi, S., Roy, D., & Aral, S. 2018 . The spread of true and false news online. Science, 359 6380 , 1146–1151. https://doi.org/10.1126/science.aap9559 https://doi.org/10.1126/science.aap9559 Wardle, C., & Derakhshan, H. 2017 . Information Disorder: Toward an interdisciplinary framework for research and policy making. Council of Europe. Aristotle. c. 4th century BCE . Rhetoric W. Rhys Roberts, Trans. . Entman, R. M. 1993 . Framing: Toward clarification of a fractured paradigm. Journal of Communication, 43 4 , 51–58. https://doi.org/10.1111/j.1460-2466.1993.tb01304.x https://doi.org/10.1111/j.1460-2466.1993.tb01304.x Da San Martino, G., Yu, S., Barrón-Cedeño, A., Petrov, R., & Nakov, P. 2019 . Fine-Grained Analysis of Propaganda in News Articles. EMNLP-IJCNLP 2019. arXiv:1910.02517. See also SemEval-2020 Task 11. Walton, D. 2008 . Informal Logic: A Pragmatic Approach 2nd ed. . Cambridge University Press. Plutchik, R. 1980 . A general psychoevolutionary theory of emotion. In Emotion: Theory, Research, and Experience Vol. 1 . Academic Press. Kahneman, D. 2011 . Thinking, Fast and Slow. Farrar, Straus and Giroux. Pennycook, G., & Rand, D. G. 2021 . The Psychology of Fake News. Trends in Cognitive Sciences, 25 5 , 388–402. https://doi.org/10.1016/j.tics.2021.02.007 https://doi.org/10.1016/j.tics.2021.02.007 Webb, N. L. 2002 . Depth-of-Knowledge Levels for Four Content Areas. Wisconsin Center for Education Research. Ji, Z., et al. 2023 . Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55 12 . arXiv:2202.03629