Show HN: MediationBench – testing AI mediators in 62 simulated conflicts

Human Assisted Intelligence, PBC released MediationBench, a benchmark that compares AI mediators across 62 authored simulated two-party conflicts, with language models playing both parties and the tested system deciding when to speak, wait, or end within a fixed 3-to-25-round window. Each provisional row shows a 0-to-100 score from judge model Sol, averaged across all 62 conflicts, plus how often the final judge found the conversation ended appropriately; the organization cautions that results come from one complete run scored by one or two judge models and that numbers may change while the working paper remains a draft. The benchmark states it does not measure human safety, fairness, legality, or whether an agreement will last.

Published by Human Assisted Intelligence, PBC MediationBench MediationBench compares AI mediators on the same 62 simulated two-party conflicts. Within each test, every system works with the same model-played parties under the same rules. Early results are labeled provisional: one complete run, scored by one or two judge models. How it works Each run covers 62 authored disputes. Language models play both parties, and the system being tested acts as mediator. The mediator decides when to speak, wait, or end within a fixed 3-to-25-round window. Sol scores every finished conversation after the run. A row scored by Sol alone is Sol's average across all 62 conflicts. Other judges can be added later and are shown separately under the row; each judge that scores all 62 conflicts is averaged into the row score, so early results stay inexpensive without pretending they are complete. How to read the results Each provisional row shows a score from 0 to 100 and how often the final judge found that the conversation ended appropriately. Rows are ordered by score. Do not assume a small score gap is meaningful. All parties are AI. The benchmark does not measure human safety, fairness, legality, or whether an agreement will last. Research and data MediationBench: Auditing AI Mediation in Synthetic Conflict The research page contains the exact method, earlier studies, limitations, corrections, and downloadable data. The working paper gives the complete design and evidence. Numbers may change while the paper remains a working draft.