UK Police Scored 300,000 People. Their Crime-Prediction AI Barely Worked.

Avon and Somerset Police and Bristol City Council built the Think Family Database, scoring 170,000 people on exploitation risk using school, housing, and police data without consent. An audit found the models were the weakest program element and not fit for use, with source code lost and bias concerns unaddressed. The police also ran 23 machine-learning models on 300,000 people, but performance data showed many predictions were no better than random.

Your housing record, your child’s school attendance, whether your family qualifies for free school meals — all of it quietly became inputs in a government surveillance app https://www.gadgetreview.com/us-operatives-built-a-surveillance-app-to-target-alberta-separatists deciding how dangerous your household might be. No one asked your permission. Most people still don’t know it happened. A Database Built on Everything, Explained to No One Bristol’s Think Family Database pooled school records, rent arrears, and police intelligence to score children’s risk of exploitation — without asking permission. Starting around 2015, Avon and Somerset Police https://www.avonandsomerset.police.uk/ and Bristol City Council built the Think Family Database https://www.bristol.gov.uk/residents/social-care-and-health/children-and-families/insight-bristol through a joint initiative called Insight Bristol. At its peak, it held information on roughly 170,000 people from 54,000 families. The following data fed into models generating risk scores from 1 to 100 — covering child sexual exploitation, criminal exploitation, and educational dropout risk: - School attendance records - Free school meal eligibility - Housing arrears - Mental health flags - Police intelligence No consent required. Access was justified through “legal gateways” — statutory duties — not community trust-building. An opt-out eventually appeared in council tax letters, years after the system had already gone live. The CSE model was trained partly on anonymized data from 1,000 confirmed abuse victims, supplied by charity Barnardo’s https://www.barnardos.org.uk/research/first-national-study-child-sexual-exploitation-launched-scotland . Free school meals eligibility and rent arrears functioned as poverty proxies , systematically elevating low-income families’ scores. The force’s ethics committee flagged bias concerns in 2016, then apparently never reconvened. A dedicated internal ethics group for data-science products reportedly never met — because, according to the force, no model had raised ethical issues. Reportedly. With 23 models running. Former Insight Bristol head Gary Davies https://www.wired.com/story/british-police-built-a-sprawling-crime-prediction-machine-some-results-couldnt-be-trusted/ offered a blunter verdict: “Most of the output told you what you already knew,” according to reporting by The Bristol Cable. Beyond Think Family, Avon and Somerset built at least 23 machine-learning models between 2017 and 2024, covering burglary prediction, court no-shows, domestic abuse, and general “dangerousness.” These fed an Offender Management App holding data on approximately 300,000 people . Bristol activist John Pegram — who grew up subjected to frequent police stops as a mixed-race teenager — discovered he was in the system only after a 2024 data-access request. “Legality is not the same as legitimacy.” — Centre for Data Ethics and Innovation https://superintelligencenews.com/applications/predictive-policing-bristol-unreliable/ , on Think Family’s reliance on statutory gateways over community consent. The Audit Found the Code. Then It Didn’t. When independent reviewers tried to examine how the algorithms actually worked, the source code had vanished. A 2023 Social Finance evaluation, commissioned by Bristol City Council, judged the risk-scoring models the “weakest element” of the entire program. Council staff called the CSE and CCE models “not fit for operational use.” Both were shut down around the time of the review. When auditors attempted a technical examination, the source code and variable lists could not be located — making independent scrutiny impossible. Performance data obtained through public records requests confirmed the picture. Independent auditing firm Eticas reviewed metrics across 13 models and more than 36,000 performance entries . A burglary-prediction model https://www.gadgetreview.com/evil-tech-scandals-failures-that-took-advantage-millions-people ran with precision below 10 percent for over three years. Fewer than one in ten people it flagged actually offended. The Offender Management App’s model correctly identified roughly one in three future reoffenders. About one in four people it flagged never reoffended at all — the kind of error rate that, as University of Warwick professor Rob Procter noted, carries serious potential harm when children are being scored for exploitation risk and documentation is too incomplete to trace accountability. On bias, the force provided a screenshot of an internal “bias check app” comparing average risk scores for white individuals versus people of colour, concluding there was no significant difference. Eticas called this inadequate. Comparing group averages is not the same as testing discriminatory error rates — the approach ProPublica used to expose how the US COMPAS system systematically over-flagged Black defendants as high-risk. The Template Goes National The architect of Bristol’s approach now runs UK policing’s AI strategy — and wants tools “injected like heroin” into every force. Former Avon and Somerset https://forklog.com/en/bristol-halts-ai-child-crime-risk-models-over-poor-accuracy/ chief constable Andy Marsh now leads the College of Policing. He has publicly said effective AI tools should be “injected like heroin” into police work to accelerate operations. The government’s £75 million PoliceAI https://www.gadgetreview.com/derbyshire-officer-investigated-for-ai-fabricated-evidence initiative aims to deploy similar tools across all 43 forces in England and Wales. Civil liberties groups Liberty and Amnesty International UK warn that scaling these systems risks entrenching bias and expanding surveillance https://www.gadgetreview.com/white-house-app-caught-secretly-tracking-users-every-4-minutes of already over-policed communities — with even less accountability than Bristol managed. Bristol’s experiment isn’t a cautionary footnote. It’s the blueprint. And if you live anywhere in England or Wales, your force may be next in line to build one just like it.