Crown Prosecution Service caught using AI hallucination evidence

The Crown Prosecution Service apologized twice to the High Court after submitting documents in an extradition appeal that cited two non-existent legal cases generated by AI. The prosecutor used generative AI for legal research but blamed human error for failing to verify the citations. The judge included the incident in the public record as a warning about AI risks in legal work.

Crown Prosecution Service caught using AI hallucination evidence British courts have had to deal with yet another case of ‘AI hallucination’ evidence – this time from the agency responsible for conducting criminal prosecutions. The Crown Prosecution Service CPS issued two apologies to the High Court after it was found to have submitted documents in an extradition appeal citing two “non-existent” legal cases. The prosecutor used generative AI for legal research but stated the “operative cause” was human error, as the reviewing lawyer failed to verify the citations before submitting them. “The CPS emphasised that this was not a deliberate attempt to mislead, but rather an isolated incident arising from inadequate checking of written work,” a judge wrote. The CPS had to apologise to the court twice for an error regarding whether it knew it had submitted evidence containing an AI hallucination. Initially, the prosecutor told the court the error was spotted by the appellants’ counsel. But it had to correct this after admitting that its own junior counsel had also independently identified the fake cases and raised them internally just before the appellants did, leading it to issue a “further apology” to the courts. Mr Justice Sweeting accepted the apologies and the assurance that there was no intent to mislead. The fake cases didn’t affect the outcome because they were found before the hearing. However, the judge included the incident in the judgment’s public record as a warning about the dangers of using AI for legal research without proper human oversight. “It would be naive to assume that there will not be an increasing use of artificial intelligence in legal work in future; indeed, that may be both necessary and beneficial. The episode highlights the risks of its use without appropriate oversight, particularly for legal research,” he said. Justice system grapples with growing problem of AI Evidence containing AI hallucinations is becoming a growing problem for the judiciary system. In May, City law firm Pinsent Masons was criticised by a High Court https://www.cityam.com/top-city-law-firm-slammed-for-misleading-ai-letters-sent-to-court/ judge after a junior lawyer submitted letters to the court that contained false legal information. It was found that the lawyer had used AI to draft two “misleading” emails in an insolvency case in which the firm was instructed. This led Pinsent Masons to refer itself to the Solicitors Regulation Authority SRA , which will investigate further. Last year, the court had to review two cases https://www.cityam.com/law-isnt-like-other-professions-misusing-ai-may-put-lawyers-in-jail/ where lawyers relied on citations and quotations generated by AI tools, some of which were entirely fictitious. Across the pond, elite US firm Sullivan & Cromwell https://www.cityam.com/elite-law-firm-apologises-for-ai-hallucinations-in-bankruptcy-case/ had to apologise in April for multiple AI hallucinations in a bankruptcy case.