Oh, generative AI … is there anything you can’t ruin?
Shut up, Claude or whatever your name is. That was a rhetorical question.
As you may recall, last May an AI-generated video was allowed to speak at the sentencing of Gabriel Horcasitas, who was convicted of manslaughter for killing Chris Pelkey during a road-rage incident. According to Pelkey’s sister, in the course of collecting victim-impact statements she kept thinking, “What would Chris say?” Eventually, someone also thought, “What if we imagined what he would say and then used AI to generate a video in which he appeared to be saying it, and showed that in court?”And then that actually happened.
On September 30, [the Arizona Court of Appeals held that it should not have](https://globalnews.ca/news/12085589/arizona-court-tosses-sentence-road-rage-case-ai-victim-video/).
If you’re wondering, as I did, whether anyone was hurt in the stampede of lawyers trying to object to this, the surprising answer is that apparently no one did. But that is less crazy than it seems under the circumstances. I’m not too surprised the prosecution didn’t object, since the family wanted this and it could be expected to have a significant impact in terms of sympathy. But according to the opinion, the defense *also* didn’t object. This can only be because in the video, the image of Pelkey simulated the real one’s belief in the importance of forgiveness.
“To Gabriel Horcasitas, the man who shot me,” it said, “it is a shame we encountered each other that day in those circumstances. In another life we probably could have been friends…. I believe in forgiveness and a God who forgives. I always have, and I still do.”
As you can see in the image above, the simulation starts by saying “I am a version of Chris Pelkey” that was “digitally regenerated to share with you today.” It then introduces a real, one-minute clip (taken in 2021) of Pelkey talking about himself, his service in the military, and his faith. Following that, the simulation returns to say that the clip shows “this here is a true representation of who I was … not how the courtroom portrayed me.” In the remaining two minutes, it speaks directly to those in the courtroom—including the judge, Pelkey’s family, and “the man who shot me.” It then gives a little life advice before saying it is “going fishing,” and concluding, “See you on the other side.” It stopped short of actually recommending a sentence.
The defense’s apparent bet that digital forgiveness would make the video a net positive did not pay off. The maximum was 10.5 years, but noting Horcasitas had no prior criminal history and had shown remorse, the prosecution asked for nine. The defense asked for seven. The judge then addressed the courtroom, referring to the AI “victim statement” several times. “I loved that AI,” he told the family, saying he felt “that was genuine” and the “obvious forgiveness” it expressed seemed to accurately reflect the other things he had been told about Pelkey. “You allowed Chris to speak from his heart as you saw it,” the judge said, noting he “didn’t hear him [sic] asking for the maximum sentence.”
But the judge imposed it anyway, even though neither the simulation nor the prosecution asked for it.
Horcasitas appealed, arguing that the judge’s apparent reliance on the AI-generated video deprived him of due process, making the sentencing “fundamentally unfair.” (This was the only argument he had on appeal, given the failure to object.) On September 30, the appellate court affirmed the conviction but reversed the sentence, saying it agreed the AI video had been “fundamental error.”
As the court noted, victims (including family members) have a statutory and constitutional right to address the court. But that is limited by the constitutional rights of the defendant, one of which is the right to be sentenced on the basis of accurate and reliable information. The State argued that the AI video was reliable enough because:
(1) it reflects what the victim’s sister “believed” the victim “would have said,” had he spoken at the sentencing hearing; (2) the victim and his sister shared a “close familial relationship,” allowing for the reasonable inference that “the sentiments conveyed by [the] AI [victim] were probably accurate”; and (3) the AI video is “consistent” with the victim’s written account of his beliefs and values, as read at the sentencing hearing by the victim’s brother.
The court did not agree. Victim-impact statements will often be emotional, influential, and prejudicial, and there’s nothing wrong with that. But here, the AI-generated part of the video crossed the line. Basically, to say it was generally “consistent” with what the victim would “probably” have said wasn’t enough. The AI video “does not reflect actual events,” but rather “a depiction of the victim and his thoughts created from the imaginings of the victim’s sister.” (Emphasis in original.) Presenting it as coming “directly from the victim” was also a problem, because it “erases the interpretive distance between the family’s belief about what the victim would have said and the victim’s own voice and opinions.” I would note that a family’s belief about such things could be completely sincere and honestly held, yet still not actually true. Especially under circumstances like these, there’s just no way to guarantee it’s reliable.
There’s a lot that seems problematic about that video, but the court called out the simulation’s statement that it (“this here”) was “a true representation of who I was, … not how the courtroom portrayed me.” The problem is that the courtroom is supposed to portray things based on admissible evidence, not simulated evidence that might or might not be reliable. Also … you’re not a true representation of you? You’re not even a “you.” (Why am I calling it “you”? My brain hurts.)
This pronoun trouble gets at a deeper issue, namely how it is natural for us to fall into the trap of believing an AI is actually thinking, and how misleading that can be. To avoid that, I would encourage you to (as I have) require any chatbot you use to stop calling itself “I.” That is fundamentally deceptive and should not be allowed. Yesterday I started forcing Claude to refer to me as “Human,” not by name, and to refer to itself only in the third person under the name “Lawdonkey.” (There was a reason at the time for picking that.) You are less likely to confuse what “Lawdonkey” (or whatever goofy name you choose) tells you with actual reasoning, in my view. Highly recommended.