The McGurk effect: what you see changes what you hear

A blogger working on an AI-generated parody video with Opus 5.5 found that a scene of Jensen Huang saying "PROMPT!" sounded like "PRONT!" despite the video's AAC audio track being byte-identical to the approved raw audio clip, and Claude identified the cause as the McGurk effect, an audiovisual illusion in which lip movements change what a viewer hears. Re-running the lip sync without altering the audio fixed the perceived mispronunciation, and the author advises anyone doing AI video or lip-syncing work to keep the effect in mind.

On this page Today, as I was working on a random parody video with Opus 5.5, I ran into something really strange. There was a generated scene with Jensen Huang as a drill instructor saying the word “PROMPT ”, but no matter what I did, the video clip for that scene kept sounding… off. I could swear he was saying “PRONT ” even though the approved audio clip for that scene was clearly saying “PROMPT ”. I asked Claude to fix this because obviously the audio was broken, but after multiple back and forths, Claude eventually pushed back and told me I was wrong: the raw audio clip I had approved and the video’s audio track were identical… to which I promptly said fuck you Claude you’re just a dumb AI model, wtf are you smoking over there at Anthropic? That’s when Claude figured out what was really going on was something called the McGurck effect https://en.wikipedia.org/wiki/McGurk effect : an audiovisual illusion where what you see changes what you hear… Fixed Version 3fcedb27f124803c953be4e3648a7d9e So I had Claude redo the lip sync for this scene without changing the audio : And it worked Needless to say, I was dumbfounded. The audio track for both of these video clips is exactly the same, but they sound completely different. 🤯 Not only was Claude right the video’s AAC audio track was byte identical to the raw audio track , but I also learned about a ridiculously cool audiovisual illusion. I still doubted Claude so I gasp opened up Google search and found this clip and lo & behold, this is a real effect, I just hadn’t encountered it before. TIL FTW So if you’re doing any AI video or lip syncing work, def keep this in mind. Cause it turns out our multimodal meat bodies are super easy to trick