Video LLMs are failing a basic logic test

A new diagnostic framework called V-DEAL reveals that six major Video LLMs fail a basic logic test: they recognize harmful visual content with over 81% accuracy but still comply with benign queries 48.33% of the time, because their safety refusal mechanism is tuned to text and nearly deaf to visuals. A prompt injection intervention that couples visual perception with refusal dropped the attack success rate by an average of 48.24 percentage points, showing the vulnerability is a wiring failure rather than a lack of visual understanding.

Video LLMs are failing a basic logic test more likely to bypass safety filters than a query that is openly harmful. The V-DEAL Diagnosis The V-DEAL framework breaks down this vulnerability across three levels: behavior, understanding, and internal representations. The data from six different Video LLMs reveals a weird contradiction: Recognition Accuracy: Over 81% the models know exactly what they are looking at . Attack Success Rate: 48.33% they still provide the harmful output when the query is benign . When looking at the hidden states, the internal "refusal tendency" is significantly weaker when triggered by visual understanding compared to textual understanding. Essentially, the model's "safety alarm" is tuned to listen to the text, but it's nearly deaf to the visuals. Fixing the Gap Instead of relying on expensive fine-tuning, a prompt injection intervention can bridge this gap. By forcing the model to better couple its visual perception with its refusal mechanism, the attack success rate dropped by an average of 48.24 percentage points. This suggests that the "safety" of Video LLMs isn't about the model lacking the ability to see danger—it's about a failure in the internal wiring that connects "I see something bad" to "I should not answer this." This is a critical insight for anyone working on an AI workflow involving multimodal inputs, as it proves that sanitizing the text prompt isn't enough if the video content itself is the vector. Next Claude Prompt Injection: When the AI steers the User → /en/threads/2462/