A benchmark named EvaOmni targets audio-visual reasoning in omni LLMs by requiring models to locate key audio and visual evidence and integrate information across modalities, rather than scoring only answer correctness as existing multiple-choice audio-visual benchmarks do. The benchmark's stated aim is to address the limits of answer-only scoring, which the authors say makes it difficult to evaluate how models reach their conclusions.
Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.