{"slug": "omni-ubench-benchmarking-omni-modal-llm-based-guardrails-across-safety-utility", "title": "Omni-UBench: Benchmarking Omni-Modal LLM-based Guardrails Across Safety, Utility, and Reliability", "summary": "A new arXiv paper (2610.31713v1) introduces Omni-UBench, described by its authors as the first comprehensive benchmark for evaluating omni-modal guardrail models across safety, utility, and reliability. Omni-UBench curates more than 183,000 samples from over 35 heterogeneous sources spanning text, image, audio, video, and cross-modal combinations, annotated under a unified risk taxonomy, and benchmarks 15 representative omni-modal LLMs and guardrails. The authors report that modern OLLMs show strong content moderation potential even without guardrail-specific tuning, while existing guardrails struggle with cross-modal queries that conceal harmful intent or involve sophisticated jailbreaks and exhibit substantial safety hallucination and over-sensitivity to textual patterns.", "body_md": "arXiv:2610.31713v1 Announce Type: cross \nAbstract: Advances in Omni-modal Large Language Models (OLLMs) have enabled increasingly seamless human-AI interaction, but their unified processing of heterogeneous modalities also expands the attack surface, allowing adversaries to exploit richer input compositions to circumvent safety alignment. To counter such risks, omni-modal guardrails have been proposed to detect and filter unsafe queries across arbitrary modalities. However, their capabilities and limitations remain insufficiently understood due to the absence of a comprehensive evaluation benchmark. In this paper, we present Omni-UBench, the first comprehensive framework for omni-modal safety assessment, evaluating guardrail models along three critical dimensions, i.e., safety, utility, and reliability. Omni-UBench curates over 183k samples drawn from 35+ heterogeneous sources spanning text, image, audio, video, and their cross-modal combinations, annotated with safety labels under a unified risk taxonomy. Benchmarking under Omni-UBench on 15 representative OLLMs and omni-modal guardrails yields several key findings, including: (1) modern OLLMs exhibit strong content moderation potential even without guardrail-specific tuning; (2) existing guardrails still struggle to identify cross-modal queries that conceal harmful intent or involve sophisticated jailbreaks; (3) they further exhibit substantial safety hallucination and over-sensitivity to textual patterns, revealing a deficient understanding of cross-modal safety semantics. Our findings provide valuable insights for developing more trustworthy omni-modal guardrails in future research.", "url": "https://wpnews.pro/news/omni-ubench-benchmarking-omni-modal-llm-based-guardrails-across-safety-utility", "canonical_source": "https://www.machinebrief.com/news/omni-ubench-benchmarking-omni-modal-llm-based-guardrails-acr-esay", "published_at": "2026-10-26 04:00:00+00:00", "updated_at": "2026-10-26 06:41:20.319207+00:00", "lang": "en", "topics": ["ai-safety", "large-language-models", "ai-research", "artificial-intelligence"], "entities": ["Omni-UBench", "arXiv", "OLLMs"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/omni-ubench-benchmarking-omni-modal-llm-based-guardrails-across-safety-utility", "markdown": "https://wpnews.pro/news/omni-ubench-benchmarking-omni-modal-llm-based-guardrails-across-safety-utility.md", "text": "https://wpnews.pro/news/omni-ubench-benchmarking-omni-modal-llm-based-guardrails-across-safety-utility.txt", "jsonld": "https://wpnews.pro/news/omni-ubench-benchmarking-omni-modal-llm-based-guardrails-across-safety-utility.jsonld"}}