AX-RAY: VIDRAFT's Open AI Safety Diagnostic Leaderboard & Dataset Now on Hugging Face

VIDRAFT has publicly released AX-RAY, a 117-item AI safety diagnostic framework, along with an open leaderboard and evaluation dataset on Hugging Face. The system detects 'Causal Leakage,' a subtle failure mode where LLMs are influenced by hidden or unintended causal cues rather than legitimate reasoning paths. Two publicly available general-purpose AI models showed anomalous signals on the Causal Leakage dimension, including one from NVIDIA.

TL;DR:VIDRAFT has publicly released AX-RAY, a 117-item AI safety diagnostic framework, along with an open leaderboard and evaluation dataset on Hugging Face. The system detects "Causal Leakage" — a subtle failure mode where LLMs are influenced by hidden or unintended causal cues rather than legitimate reasoning paths. Two publicly available general-purpose AI models showed anomalous signals on the Causal Leakage dimension, including one from NVIDIA. AX-RAY is VIDRAFT's AI safety evaluation framework designed to diagnose latent risk behaviors and safety vulnerabilities in large language models LLMs and AI agents. It is structured around 117 diagnostic evaluation items and goes beyond capability benchmarking to assess a model's potential for norm-violating or unsafe behavior. Key distinguishing features of AX-RAY: The intended scope is broad: finance, healthcare, robotics, and public services — domains where LLM misbehavior against legal or normative standards carries real-world consequences. At a conceptual level, AX-RAY probes models with structured evaluation items designed to surface latent behavioral patterns that standard capability benchmarks miss. Causal Leakage , the core diagnostic signal in this release, refers to a model being steered by spurious or hidden causal cues embedded in its inputs or context — rather than reasoning transparently from stated evidence. Think of it as the AI equivalent of a human expert whose judgment is unconsciously biased by irrelevant contextual signals they can't articulate. The framework's diagnostic approach works roughly as follows: Importantly, VIDRAFT is explicit that detecting an anomalous signal in the Causal Leakage dimension does not mean the model has caused harm or compromised any system. It indicates a potential vulnerability that warrants further investigation — closer to a security scanner flagging a suspicious pattern than a confirmed exploit. The source article does not publish quantitative scores or percentile rankings, but provides these qualitative findings: VIDRAFT plans to expand both the number of models evaluated and the scope of the diagnostic dataset over time as the leaderboard matures. The AX-RAY leaderboard and evaluation dataset are publicly available on Hugging Face . According to VIDRAFT's announcement, developers can access them directly via the Hugging Face platform. To browse or download the dataset using the Hugging Face CLI: pip install huggingface hub huggingface-cli login Then search for VIDRAFT's AX-RAY dataset on huggingface.co/vidraft Note:Specific dataset repository slugs and leaderboard URLs were not published in the source article. Visit huggingface.co and search forVIDRAFTorAX-RAYto locate the official dataset and leaderboard space. No API endpoint, model weights, or SDK beyond the dataset have been announced as part of this release. Q: Is Causal Leakage the same as prompt injection or jailbreaking? A: Not exactly. Prompt injection involves deliberate adversarial manipulation of a model's input. Causal Leakage is a subtler, often unintentional phenomenon — the model's internal reasoning is swayed by hidden or spurious causal signals in its context, even without an adversary deliberately engineering them. It's more of a structural reasoning flaw than an attack vector, though it could potentially be exploited. Q: Does the leaderboard only evaluate safety, or does it also cover capability? A: Based on the source, AX-RAY is exclusively a safety diagnostic framework . It is not designed to benchmark performance on reasoning, coding, or knowledge tasks. Its 117 evaluation items specifically target latent risk behaviors, potential norm violations, and regulatory non-compliance — making it complementary to, not a replacement for, capability benchmarks like MMLU or HumanEval. Q: Why does AX-RAY include jurisdiction-specific and cultural normative frameworks? A: AI safety is not culturally neutral. A model deployed in a financial advisory context in one country may face entirely different legal obligations than the same model deployed in a healthcare context elsewhere. By mapping diagnostic results to local laws and — in some cases — religious or social normative systems, AX-RAY aims to make safety evaluations actionable for teams building region-specific or regulated-industry applications. Q: Does being flagged on AX-RAY mean a model is unsafe to use? A: No. VIDRAFT is clear that a flagged Causal Leakage signal represents a potential vulnerability identified in evaluation scenarios — not evidence that the model has produced harmful outputs or security breaches in production. Teams should treat it as a signal for deeper investigation in their specific deployment context. Originally reported by IT조선 2026-08-18 — source article.