{"slug": "ax-ray-vidraft-s-open-ai-safety-diagnostic-leaderboard-dataset-now-on-hugging", "title": "AX-RAY: VIDRAFT's Open AI Safety Diagnostic Leaderboard & Dataset Now on Hugging Face", "summary": "VIDRAFT has publicly released AX-RAY, a 117-item AI safety diagnostic framework, along with an open leaderboard and evaluation dataset on Hugging Face. The system detects 'Causal Leakage,' a subtle failure mode where LLMs are influenced by hidden or unintended causal cues rather than legitimate reasoning paths. Two publicly available general-purpose AI models showed anomalous signals on the Causal Leakage dimension, including one from NVIDIA.", "body_md": "TL;DR:VIDRAFT has publicly released AX-RAY, a 117-item AI safety diagnostic framework, along with an open leaderboard and evaluation dataset on Hugging Face. The system detects \"Causal Leakage\" — a subtle failure mode where LLMs are influenced by hidden or unintended causal cues rather than legitimate reasoning paths. Two publicly available general-purpose AI models showed anomalous signals on the Causal Leakage dimension, including one from NVIDIA.\n\nAX-RAY is VIDRAFT's AI safety evaluation framework designed to diagnose latent risk behaviors and safety vulnerabilities in large language models (LLMs) and AI agents. It is structured around **117 diagnostic evaluation items** and goes beyond capability benchmarking to assess a model's potential for norm-violating or unsafe behavior.\n\nKey distinguishing features of AX-RAY:\n\nThe intended scope is broad: finance, healthcare, robotics, and public services — domains where LLM misbehavior against legal or normative standards carries real-world consequences.\n\nAt a conceptual level, AX-RAY probes models with structured evaluation items designed to surface latent behavioral patterns that standard capability benchmarks miss.\n\n**Causal Leakage**, the core diagnostic signal in this release, refers to a model being steered by spurious or hidden causal cues embedded in its inputs or context — rather than reasoning transparently from stated evidence. Think of it as the AI equivalent of a human expert whose judgment is unconsciously biased by irrelevant contextual signals they can't articulate.\n\nThe framework's diagnostic approach works roughly as follows:\n\nImportantly, VIDRAFT is explicit that detecting an anomalous signal in the Causal Leakage dimension does **not** mean the model has caused harm or compromised any system. It indicates a *potential vulnerability* that warrants further investigation — closer to a security scanner flagging a suspicious pattern than a confirmed exploit.\n\nThe source article does not publish quantitative scores or percentile rankings, but provides these qualitative findings:\n\nVIDRAFT plans to expand both the number of models evaluated and the scope of the diagnostic dataset over time as the leaderboard matures.\n\nThe AX-RAY leaderboard and evaluation dataset are **publicly available on Hugging Face**. According to VIDRAFT's announcement, developers can access them directly via the Hugging Face platform.\n\nTo browse or download the dataset using the Hugging Face CLI:\n\n```\npip install huggingface_hub\nhuggingface-cli login\n# Then search for VIDRAFT's AX-RAY dataset on huggingface.co/vidraft\n```\n\nNote:Specific dataset repository slugs and leaderboard URLs were not published in the source article. Visit[huggingface.co]and search forVIDRAFTorAX-RAYto locate the official dataset and leaderboard space. No API endpoint, model weights, or SDK beyond the dataset have been announced as part of this release.\n\n**Q: Is Causal Leakage the same as prompt injection or jailbreaking?**\n\nA: Not exactly. Prompt injection involves deliberate adversarial manipulation of a model's input. Causal Leakage is a subtler, often unintentional phenomenon — the model's internal reasoning is swayed by hidden or spurious causal signals in its context, even without an adversary deliberately engineering them. It's more of a structural reasoning flaw than an attack vector, though it could potentially be exploited.\n\n**Q: Does the leaderboard only evaluate safety, or does it also cover capability?**\n\nA: Based on the source, AX-RAY is exclusively a **safety diagnostic framework**. It is not designed to benchmark performance on reasoning, coding, or knowledge tasks. Its 117 evaluation items specifically target latent risk behaviors, potential norm violations, and regulatory non-compliance — making it complementary to, not a replacement for, capability benchmarks like MMLU or HumanEval.\n\n**Q: Why does AX-RAY include jurisdiction-specific and cultural normative frameworks?**\n\nA: AI safety is not culturally neutral. A model deployed in a financial advisory context in one country may face entirely different legal obligations than the same model deployed in a healthcare context elsewhere. By mapping diagnostic results to local laws and — in some cases — religious or social normative systems, AX-RAY aims to make safety evaluations actionable for teams building region-specific or regulated-industry applications.\n\n**Q: Does being flagged on AX-RAY mean a model is unsafe to use?**\n\nA: No. VIDRAFT is clear that a flagged Causal Leakage signal represents a *potential vulnerability* identified in evaluation scenarios — not evidence that the model has produced harmful outputs or security breaches in production. Teams should treat it as a signal for deeper investigation in their specific deployment context.\n\n*Originally reported by IT조선 (2026-08-18) — source article.*", "url": "https://wpnews.pro/news/ax-ray-vidraft-s-open-ai-safety-diagnostic-leaderboard-dataset-now-on-hugging", "canonical_source": "https://dev.to/ai_openfree_b23025ef075cf/ax-ray-vidrafts-open-ai-safety-diagnostic-leaderboard-dataset-now-on-hugging-face-16ba", "published_at": "2026-08-19 07:01:19+00:00", "updated_at": "2026-08-19 07:12:13.694853+00:00", "lang": "en", "topics": ["ai-safety", "large-language-models", "ai-research", "ai-tools", "ai-ethics"], "entities": ["VIDRAFT", "AX-RAY", "Hugging Face", "NVIDIA"], "alternates": {"html": "https://wpnews.pro/news/ax-ray-vidraft-s-open-ai-safety-diagnostic-leaderboard-dataset-now-on-hugging", "markdown": "https://wpnews.pro/news/ax-ray-vidraft-s-open-ai-safety-diagnostic-leaderboard-dataset-now-on-hugging.md", "text": "https://wpnews.pro/news/ax-ray-vidraft-s-open-ai-safety-diagnostic-leaderboard-dataset-now-on-hugging.txt", "jsonld": "https://wpnews.pro/news/ax-ray-vidraft-s-open-ai-safety-diagnostic-leaderboard-dataset-now-on-hugging.jsonld"}}