{"slug": "i-tried-to-build-a-specialized-8b-security-model-the-most-interesting-result-was", "title": "I tried to build a specialized 8B security model. The most interesting result was that I had to reject it", "summary": "An open-source research prototype called XSS Specialist rejected every candidate of its fine-tuned 8B security model after each one repeatedly failed an adversarial near-miss test in which small changes to security-relevant identifiers produced incorrect safety judgments, the project's author reported. Retrieval, LoRA and gated continual learning did not eliminate the failure, so the author moved confirmation authority outside the model to an independent execution layer, where a potential XSS finding becomes CONFIRMED only when a real headless browser executes a benign sentinel; the resulting system scored on a frozen 102-case synthetic live benchmark, which the author says is not evidence of production readiness. The author open-sourced the code, architecture, benchmarks, evaluation methodology and negative results, arguing reliable AI systems should be designed to remain reliable when the model is wrong.", "body_md": "I’ve been experimenting with a question that I think goes beyond cybersecurity:\n\n**What if an AI model is allowed to reason, but is never allowed to be the source of truth?**\n\nI built an open-source research prototype called **XSS Specialist**, using XSS vulnerability analysis as the test environment.\n\nThe original goal was relatively straightforward: take a small open 8B model and see how far specialization could push it using retrieval, LoRA, gated continual learning, and adversarial evaluation.\n\nThe unexpected part was what happened next.\n\nThe specialist became significantly better at several parts of the task, but repeatedly failed an adversarial near-miss test. Small changes to security-relevant identifiers could still cause incorrect safety judgments.\n\nFine-tuning didn’t eliminate it.\n\nRetrieval didn’t eliminate it.\n\nMultiple specialist iterations didn’t eliminate it.\n\nSo instead of lowering the benchmark, **the promotion gate rejected every model candidate.**\n\nThat changed the direction of the experiment.\n\nI redesigned the system around a different assumption:\n\n**The model can reason. The model can propose. But the model cannot establish truth.**\n\nFor live assessment, confirmation authority was moved outside the model to an independent execution layer. A potential XSS finding only becomes `CONFIRMED` when a real headless browser actually executes a benign sentinel.\n\nOn the frozen 102-case live benchmark, the resulting system achieved:\n\nThe benchmark is synthetic/local and the project is still a research prototype, so I don’t consider these numbers evidence of production readiness.\n\nThe result I find more interesting is architectural:\n\n**Maybe reliable AI systems shouldn’t depend on making the model trustworthy enough.**\n\n**Maybe they should be designed to remain reliable when the model is wrong.**\n\nIn this experiment:\n\n**The model failed. The architecture didn’t.**\n\nI’ve open-sourced the code, architecture, benchmarks, evaluation methodology, failures, and negative results:\n\nI’d particularly appreciate feedback from people working on **small-model specialization, agents, model evaluation, verification, or AI security**.\n\nI’m curious: **Where else could this separation between model reasoning and external verification be useful?**", "url": "https://wpnews.pro/news/i-tried-to-build-a-specialized-8b-security-model-the-most-interesting-result-was", "canonical_source": "https://discuss.huggingface.co/t/i-tried-to-build-a-specialized-8b-security-model-the-most-interesting-result-was-that-i-had-to-reject-it/180765#post_2", "published_at": "2026-10-21 02:33:24+00:00", "updated_at": "2026-10-21 02:50:22.201300+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "large-language-models", "ai-agents", "ai-tools"], "entities": ["XSS Specialist", "LoRA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-tried-to-build-a-specialized-8b-security-model-the-most-interesting-result-was", "markdown": "https://wpnews.pro/news/i-tried-to-build-a-specialized-8b-security-model-the-most-interesting-result-was.md", "text": "https://wpnews.pro/news/i-tried-to-build-a-specialized-8b-security-model-the-most-interesting-result-was.txt", "jsonld": "https://wpnews.pro/news/i-tried-to-build-a-specialized-8b-security-model-the-most-interesting-result-was.jsonld"}}