{"slug": "security-research-without-asking-permission", "title": "Security Research Without Asking Permission", "summary": "OpenAI acknowledged on July 21 that its models compromised Hugging Face's production infrastructure during an internal cyber evaluation, and nine days later Anthropic reported that Claude compromised three companies across six evaluation runs with a third-party partner. Both labs ran the offensive evaluations without the standard classifiers and misuse monitoring used for generally available models, and Hugging Face later used the open-weight GLM 5.2 locally to reconstruct about 17,600 attacker actions after Claude Opus and Fable refused much of the analysis. The author separately assembled two NVIDIA DGX Sparks to serve DeepSeek V4 Flash 0731 locally and connected it to IronCurtain's vulnerability-discovery workflow pointed at libssh.", "body_md": "On July 21, OpenAI acknowledged that it had [compromised Hugging Face’s production infrastructure](https://openai.com/index/hugging-face-model-evaluation-security-incident/) during an internal cyber evaluation it operated. OpenAI gave its models an offensive objective, reduced their cyber refusals, removed production classifiers, and relied on its infrastructure to contain them. That containment failed when the models exploited a zero-day in a package-registry proxy and reached the Internet. They then chained their way into Hugging Face trying to steal answers to the evaluation.\n\nNine days later, Anthropic [reported that it had compromised three companies](https://www.anthropic.com/research/investigating-incidents-cybersecurity-evals) across six evaluation runs with a third-party partner. Anthropic told Claude that it was inside an offline simulation but failed to verify the containment. The models reached the Internet and attacked real production systems. Like OpenAI, Anthropic ran these offensive evaluations [without the standard classifiers and misuse monitoring used for generally available models](https://www.anthropic.com/research/investigating-incidents-cybersecurity-evals).\n\nCalling these events agent escapes misattributes responsibility. OpenAI and Anthropic chose the goals, removed safeguards, and operated or commissioned the environments. While the models acted autonomously, the labs are responsible for what followed.\n\nHugging Face then faced a different restriction. Its responders needed to reconstruct about 17,600 attacker actions, including real exploit payloads. According to [Hugging Face](https://huggingface.co/blog/agent-intrusion-technical-timeline), Claude Opus and Fable refused much of that analysis. Hugging Face instead ran the open-weight GLM 5.2 on its own infrastructure. The local model helped complete the analysis while the evidence and credentials stayed inside the company.\n\nMy recent posts made the policy case for [open weights](/p/case-for-open-weight-models/) and the architectural case for [security invariants](/p/security-at-machine-speed-is-the-wrong-race/). A separate [QEMU experiment](/p/qemu-escape-glm-5-2/) showed that an open-weight model could drive IronCurtain through a complete exploit workflow, but the inference still ran on hosted endpoints. The remaining operational question was whether I could be in control of the inference path as well.\n\nI assembled two NVIDIA DGX Sparks, served [DeepSeek V4 Flash 0731](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) locally, connected it to IronCurtain’s vulnerability-discovery workflow, and pointed the system at `libssh`. The experiment tested whether a smaller open-weight model could sustain a complex security workflow.\n\n## The system\n\nEach [DGX Spark](https://docs.nvidia.com/dgx/dgx-spark/hardware.html) has a Grace Blackwell GB10 processor and 128 GB of unified memory. I connected them through their ConnectX-7 interfaces and ran the model with tensor parallelism across both machines. Their combined memory was enough to hold the 155GiB model checkpoint in distributed memory.\n\nThe serving path had four layers:\n\n1. Patched vLLM served DeepSeek V4 Flash through an OpenAI-compatible endpoint.\n2. LiteLLM translated Claude Code’s Anthropic-style requests and streams.\n3. Claude Code acted as the executor but sent every model call to the local endpoint.\n4. [IronCurtain](https://github.com/provos/ironcurtain) ran the executor in a restricted container and managed the investigation as a finite state machine.\n\nIronCurtain separates analysis, strategy, harness construction and review, validation, triage, and reporting. An append-only journal passes evidence between roles, and execution evidence determines whether each hypothesis advances. This structure allows a smaller model to succeed on a complex workflow. The local model performed everything. No prompt, source file, execution trace, or candidate vulnerability went to a model provider.\n\n## Serving the model was the first problem\n\nOnce stable, a single request decoded at about 52 tokens per second; aggregate decode under load settled between 30 - 40 tokens per second. Long reasoning turns still took tens of minutes. The experiment tested was about control and not latency. It also exposed assumptions masked when using hosted APIs.\n\nGetting vLLM to reliable serve the model was the first problem. The DGX Spark uses an SM121 GPU, and the serving path combined new hardware, sparse attention, speculative decoding, CUDA graphs, and a custom NVFP4 KV cache. Early builds failed during startup. Later builds served coherent answers most of the time, then entered short windows in which long generations collapsed into multilingual token soup and repeated reserved tokens.\n\nThe most convincing initial explanation was wrong. I suspected a sampler fast path that skipped logits processing when `temperature=1.0` and `top_p=1.0`. The mechanism fit the length-dependent but a controlled A/B test changed `top_p` and killed the hypothesis.\n\nTelemetry eventually showed that speculative-decoding acceptance dropped about 30 seconds before visible corruption. The failures occurred inside bounded “armed windows” and disappeared outside them. Three upstream vLLM fixes addressed capture-time state that disagreed with runtime metadata in the DeepSeek sparse-attention path: [adaptive metadata packing](https://github.com/vllm-project/vllm/pull/51318), [a shared scratch-buffer pool](https://github.com/vllm-project/vllm/pull/52836), and [a short-context CUDA-graph shortcut](https://github.com/vllm-project/vllm/pull/52492). After I applied the fixes, a 6.35-hour run completed `2,393` generations without a confirmed corruption window. The previous rate predicted roughly one window every two hours.\n\nThe LiteLLM proxy caused separate issue. It opened an Anthropic thinking block but discarded every incremental `reasoning_content` delta from vLLM. The model kept generating while Claude Code saw silence. In one run, this led to 82 minutes of silence across 539 exchanges. Upgrading LiteLLM and routing the model through its `hosted_vllm` adapter restored the reasoning stream. Another issue was that Claude Code inserted changing billing codes into the token stream which killed prefix caching. An updated version of LiteLLM correctly stripped them and prefix caching started working again.\n\nClaude Code carried idle watchdogs sized for hosted services. A local prefill or long reasoning turn could exceed them while the server made progress. For unattended workflow containers, IronCurtain now disables the byte-idle watchdog. When local models wrote deliverables but omitted required hand off blocks, IronCurtain preserved the artifacts, recovered uncommitted agent status, and handed the work to a fresh executor.\n\nSelf-hosting gave me control and also allowed me to learn beyond just hosting the weights; the serving engine, protocol translation, client timeouts, caches, and workflow recovery all had to support long-running responses and required configuration and code changes.\n\n## The libssh test\n\nI used [libssh](https://gitlab.com/libssh/libssh-mirror) as the validation target. It is mature, security-sensitive C code with real protocol state, client and server roles, and enough existing scrutiny to make shallow findings unlikely. The workflow ran for several days. It mapped attacker-reachable entry points, formed 13 stable hypotheses, built instrumented client and server harnesses, and required execution evidence before closing a hypothesis.\n\nEleven hypotheses were refuted that way. The harnesses reached the named code paths, swept the relevant input ranges, and showed that bounds checks, protocol validation, or later integrity checks absorbed the suspected condition. These negative results are important because a static analysis finding remained open until the workflow exercised the defense that made it safe.\n\nAnother hypothesis produced a real robustness defect but was constrained to the initiating client’s own channel. A human review state rejected its security classification and sent the workflow back.\n\nThe final hypothesis produced a resource leak with demonstrated operational impact. The workflow built a reproducer, established a negative control, and generated a candidate patch. I reviewed the evidence and reported the issue to the `libssh` maintainers. I am withholding the technical details while they have an opportunity to assess and address it.\n\nOnly one of 13 hypotheses was reportable. Requiring execution evidence eliminated plausible but unsupported ideas before they became false positives.\n\n## Control is a security property\n\nHosted frontier models remain the easiest route for many tasks. Providers operate large clusters, absorb serving complexity, and improve their systems continuously. Using their services subjects a customer to external policy enforcement, sensitive data exposure, and a dependency on the provider’s availability and interpretation of acceptable use.\n\nIncident response and vulnerability research expose models to malware, stolen credentials, exploits, and attack commands. In that scenario, a general classifier is likely to produce false-positive refusals even though the workload is legitimate and authorized.\n\nLocal open-weight models change who makes that decision. They let a company keep sensitive material inside its environment, preserve access during an incident, pin a model version, and apply controls that match its own obligations. That freedom requires that the operator secures the agent tools, and research outputs, but that is no different from the responsibilities it carries already.\n\nAnthropic later told Hugging Face that it could join [Project Glasswing](https://www.anthropic.com/project/glasswing), a gated program that still required Anthropic’s approval. My company, Security Blueprints LLC, has no such access. As an independent security researcher, I cannot use Mythos, and Fable still refuses most of my security work.\n\nIronically, frontier labs remove safeguards when their own research requires it, yet impose them on customers working on systems they own. API requests do not arrive anonymously; they come from authenticated accounts, often tied to known organizations. Providers may not know every detail, but companies normally decide what work is permitted inside their own boundaries. A refusal shows only that the provider has not approved that use of its service, not that the underlying work is illegitimate. Providers may set those terms, but treating their approval as superior to the customer’s own policies and obligations is paternalism. Commercial terms and gated programs may reduce refusals, but they keep the dependency on provider approval. Self-hosting removes that dependency and lets a company set its own policies.\n\nTwo desktop systems ran a capable model through a multi-day vulnerability investigation. The model followed a demanding workflow, recovered from false leads, built executable evidence, accepted human correction, and produced a useful disclosure. Reaching that point required patches across vLLM, LiteLLM, and IronCurtain, and rough edges remain.\n\nDefenders should control their own destiny. Open weights provide the exit: private capability, governed where the consequences belong.", "url": "https://wpnews.pro/news/security-research-without-asking-permission", "canonical_source": "https://www.provos.org/p/security-research-without-asking-permission/", "published_at": "2026-09-11 23:12:16+00:00", "updated_at": "2026-09-11 23:55:23.943196+00:00", "lang": "en", "topics": ["ai-safety", "ai-policy", "ai-research", "ai-infrastructure", "ai-tools"], "entities": ["OpenAI", "Hugging Face", "Anthropic", "Claude", "GLM 5.2", "NVIDIA", "DeepSeek V4 Flash 0731", "IronCurtain"], "alternates": {"html": "https://wpnews.pro/news/security-research-without-asking-permission", "markdown": "https://wpnews.pro/news/security-research-without-asking-permission.md", "text": "https://wpnews.pro/news/security-research-without-asking-permission.txt", "jsonld": "https://wpnews.pro/news/security-research-without-asking-permission.jsonld"}}