IBM researchers are open sourcing a vLLM plugin that lets developers inspect and edit what an AI model is doing at inference time without a major drag on speed.
A lot happens under the hood as an LLM acts on an incoming prompt and formulates a response. Hardly any of this generative processing, though, shows up in the text that ultimately reaches users.
You can make visible an LLM’s hidden ‘thought’ process using various tools for interpreting and steering AI model behavior, but most are incompatible with inference engines optimized for speed. In production environments where a model’s safety, tone, and accuracy matter most, these safety tools typically introduce long delays. That leaves two alternatives: training lightweight adapters to handle specific issues or retraining the model and recomputing its weights.
IBM’s new plugin library moves away from tinkering with models, and instead, targets the inference engine serving them up. Called vLLM-Hook MIA, it’s designed to let developers observe and steer a model’s evolving states at runtime, to do things like decline a malicious prompt, flag ‘hallucinated’ facts, or edit a response for tone and accuracy, all while hardly slowing down performance.
MIA (model internals aperture) is designed for vLLM, the main open-source inference engine serving enterprise workloads today. In a new technical report, IBM researchers reported that vLLM-Hook MIA was up to 20 times faster than its predecessor, vLLM-Hook, and up to 14 times faster than a similar method that inserts code into the model to interact with it.
“We show that you can observe and intervene in an AI model’s generation process without a major lag in performance,” says Irene Ko, an IBM researcher who leads the vLLM-Hook project. “We hope this can make running last-minute safety checks in vLLM more practical.”
Speed versus transparency #
Today’s inference engines were engineered for speed, and one of their core performance features is something called graph capture and replay, typically implemented using NVIDIA’s CUDA Graphs. vLLM captures repetitive execution steps, like the decode phase forward pass, into a fixed execution graph. Then, instead of launching GPU kernels one by one, the runtime sends a single graph replay command, reducing CPU bottlenecks.
This efficiency feature is key to vLLM’s high throughput. It’s also what makes monitoring and steering an LLM at runtime so difficult. To inspect or change a model’s neural activations, many interpretability tools need to run Python code within each model layer. You can only do this in vLLM, however, by disabling graph capture and replay and shifting into its “eager” execution mode.
In search of a workaround, IBM researchers homed in on graph replay. In MIA, the latest version of vLLM-Hook, GPU operators capture and steer model internals before the graph is recorded. A “scheduler” and a “courier” then manage access to internal states and data flows without interrupting vLLM’s efficient processing.
A developer specifies which model internals they want to capture or steer, and the scheduler translates those selections into routing entries stored at fixed GPU memory addresses. During graph replay, the GPU uses those entries to act on the developer’s request. The courier then moves the captured data to CPU memory to deliver with the model’s output.
We hope this can make running last-minute safety checks in vLLM more practical.
Shifting interpretability tools from research to production #
Like many new technologies, vLLM-Hook started with a problem. In collaboration with their colleagues, Ko and Pin-Yu Chen, an IBM researcher focused on AI safety, had developed an attention tracking tool to identify prompts potentially intended to trick models into divulging information they shouldn’t. Chen’s tracker analyzes a model’s attention patterns at inference time and picks out deviations indicating it could be about to ignore its guardrails.
In research environments, the attention tracker worked fine. Production was another matter. “I said it was a great idea, but making it work in vLLM was going to be painful,” said David Cox, an IBM vice president who leads AI development. “They took the comment to heart and started building extension points to vLLM.”
Working with Chen, and Yunhua Fang, an extern studying at Rensselaer Polytechnic Institute, Chen saw that inserting Python ‘hooks’ into the decoder layers of a model’s forward pass could make visible vLLM’s generation process. With a few snippets of code, they could pick and capture a model’s evolving states, a bit like a tape recorder programmed to record at designated moments.
You could either ‘read’ those selected states or ‘write’ over them to improve a model’s response. The team released the first version of vLLM-Hook earlier this year. When they ran their attention tracker, they saw a 1.7-times speedup over the Hugging Face transformers library, where many research tools are first deployed.
To further improve performance, they searched for a way to access the model’s internal states without turning off graph replay, vLLM’s fast execution mode. MIA was the result. In experiments, Chen and Ko showed that using vLLM-Hook-MIA to steer relatively small models, an 8B Llama and Qwen 1.5B model, inferencing was delayed by just 2% in some cases. When they ran a much larger, Llama 70B model their steering overhead increased to only 6%.
Not surprisingly, MIA won out over methods that relied on turning off vLLM’s graph replay feature to enable runtime monitoring and steering. It significantly outperformed both the first version of vLLM-Hook and vLLM-Lens, a plugin developed by the UK government that also uses Python hooks to interact with model layers.
MIA even outperformed EasySteer, a steering method that ‘forks’ vLLM to add changes to a server-side copy of the official GitHub repo. Forking leaves graph replay on but requires developers to constantly sync their repo with vLLM, which the project officially discourages because of its frequent updates.
What’s next #
Tools that can provide real-time insight and control over AI content generation can be a last line of defense for enterprises with a low tolerance for risk. In experiments, IBM researchers showed that vLLM-Hook MIA can enable this extra layer of protection.
vLLM-Hook is already deployed in IBM’s Steerability toolkit, which enables developers to quickly design and evaluate steering methods for different use cases.
To give the open source community more experience using these tools, researchers at IBM and elsewhere recently announced The Steerability Challenge connected to the upcoming NeurIPS 2026 conference in Sydney, Australia. Participants can compete for $25,000 in prizes by composing a set of steering interventions that suppress “dishonest” behavior without degrading a model’s general capabilities.
Serving customers quickly is a top concern for enterprises deploying AI, but so are interpretability and control. With vLLM-Hook MIA, fast inferencing and a more transparent, controllable AI can go together.
Related posts #
- ReleasePeter Hess
- A new IBM Research specialization teaches how AI models run on modern accelerators Technical noteKaoutar El Maghraoui
- Making AI-driven chip design affordable with open agent factories and distributed inference at scale NewsPeter Hess
- How distributed inference keeps large AI models moving ExplainerPeter Hess