{"slug": "presentation-opening-the-black-box-critical-explainability-for-ai-agent-tool", "title": "Presentation: Opening the Black Box: Critical Explainability for AI Agent Tool Selection with Kiji Inspector", "summary": "Dataiku machine learning engineer and 575 Lab lead Hannes Hapke presented Kiji Inspector, an open-source tool that uses sparse autoencoders to inspect model activations and explain how AI agents select tools, arguing that prompt engineering, behavioral testing and chain-of-thought reasoning offer only plausible narratives rather than causal explanations. Kiji Inspector is one of 575 Lab's two signature open-source responsible-AI projects alongside the Kiji Privacy Proxy, which is designed to keep data from leaking to closed-source models.", "body_md": "## Transcript\n\n**Hannes Hapke:** Today we want to open up the black box of our lovely LLMs and agents and want to figure out what to do with them. My name is Hannes. I'm a machine learning engineer. I'm director of engineering at Dataiku, but I also have the great opportunity to run Dataiku's open-source lab, it's called 575 Lab. We work on open-source projects in the responsibility space. I've had the chance to co-author some machine learning publications, and I'm fascinated with machine learning systems, to put them into production, to work on open-source projects. I'm less the research person, I'm more like the person who brings machine learning models into production and building systems around it. I have the great chance to represent 575 Lab here. That is a wonderful organization within Dataiku where we can build open-source tools around responsible AI. Dataiku strongly believes that responsible AI has to be open, so anything that touches privacy or explainability, that is part of our lab.\n\nAnything we share here and anything you will see, it's fully open source. We also contribute to other open-source projects like vLLM, but we have those two signature projects, one is the Kiji Inspector, that's what we're going to demo today, and one is the Kiji Privacy Proxy. If you want to protect your data from leaking to closed source models, that is something you can do, and also you can fine-tune it to your use cases.\n\n## The Problem: Opaque Agent Decisions\n\nFor today, we want to focus on what we call the Kiji Inspector. It is a tool that is helping you to understand how agents \"Think.\" The fundamental problem that we have with agents is that we don't know how they actually make their decisions. We have a number of tools out there. We have prompt engineering, behavioral testing, or probably the most sophisticated processes like chain of thought. None of them are actually providing you causality, they're often just giving you plausible explanations or plausible narratives of what the root cause could be. When we were asked to figure out how can we make sure that the model that is running an agent is picking the right tool, how do we know that it's thinking about the right context, we decided that we needed to look into the model.\n\n## Part I - The Solution\n\nThe way we do this is by using sparse autoencoders. Before we get there, let me start explaining what an autoencoder is. This is a concept that has been around for a very long time, probably three decades by now. Autoencoders take a set of inputs, reduce this to a smaller latent space vector, and then reconstruct them through a decoder. The whole point of that autoencoder is that you want to minimize the error between the input and the reconstructed output. In an optimal system, there would be zero error, and the output here would be exactly as the input. Then you know that the representation of your vector z here in your latent space is most optimal. This has been used many times in various things like embeddings and vector searches and information compression, dimensionality reductions of various sorts of kinds. This is a really interesting way of forcing your model to reduce the amount of information to a smaller vector space.\n\nThe cool thing about this concept is that it's self-supervised. You don't need any labeling. You use the input as the output, that's your target, and then in between, that's what you would use in production. With sparse autoencoders, you would flip this concept around a little bit. Here in this case, we wouldn't create a bottleneck for the information, but we want to go wide here, with a little catch, and I'll explain the catch. Let's say you grab an activation from a model, or a hidden state, and you pass it through the sparse autoencoder, you can go wide in the information, and that means you force the model to pick certain features to represent the information. When you take a look at the activations inside of a model, there's something that we call polysemantic representation of the context, which means that one neuron that fires can have multiple meanings or topics it would fire on.\n\nIn our feature representation, if we want to understand the model, we need to separate them and put them apart from each other, and so that's why we're going wide in the feature representations here. Then again, we try to then have a decoder where we narrow that information piece again, and we try to reconstruct the activations that we used as an input. The catch here is that in this activation function, we are then forcing the model to only use a small set of features to be actually active, and the rest of them, roughly 95%, that's the ideal number, we want to force them to be zero. In that moment, we actually force the model during the training process to just be really selective on a handful of features, and then we can be absolutely certain that those features are representing the themes and the context of the model itself, what the model was thinking in this given layer at the processing of a certain token.\n\nThen we can go one step further. Once you have those feature vectors you generated, those indices don't mean anything to us, to humans. What we can then do is, once we have a corpus, we can grab all the documents that fire highly on, let's say, dimension 3 and dimension 6, and then we can look at those documents and say, what do they have in common? It could easily be that they say like, the first one has something to do with Python, so then over the course of the entire corpus, we could say, those two features, they represent a programming language or an API vocabulary or things like this.\n\nWhat we have tried to do in this project is combining a number of things and package them up so that everybody can use them in production. The first thing that we want to do is, sparse autoencoders have been around for a number of years now, and OpenAI published some research, and Eleuther published some research on this, but they focused on the entire LLM generation. What we changed here in our approach is we were only interested in the agentic decision-making. We focused on the decision token rather than the entire prompt, and then averaging those activations across the entire prompt. Then we put a bunch of concepts together. For example, Eleuther came out with this concept of token-level fuzzing, and so we added this to our framework so anybody who wants to use that framework and trains their own models, they can use that token testing. Then also we added causal validation testing as well so that you can make sure, is it actually causal or is it just correlation?\n\nThen the other sweet thing, what we changed here in this project, what is novel, is that we're not just training on a number of prompts, but we created those pairs, training pairs. We have a given intent — I'll show you an example later — but for a given intent, you have something that we call the anchor that's the correct answer and the correct tool use, and then there's an adversarial reply, which is very similar, but very different. It's basically like look something up in an internal database versus look up something online. Two different use cases, very similar in the prompt response, but very different outcome in terms of the tool selection.\n\nLooking at the training pipeline, let's say we have all these pairs generated, which I will show you how we do this. We then take our target model, and little caveat here, those have to be open-source models. We can't do this with OpenAI's models because we don't have access to their weights, but you can do this with any open-source model, and that's another novelty in our framework that we're not just focusing on Gemma models from Google or Nemotron models from NVIDIA. That tool, that framework here is working with various open-source models that are out there. You pick your subject model, that's the model you want to use the explainability for, and then you run all your prompts through the model itself, grab all the activations from the particular layers you want to watch. Then with those activations, you can train your sparse autoencoder as I just showed you with putting them in, widening them up, and then forcing the decoder again to reconstruct that activation.\n\nThen once you have the model trained, you can do a contrastive analysis, and I'll show you this later, how we then figure out through those pairs of what's actually relevant in the context or not. Or you can go to a large language model and say, tell me what those documents have in common, and then you can generate those themes. This all happens in that package where we call the Kiji Inspector. You put in your corpus, you nominate those models, and out you get a feature definition of what those features mean when you run those LLMs and agent requests.\n\nEarlier, I just mentioned you have to pick a certain layer. There's a little bit of an art what layer to pick rather than science. There's a little bit of an explanation here that if you're early in the stack, like maybe in the first third, the model hasn't fully conceptualized all the concepts of your prompt request. You're basically activating things that are highly relevant to the prompt itself. If you're further down the stack, there's a lot of intricacies of the model itself. Here in this particular model, we're looking at the Nemotron 3 from NVIDIA. That's an MoE model. That's because every different request goes through different routings, you have crazy errors. Learning something later on is very difficult. Also, the concepts you extract are so abstract in some cases that even for humans, it's impossible to understand what's going on. Then also for the models, it's difficult.\n\nWhat we wanted to do is, for our particular experiments, we picked layer 20 because it is already down the pipe. It's like the model extracted some abstract concepts. It's not too far down the pipe that it would be too non-human understandable. In our open-source models, what we're publishing, we're publishing multiple layers. You can pick wherever you want to be in your model layer setup. Again, this is another representation of the training pipeline. Basically, you have your input. You calculate those representations. Then you do the reconstruction back to, in our case, it's 2688 dimensions. Then you calculate the loss. It's a very standard autoencoder model architecture. The key thing, and we show this in the following slide, we're using a particular activation, what we call the Jump Regularized Linear Unit. I will explain this here why this is in this particular case.\n\nThen, how do we actually focus on the decision token? The way we do this is by forcing the agent to respond to a given prompt and says, I will use blank. Whatever tool, whatever the model then responds is expected to be the tool name, because we've provided the selection of tools in the prompt itself. Then in the response, we would see what tool the model has chosen. We do this a million times. We generated a training set with a million samples, basically 500,000 pairs, good and bad examples. The pairs come in later when we try to evaluate the model. During the training process itself, we use them raw, and we have those 1 million samples. Why are we doing this contrastive pair design again? Here's a little example. Let's say you have a given intent, resolve a password issue, and you can have the request, how do I reset my password?\n\nThat would trigger the tool knowledge base. You could also say, I tried to reset my password three times, and that could be a completely different tool. It's very similar in the response or in the request, but it's a very different tool outcome in this particular case. Later on, what you will see is we will then use those pairs and figure out which of the features we generated are actually relevant for those given tools. When we generate the features for both tools, the dimensions where the features are different have mostly contributed to the decision-making between knowledge base versus ticket lookup.\n\nI don't want to go through the math here, but there's an interesting concept regarding the JumpReLU. As you know, Regularized Linear Units basically are, anything negative is zeroed out and anything positive is the input. It's like a linear function, but it's not differentiable because of that bend. Here in the JumpReLU, that's another derivative of the ReLU activation function. It's basically the output is the input, but there's a threshold. The interesting thing here is when we train the model of, figure out which activation is actually relevant and not relevant, we can then tune that threshold during the training process as the output activation. That's another tweak we can have to teach the model which feature is relevant for which given activation. When we look at some results per layer, then we see some really interesting side effects of early on, the error is low, but the model hasn't learned much.\n\nLater on, the error is crazy high because this is logarithmic. Then we have too many, basically we learn something, but the error is exorbitant high. That's why we ended up in our practical example, we ended up somewhere in between at layer 20. Again, if we look at some stats here, like total, I think we had like 2,600-and-something input features, we widened this to 10,000 dimensions. Of those 10,000 dimensions, a little bit above 80% of those dimensions are alive. That means somewhere through the training, they have been nonzero. Then only less than 0.2% are completely dead, meaning those features have never been used through the entire training process. Then when we do the testing later on, we actually figured out that out of those 10,000 features, 668 are actually highly relevant to all the tool selections we wanted to investigate. This is for a given model for a given layer and a given dataset. If you retrain this, you might see different results.\n\nThen somebody could say, why do we have to go through all this craziness of sparse autoencoders and training another machine learning model in all these crazy setups? As you can imagine, it's a lot of computation there. This is basically a couple comparisons. If you take the activations and then train a linear model on this, could we then figure out is this tool correct or incorrect? If you compare this and a binary classification here, we get to almost 80% accuracy. It's not bad, but it's also not perfect. Then if you do some just accuracy versus F1 score, that's a little bit lower. Then somebody could say, what happens if we just take those activations with those 2,000 dimensions and we just cluster them? Could we from the cluster then figure out which tool is the correct one? You're better off flipping a coin than actually clustering.\n\nI think that was for us the interesting thing to see with the sparse autoencoders. I'll explain what fuzzing is. We can bridge the gap between interpretability and also having some causality testable features. I just mentioned token-level fuzzing. That was introduced by Eleuther sometime back when they looked into sparse autoencoders for entire prompts. What they did was they took a look at the prompt and then swapped out individual tokens, and then looked at how is the activation affected by swapping out those tokens and how is the end result being affected by that. That's the same concept we provide in our tool case here, in our framework that people can use when they want to evaluate their sparse autoencoders. Then when you look at the individual features and the quality of them, we can see that for the majority of the features they actually correlate really well to what we predicted the outcome to be. Only a small number of them have very poor influence on the overall decision-making. Again, from those 600 features we selected, it felt pretty good. This is heading in the right direction.\n\n## Part II - The Causality Test\n\nThen, let's take a look at causality and those other things. That's always the big question. We started this project where like, can we explain how an agent selected a tool? At some point, we needed to take a look at, can we actually prove causality? Little side note, I'm not a statistician, so I would love to have a better conversation about this with somebody who is more experienced with this. What we have done is looking at the activations. While the activation in this given layer in the model was produced, we then zeroed out the top 10 activations in that given vector. Then we looked at, is the tool selection changing or not? This was our setup. It's a little bit of like wiring and intercepting all the activations and those types of things. Don't worry, through the framework, we provide the other tools where you can replicate that.\n\nThe way this works is for our experiment setups here, where we looked at various tools, we were actually able to show that if you do the contrastive ablation, that worked extremely well for a number of tools. There were certain tools where we couldn't prove it, but for the majority of the tools, it actually looked very promising. Just a couple results here, the p-score is very low, so we were pretty happy about this. Through those testing, we were able to show that 23 of those contrast types actually worked. That worked better than randomly flipping. We felt that this is also, again, heading in the right direction. Like, yes, those particular activations, those 10 dimensions we blanked out or zeroed out actually contributed actively to the tool selection.\n\n## Part III - Using Kiji Inspector\n\nThis was a lot of theory and a little bit of math and all those types of things, but you don't have to do this yourself, and that's why we have the Kiji Inspector, and that's why we open sourced that at Dataiku. What we're trying to provide here with a toolkit and the framework is like, we wanted to provide a model-agnostic framework that everybody else can use in the industry. Companies like Google have published similar frameworks, but they're always very tool specific. For example, Gemma Scope is the explainability tool for Gemma 2 or Gemma 3 or Gemma 4, or there's no Gemma 4 expanded model yet. There is no training pipeline. There's a lot of repos floating around with scripts and everything, but it's not like something you can download and fork and run something end-to-end on your site for your particular models and your particular use cases.\n\nThen we wanted to provide a full production setup. We see a lot of research in those cases, but we hadn't seen a good production setup where people can run vLLMs and also get the activations back. Because in most setups like vLLM or SGLang, you get the response back but those inference tools don't give you the activations of a particular layer back. We provide a patch to vLLM that allows you to do that, and we also give you the containers so you can run this in production. Then as I said, everything is fully open source. We provide Docker containers, there's a PyPI package as you will see, and then all the models we have trained are on Hugging Face. This is only a subset, the list is way longer of what we're providing on Hugging Face. With our large training set of those 1 million data points or 1 million samples and the 500,000 pairs, we recently just trained Gemma 4.\n\nWho knows if Google actually comes out with their sparse autoencoders. We wanted to beat them, and so we have the sparse autoencoders for the Gemma 4 and then for the latest Nemotron models, the Nemotron 3, we also provide the sparse autoencoders. Unfortunately, we don't have infinite compute resources as Google, that's why we have to pick certain layers to support them. If you have specific reasons, and you want a particular layer of a particular model, let us know, we can probably help you there as well.\n\nHow would the inference setup look like? If you run for example an agent and says, analyze my Q3 report, you would go through a large language model. Again, it has to be an open-source model, because you have to host that model so you can patch your vLLM version. We're hoping to get this into vLLM at some point so that hosting platforms like Baseten and others can provide those features as well. Then, you not just get the response back but you also get the activation vector for a given layer for your response. Then that goes into your sparse autoencoder. From your 2688 dimensions, you would then go up to 10,000 dimensions. Remember most of the dimensions are actually zero. Then you can take a look at the top features that are active in this large vector, and then you can take a look at those indices and figure out what are the themes you have determined during the training itself.\n\nThen you can take those themes and put them back into a large language model as an explanation. Now somebody might ask, what's the difference now between doing all of this and chain of thought? The key difference is this information piece was directly derived from the activation during the model when that token was generated. It is not chain of thought where it first goes through some reasoning steps and then tries to execute. In chain of thought it could say like, here are two things I want to do, and then it executes 15 other things. There's no connection between those two steps. Here, there's the direct connection between the activation at a given layer. It goes into the sparse vector, and then we have the connection to the feature map we generated during the training.\n\nHow does it work in production if you want to use this? We've generated this PyPI package called the Kiji Inspector. Until recently you also had to update your transformer libraries for Gemma 4, but all the Gemma 4 changes lined it so you can use the latest library here. Then the setup is very straightforward. If you're familiar with Hugging Face, that's how you would infer a model. In our case we define like a Gemma model. We give our given prompt. This is like a home repair thing. \"My dishwasher is smelly. What should I do?\" Then we define our layer. The Gemma 4 model, this is the E4B model that's a little bit smaller so we go a little bit earlier in the stack. In this case we use layer 8. That setup is standard cookie cutter. The only difference, what I would point out is like, if your GPU supports bfloat16, use it. It's just an inference. You don't train anything so just save the memory for your inference. Then again this is just standard prompt stuff. Nothing special here. Nothing unique. Very boilerplate Hugging Face code.\n\nNow the actually fun part starts. In Hugging Face, you can define what we call forward hooks. When the model is being executed, in Hugging Face the model is not compiled so you can easily hook into the execution, and say, give me the vector of a given layer for that given input. Then, we can grab the layer from here. When it's being executed, we can then actually ship it back to the response and send it to the user. I can show you another setup we have where you can also work with compiled models so you get a better performance. Then you run the inference, you retrieve the vector, and that is your activation. You're ready to go to use your sparse autoencoder. If you use one of the models we already support, you can then use our Kiji Inspector. We followed the Hugging Face setup. You can say like, from pre-trained, you mention the base model and under the hood we figure out like, what sparse autoencoder it needs to download.\n\nWe go to the correct repo, and then we use the correct model for the correct layer. Under the hood we convert this into the correct feature vector and then also do the lookup of the features to the themes. When you run SAE describe, that's how you get the description and not just a random vector. This is how it looks like, and I will show you this, again, but this is a little bit better readable. You see all the indices up here, they basically tell you what indices is the top five the ones that are firing. You see the number one it says unknown. What does it mean? It means like during the training, that feature in that feature vector was not of high importance and we haven't described that. Maybe we need more training data or maybe we need to tweak the training process. The interesting thing is that when we look at the second feature that was firing, it was actually something related to dishwasher gasket issues.\n\nSeems relevant to the prompt we have. If it would fire on something completely unrelated here then there would be a good sign to go back and say like, maybe there's something going on in the model or maybe my prompt is incorrect or something else happening. Those are the things you can then see from the inspection and the data we provide through the package.\n\n## Demo - Kiji Inspector, Live\n\nLet me show you here a little demo of how this would work and how you can actually use this in production. Again, this is the same code that I showed you earlier where we just stepped through. I pre-ran this in a Colab notebook. If you want to follow along you can go to our GitHub repo. The quick start example works in a free Colab. If you want to use a larger model you have to pay for the bigger GPUs nowadays. This is still giving you an overview of how this works under the hood. Here's a truncated information on the slide. You see some more information, and we also provide the bottom examples. Basically, here you see all the things that are least related to that request. Then you can go down, and you can turn this into a description and you can see what the confidence level is.\n\nHere's another thing that I think could be really interesting for folks. As I mentioned at the beginning, we provide patches for vLLM. Here in this case is like you can also execute the whole thing in vLLM. When you do vLLM generate, you can then also get the same vectors back with the explanations. We tied this all together in vLLM. Then we have another interesting example I think that is a use case where we're stepping through a larger agent setup. Instead of going through the code I want to show you how the result could look like, and maybe that gives you a feeling of how you can use this in your product. We built the standalone demo of like, this is your home repair agent. It tells you if something is broken, can you fix it yourself? Should you call somebody else? It could potentially look up YouTube videos of how to fix it.\n\nIn order to make sure that we know what the agent is doing, for every agent step, we then also look up what the sparse autoencoder features are. Here in this particular case, this was the dishwasher example, it relates to a machine handling water I would think, washing machines, and there's something like old machine troubling. At least we know that the agent is heading in the right direction when it made the decision of like a dishwasher is leaking in this particular case. You can do this for every different tool call. This one is completely empty, it's a good example, because when we trained this there was not a tool call in the training set at all for YouTube videos. For us it will be a good sample of like, we need to add more real-world examples in our training set. I showed you this example. There's a couple more, there's a left example there as well where we show this in a live demo, you can follow along what the agent is doing in the frontend while the agent is figuring out what is actually happening. It's pretty fun to look at.\n\nWhat are the particular use cases you can use for your applications and where you could use the Kiji Inspector. One thing is like you could figure out what an agent misbehavior would be. Anything that is like, has it picked the wrong tool, as I showed you earlier in the examples, that is a something. You can do it for audit compliance. Some companies looking at this and figuring out, it's like, if we have another track record, traces is one thing but explainability is another thing. When we do traces, we basically see what tool has been selected but we still can't explain why that tool was selected. Here we generate another set of logs and figuring out why the agent has particularly selected this tool that is shown in the traces. Then we can do validate of prompt engineering. If you're not quite sure, like you want to optimize your prompt for particular goals and you don't want to do this iterative loop, you can then also investigate like how is that tool responding to your given prompt based on the activations.\n\nThen obviously monitoring production drift. If your data changes, let's say you test your agent with a set of prompts, you watch those activations, and then over time you can see if those activations change. That is a really good indication that the data that is going through that agent has changed. Maybe you should take a look at that as well.\n\n## Key Takeaways\n\nJust to summarize this, what are the key takeaways? We have the chance to prove that sparse autoencoders can work to explain why the agent made a certain decision. We showed how the token-level fuzzing works, and as part of the package we can prove that tweaking those prompts lead to different activations. I also showed the causal evidence of the tool selection is affected by zeroing out certain activations in the model itself. Then we did some comparison with construction only baseline, meaning, because it's a decoder and there's a loss effect, if we take this out, then we can actually show that there's a causal effect in the sparse autoencoder.\n\n## Limitations and Future Directions\n\nI want to let you go with a couple limitations and also pointing out some future work. As I said, you do need the model itself. It doesn't work with closed source models. Hopefully at some point they give us similar tools on there, and that would be fantastic. The training is pretty compute intensive, especially if you want to create your own datasets. If you want to use the tool and you want to create your own training sets, come talk to us. We can share some more tricks and tips of how to run this effectively. As I mentioned earlier, it is a synthetic dataset. If anybody wants to use this and has some ideas of what is missing in the dataset and where should we go in terms of making it more real-world replicable, then let us know. We would love to collaborate on that as well.\n\nThere's a future work here. We want to support more open-source models and provide those models from the start through Hugging Face. Something that is on mind is currently Qwen. More multi-layer experiments. Currently everything is based on single layers. It could be really interesting if you take a layer from the first third and then a layer from the last third, and then combine those and see what actually happens. Then another interesting idea we had was like what happens if you do cross-model transfer. Basically, you train on a Gemma model. You take those vectors. You learn certain features. Is there something we can generalize then if we go to another model and so we see certain patterns. That is all future work. We haven't touched this yet. If anybody wants to collaborate on this let me know. Everything is fully open source. We live from contribution from others. This is not our project. It's a community project.\n\n## Questions and Answers\n\n**Participant 1:** Is there anything that stops this from being applied to something by the training, of course, beyond tool calls. For example, if we were looking at safety alignment. The reason being that at least from the results of the causal inspection that you did, it seems to be possible to do some kind of steering after you train the model. Do you have any insight on any ways that we could possibly change something like this for alignment and then use that for active steering after deployment.\n\n**Hannes Hapke:** I think there's nothing that I would say that it's not possible. One of the things we're looking into is the security aspects of like, can we detect malicious attacks in those activations? Yes, I think that's possible. I think from the research I showed earlier when I showed the clustering versus linear classification versus the sparse autoencoder, it was visible to me. I think to me that's sort of the proof that if you just look at their pure activations you can't figure out what is going on. You do need the step through the sparse autoencoder. Yes, nothing is limited to just a tool call. For us, it was like a feasible project scope. The team is pretty small, so for us it felt feasible. Yes, totally.\n\n**Participant 2:** Do you have any real-world examples where using your approach gave you a different understanding of why a tool was selected versus using chain of thought tells?\n\n**Hannes Hapke:** Yes, we have a bunch of anecdotal examples. We haven't summarized them yet into something scientific that we say here's my prompt and here's my analysis. Chain of thought was maybe a bit abstract. Then in the features, we could see like, this is actually looking for a particular problem. For example, this dishwasher repair thing, it picked water damage on an electrical equipment or something. It felt more precise than the actual chain of thought. There's a little caveat. The sparse autoencoder we trained for those examples, they were domain specific. Meaning that we were focusing on a very particular specific domain and that's what we built the training set for. There was a little bit of an unfair advantage. Also, it changes with model size. If you use a smaller model like the Gemma E4B, I think those interpretability tools have a benefit of it. Those were the two things where you could see the difference between the chain of thought versus the explainability tool.\n\n**Participant 2:** For those who are stuck with the closed source proprietary models, could you imagine surrogate models working ok enough? Do you think that the closed and open-source models are similar enough with why they would make certain choices?\n\n**Hannes Hapke:** I can't speak for OpenAI and others. I would think the models are bigger than what we see in the open-source world obviously. I think you have maybe a better chain of thought experiments. That's an interesting question. We should compare different prompts and run them through a large Qwen model versus a large like OpenAI 5.5, and then look at the thinking steps, and see, are they comparable or not? Generally, we will never see this technology in closed source models, because, A, I think all the companies are walking back their logics now, because through the logics you could figure out model internals. This is even more like internals. They never want to share this. The other thing is also like, yes, it's proprietary modeling. They don't want to really open source that or share that.\n\n**See more [presentations with transcripts](https://www.infoq.com/transcripts/presentations/)**", "url": "https://wpnews.pro/news/presentation-opening-the-black-box-critical-explainability-for-ai-agent-tool", "canonical_source": "https://www.infoq.com/presentations/explainable-ai-llm-agent-decisions/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global", "published_at": "2026-11-12 11:00:00+00:00", "updated_at": "2026-11-12 11:23:10.582129+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-safety", "ai-tools", "ai-ethics"], "entities": ["Dataiku", "Hannes Hapke", "Kiji Inspector", "Kiji Privacy Proxy", "575 Lab", "vLLM"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/presentation-opening-the-black-box-critical-explainability-for-ai-agent-tool", "markdown": "https://wpnews.pro/news/presentation-opening-the-black-box-critical-explainability-for-ai-agent-tool.md", "text": "https://wpnews.pro/news/presentation-opening-the-black-box-critical-explainability-for-ai-agent-tool.txt", "jsonld": "https://wpnews.pro/news/presentation-opening-the-black-box-critical-explainability-for-ai-agent-tool.jsonld"}}