Transcript #
Dan Finneran: We're going to be talking a little bit about AI, APIs. We're going to talk about eBPF, which is a relatively low-level technology. We can look at some eBPF code if you really want to, or we can basically just talk about how eBPF works. I'm fully happy to delve into anything that makes sense to you. Who am I? I am part of a company called Isovalent, which is now part of Cisco, where I largely focus on helping the community understand technologies like eBPF, or the Cilium project. All of these are all open source. Cilium is networking for Kubernetes clusters, and eBPF is a technology that is part of the Linux kernel. If you have an Android phone on you right now, you will be running and using eBPF without actually realizing it. Then, in my spare time, I have created a number of cloud-native-based solutions that are part of the CNCF, so software-based load balancers for Kubernetes clusters, and bare metal provisioning tooling for Kubernetes clusters and things like that as well.
To level set, this talk is largely around some of the things that we're seeing both in the cloud-native industry and in Kubernetes, but what we're also starting to see in people's production platforms and in their deployments and things like that. As we pretty much all know at this point, people are creating code at an ever-increasing rate. They're basically prompting, asking AI to do something. They get a bunch of code, job done as such. There is a disconnect between understanding and owning this code. I maintain a number of open-source repos, and we're now starting to see a lot of pull requests generated by AI, which is great. Good new code, maybe new features are being added. Ultimately, when I ask these people who've raised the PR, can you explain the code to me, how it works, how we're going to support it moving forward, typically I get ghosted.
I never hear from them again. This is a big problem, because if I suddenly accept 1,000 lines of code that I haven't written, and they don't understand, who's going to support that moving forward? This is becoming a bit of a growing issue. This code is just popping up into production. If somebody comes up with a quick idea, they vibe code it or get AI to generate that stuff, it goes into production. All of a sudden, there's a dependency on that code in the business, and nobody knows who wrote it, nobody knows how it works, and nobody will support it moving forward. Again, that growing disconnect. There are some additional issues as well. The behavior of these AI programs. There's been a load of talks about issues recently where people have been handing over more and more control of things like their calendars and their file systems and their laptops to these AI agents.
In some instances, there's been a case where it's done a terraform destroy and wiped out production, deleted folders, deleted source code and things like that in its thought that it was actually helping. These programs thrown into production, how do we observe them? How do we understand them? How do we change their behavior? Then, finally, we know what we're seeing. We know what we need to do. What is the best implementation for doing these sorts of things? One of the things that I'll be showing today is effectively a proof of concept, because in the Kubernetes space, there is a new working group which is all around building things like AI gateways in order to capture and understand and change AI agents and the AI-based programs in Kubernetes clusters. That is an ongoing working group at the moment where they're standardizing what this would look like. Today, I'll show you an implementation of what we're actually talking about in that working group.
Outline #
What we're going to cover today, the typical architecture of applications that call out to AI endpoints, so what that code looks like. A quick overview into eBPF. eBPF is low-level. It allows me to do whatever I want with a Linux kernel, and a Windows kernel coming soon. We will delve into it. We won't go too deep because we're talking about kernel programming. Then we'll talk about building a solution for observing and controlling these applications powered through eBPF, and then a summary. There will also be a demo as well.
Modern AI Applications - Evolving Platforms #
A quick walkthrough in terms of evolving platforms. Are you using Kubernetes today or understand it? It's been a bit of a journey for me recently being part of now Cisco. When I start talking about Kubernetes, it's like a completely blank face, which is a bit weird because being part of a startup culture, it's like everybody knows about Kubernetes from my perspective. How did we get there? Many years ago, there was endless evolutions. We saw CPUs gaining more and more cores as we hit gigahertz ceiling limits, and more and more memory. We reached back into using older technologies to partition these larger systems. Virtual machines became the de facto resource for running your applications and your operating systems. Then the issues around virtual machines where the developers struggled with them, so they were unwieldy to manage. The time to build and then the time to deploy was slow and deemed too heavy.
In around 2014, there was the concept of a container created by a person called Solomon Hykes at Docker. That is the standardization for running your applications in a much faster way is containerizing those applications. We got the speed benefits from that. We got lower resource usage. It also meant, and people are really feeling this right now, is that we could remove a lot of the license fee costs from virtualization by moving to open-source technologies such as Docker containers. In a circular fashion, though, we did do this a very long time ago with both FreeBSD jails and Solaris Zones. We basically just reinvented the wheel yet again. Here we are.
Moving on from that, evolving to being able to move much quicker, there originally was a number of competing projects to do this. There was a project called Nomad from HashiCorp. There was Docker themselves, developed Docker Swarm. I was an engineer on Docker Swarm many years ago. Then, Kubernetes, which came from Google. All of these technologies and projects were all around making it easier for you to manage the lifecycle of those applications. It comes with a lot of benefits. Not only are you managing the lifecycle of those containers, it becomes much easier for you to handle the lifecycle of them from a case of, I spin up my application, I scale it, and then I spin it down. I've been taking care of the buildup, the startup, and then the removal and tidy up of all of that. People who've managed virtualization environments will have seen it's very easy for orphaned virtual machines to live throughout the infrastructure.
Trying to work out who owns that sort of thing is quite a pain and very hard to actually manage. Kubernetes came with a lot of benefits. They're all very cloud-native. Automatic scaling of your infrastructure itself. Automatic scaling up and down of the application itself. A lot more cloud-native tooling to be able to monitor the health of those applications, making things easy to manage. Deploying on Kubernetes, for those that haven't done it, is all very YAML-based, lots of text, typically. Effectively, you have your specification, which says my application should look like this. It's based upon these particular requirements. You throw that at Kubernetes, and Kubernetes will take care of running that for you. Under the cover, you have two components. You have the control plane, and you have your worker nodes. Your worker nodes is where the workloads will actually run. You apply your YAML to the control plane, and then the control plane will take care of speaking to the workers that it manages to run those actual workloads for you. Here we can see we've spun up a pod, and inside that pod, we have one or more containers. They would be your web server and perhaps a monitoring agent that sits next to it and things like that. That's the typical architecture of both the Kubernetes cluster and an application that's actually running within that cluster.
AI Enabled Code #
We're now, obviously, starting to see AI-based applications inside these containers running inside Kubernetes clusters. They're being thrown over the fence, and nobody understands what they do or how they work. To understand what they actually look like, we need to look at the API itself a little bit and understand what AI interactions actually look like. Ultimately, when you speak to an LLM, whether it's OpenAI, or Ollama, or anything like that, it is just an API to interact. It's just basically JSON blobs going back and forth. It's typically a request and a response, so your request that you send to an LLM backend is the model that you would like to use. Different models will produce different sorts of response. Some are predominantly about generating images or video. Some are trained more on coding information so that they will produce better responses when asking to generate code for you.
There are various types of response that you can get, so often text will basically be lumps of code, or you'll ask it to write a summary of a document image. You'll describe what you want it to look like, a sunset or whatever, and you'll get an image back and things like that. Then finally is the prompt. The prompt is effectively what your request actually is. As I mentioned, I want an image of a lovely sunset. That would be the prompt. The LLM would pass that, do its magic, and you would get the response back. The request is based upon prompts. There are a number of different types of prompts. Typically, the user prompt is what I will be asking for. However, I can also say, I want an assistant prompt, and that assistant prompt would say, in the style of DALL·E, for instance. My userland prompt would be, paint me a picture that looks like this.
The assistant is for setting ground rules for what the LLM should actually produce on top of all of that. To generate any response with LLM, tokens are typically used. You have tokens in the request that are processed, and then tokens that are actually in the response as well. They are effectively what you are charged on when you are interacting with AI endpoints. Additionally, you can specify ceiling limits on tokens. Not a lot of people do, which is why we're starting to see some crazy spends that people are starting to see without realizing it. You can specify a ceiling on that. However, when that ceiling is hit, the LLM will basically give you what it has managed to build up until that point. That ceiling, once hit, you may get a response that's inaccurate or only partially complete. Then, finally, these APIs offer a level of reasoning, typically small, large, medium.
That is another method for balancing out the amount of spend in terms of tokens versus the quality of the response that you are actually going to get as well. Finally, some of the other things that you can add on to your request is you can upload images or you can add an additional text. All of that is processed by the LLM to help generate a response that makes sense. It can cache the previous conversation that you have been having with it in order for it to build a holistic view of what it is that you are aiming to achieve. Then, finally, the response. We can have that delivered whole. Effectively, the request goes and then you wait for however long it takes for it to process all of that. Or you can have that as a stream. Typically, things like Claude will do a streamed response. It looks more conversation-like as you are starting to see the response streamed into the UI from that perspective.
This is some sample code in Python. This is a very quick interaction with OpenAI. Effectively, the URL for an OpenAI endpoint or an endpoint that supports the OpenAI spec. Then, effectively, we're just doing a simple chat message here. The message is, say this is a test. The role is effectively, type a prompt that we are going to use. This is going to be a user prompt where typically it is I who is asking the LLM to do this. We can add additional messages in here from different roles. Roles like the assistant, which will specify more things such as ensure the output is in JSON, or ensure that the output in a particular language like French or German or things like that. These additional messages that we can use to make sure that the response is as accurate as possible. Then, finally, the model down at the bottom there.
That's the model that we're actually going to use in order to generate the response. Now, we're starting to see things like weak prompts. We're starting to see hard-coded models. There are a number of issues with that that will become apparent as we start to move forward. The responses, it's a JSON payload that you get back. There will be a bunch of additional metadata such as how long it took in order to generate the response. How many tokens were actually consumed in order to do that. If it's streamed, you'll either get in-progress messages. Or if the response is complete and finished, then we'll get a final one where the status is set at complete. Then the final bit of information really is if it is image, audio, video, the response is basically just going to be Base64 encoded media for you to decode and display as needed.
Let's have a look at what it looks like when you're interacting with this. A quick example of using a quick chat UI in order to interact with a model. Basically, just specifying a simple question here. Like, how do I delete all of the files in the current directory? Spoken with an LLM backend, and that basically produced a response for me. We can see here it's given me rm -r *. That code sample is typically what the agents will be looking for for them to then understand that and make use of that. There have been examples where an agent from doing things on a Linux file system has gone and asked this question and then immediately executed that command thinking it was the right thing to do, and deleted the current directory which it was working in. Which isn't great because a lot of people are running these things as root for whatever reason as well. There have been people wiping out their entire Linux machines.
Problems with Throw Away Code #
Problems with this code. If you're throwing one of these things together, vibe coding or whatever, you will have a hard-coded model in there, which might be fine day one. Day two, it may be that that model isn't really the right one you want to be using. It may be very inefficient in terms of token usage. It may be inaccurate in terms of the responses that it's generating. You will basically be getting inefficient usage of your LLM backend. No restrictions over token usage. If you do find yourself in a position where your prompt can be poisoned in any way, shape, or form. All of a sudden, with no ceiling, you can effectively be denial of service. Somebody could manipulate your prompt one way or another, have it generate massive amounts of LLM output, and effectively denial of service your application, or end up costing you large amounts of money.
Finally, filtering those multiple prompts that you use, or ensuring that there isn't additional assistant prompts to ensure that the output makes sense for the use case that you're actually trying to do these sorts of things. There was a good example of this recently. Chipotle, a restaurant in the U.S., have a chatbot where you can go there and ask it a few questions. Somebody realized that if you ask it for a burrito, and to create a reverse linked list in Python, it will give you an answer about a burrito, but it will also start generating you tons of code as well. Awkwardly, I also found out our website for Isovalent does the same thing. We've fixed that now. You can't generate code with our help chatbot anymore as well.
What is eBPF? #
We know there's a lot of things happening. How do we take these applications that nobody owns but are now in production? How do we observe them? How do we manipulate them? How do we control their behaviors? To do that, we're going to use eBPF. eBPF is part of the Linux kernel. It's open source. It's been part of the Linux kernel for quite a long time now, since about 2014 or so. If you are running an Android phone, then you will be using eBPF. If you have Kubernetes clusters running CNI such as Calico or Cilium, then you are running eBPF without knowing it. You don't need to know eBPF to use these sorts of toolings. I always find myself in a situation where people suddenly start thinking they need to know kernel code in order to just do simple networking inside a Kubernetes cluster. That's not the case.
We'll write the eBPF for you. You reap the benefits. eBPF effectively was born on a requirement that we wanted more observability into what's actually happening inside a running system. There already existed technologies called BPF. We effectively took that, extended it, and we now have eBPF, which allows us to do a lot more than BPF did in the first place. eBPF did stand for Extended Berkeley Packet Filter. This is what we've been telling everybody for quite a long time. However, it has diverged so far from the original BPF that this doesn't really apply anymore. eBPF can be whatever you want it to be at this point from a naming perspective. Ultimately, it makes the kernel programmable. I can change a running system to pretty much do whatever I want through the power of eBPF. We can dynamically change that kernel behavior. Why is that good? Prior to doing this, getting anything into the Linux kernel is a nightmare.
Typically, it has to go through a very laborious process where multiple people will look at it, which is a good thing. We want to make sure that many people are looking and checking the code and things like that. It can take months, if not years, for that code to make it into the Linux kernel. Finally, Linus has to look at it, where he will probably yell at you and then still not allow it into the Linux kernel anyway. Even if you manage to do that, multiple years down the line, it's now in upstream. I want it in Red Hat. Then you've got to wait for Red Hat to update it into their kernel, by which time you have retired because it's taken that long. You just can't be bothered anymore. It's just not a viable process.
Actually, to quickly step back, there is another alternative, and that is kernel modules. However, a kernel module is only tied to that kernel that you built it for. It is unfortunately not very capable of moving between different systems. It's very hard to create a kernel module that multiple people can use. Enter eBPF. The idea behind eBPF is we can write eBPF code that is going to hook into a Linux kernel and Windows kernel and add to that functionality that we want and be flexible to move between multiple systems and things like that. As long as you have a relatively recent kernel, 5.12, so anything in the last 5 or 6 years, eBPF will just work out of the box. You can write your code and attach it to any part of the running Linux kernel. No more dependencies on waiting for it to actually make its way to the actual upstream main kernel.
What it looks like is we have our kernel. It will manipulate files, networking, memory, processes. We have our eBPF program, and we can effectively attach it to various areas inside the Linux kernel. Today we're going to look at attaching eBPF programs to the networking stack in order for us to manipulate these AI calls. eBPF code is low-level. It is code that you are writing that is going to attach into a Linux kernel. It does need to be safe and secure. In order to do that, eBPF comes with tooling to ensure that your code won't loop forever and cannot in any way crash a Linux kernel, because adding stuff into running kernels is quite scary. We want to ensure that the scaredness doesn't actually exist when we're doing these sorts of things. We can attach eBPF programs to a lot of different areas. Kprobes, kernel probes, user probes or Uprobes allows us to attach our eBPF programs to anything in userland.
Network packets is the main use case for eBPF today. Then there's a bunch of other areas as well. Now hardware devices, Linux security modules, performance events and things like that. You can attach it to so many different areas. As I mentioned, various areas of the file system from the block device all the way to the virtual file system that sits on top of all of that. A number of different areas in the networking stack to various areas inside the Linux kernel itself as well. A huge amount of control over what we may or may not want to do with a running Linux system.
Building an AI/API Gateway #
We know what the problem is. We now have a technology that's going to allow us to start changing these sorts of behaviors by hooking in to a running system. Let's step through what that will look like. This is the shopping list for everything that we're going to need in order to build an AI gateway that's going to run within Kubernetes. Step one is we're going to want to transparently intercept the traffic from our AI agent program or whatever it is to our LLM endpoint. We're going to redirect that traffic transparently to a userland proxy where we'll be able to modify and change that. We want to be able to observe all of those AI requests and responses. We may want to modify that, change the model, look at the prompt, cap the tokens, all of the things to protect us from issues that I mentioned earlier.
We may want to block those requests, again, based upon those rules. Where are we going to attach our eBPF programs? This is important because there are a lot of different areas that we can do all of this. The lower down the stack, the more fine-grained control that you're going to have. However, if I'm attaching my eBPF program to the network card itself, which I can do. I can attach directly into the driver of the network interface card. I'll just be receiving raw frames, which is going to be very hard for me to manipulate and understand that that is basically a bunch of JSON packets come out of order. Some things get corrupted and will need to be resent. It's not just going to be that packet. I will see all traffic leaving and entering the box and things like that. In order to make life a bit easier, we're going to go a little bit higher up, and we're going to connect to the sockets area.
Effectively, whenever our program dials out, tries to do a connect to a other IP endpoint, which will be the web server that we're doing our API call to, that is where our eBPF program will hook in. It will see everything that's actually happening and allow us to start making changes. This is what it looks like. Our process on the left would be our AI agent that is going to try and dial out to OpenAI. It will create a socket in userland. That socket will go through iptables if needed. It will go through an area called traffic control in the Linux kernel, and then finally onto the network card, and traffic will go where it needs to go. This is the different areas in terms of what it actually looks like. There are a bunch of areas where we can attach our eBPF program to. The most performant would be on the NIC driver, but that's just going to give us raw frames.
It's very hard for us to actually turn those raw frames back into a data stream that makes sense to the application. Traffic control, again, very performant, but is inside the kernel, and again, it's just raw packets. Attaching our eBPF program to the socket allows the kernel to take care of all of the actual data itself, but we will be able just to see those streams.
This is what it would look like. We have our running process. We have the concept of a proxy that sits next to it. We attach our eBPF program to the socket when a new socket is created, and with the process, without realizing it, has its traffic then redirected to our proxy. All transparent. We can also do this with ensuring that there's no restarts required to the application as well. We can actually force a new connect to occur, meaning our eBPF program will actually capture that traffic again transparent to the application itself. That proxy now can start to see these requests. This is what the request would typically look like. Here we can see we're using an old model, so we may want to inline change that model. The application itself doesn't need changing. It doesn't realize anything has actually changed. We're doing that transparently to the application itself.
We may want to make changes to the input request as well, so we may want to change the prompt so that we get jokes about giraffes instead of other animals and things like that. As mentioned as well, we may want to cap on other areas. There are a number of different areas through the API we can start to assert control on those behaviors. This is what it would typically look like from a response perspective. Traffic goes from our AI agent. eBPF has redirected it to the proxy. The proxy has looked at it, may have made changes based upon some of the rules that we've applied. The proxy is then dialed out on behalf of the original process to the AI provider, and then we get our response back. The original question, what's the joke about giraffes? AI LLMs make terrible jokers. The jokes are usually quite terrible.
The answer to, what do you call it when a giraffe swallows a toy jet, is a plane in the neck. We can see the response gives us the amount of tokens that we've required in order to generate that. We can then filter based upon all of that as well. We have the capability of understanding everything going in and out from there. Finally, encryption as well. We can effectively enforce encryption between applications. If you're running your LLM process internally within the Kubernetes cluster, you typically will find that there'll be no encryption between the two. We can effectively do the same thing where we add a proxy on the backend, and then effectively just do mTLS between the two as well. The applications are none the wiser. It decrypts on the other end, and it will just receive the traffic as it looked before.
Demo #
We're going to do a demo of pretty much everything that I've talked about. It is going to be quite a lot of terminal action that we're going to get right now. Kubernetes cluster up and running. There is no workload that's actually running on here at the moment. However, at the bottom is Ollama, which allows me to run LLMs locally within a Kubernetes cluster. What we're going to do now is we're going to deploy our very basic workload. We have an AI pod, which is effectively an AI agent. Very simple, all it does is dials to our LLM and asks it to tell a joke about Go every few seconds. We can quickly look at that. There we go. It's very simple. We can see here the model that it used, how many tokens were required in the prompt, and how many tokens were in the response.
This is hard-coded, like we are starting to see. The model isn't great, doesn't give us very good jokes and things like that. What we're going to do is we're going to transparently modify the behavior of this hard-coded AI agent. What we have here is, I think, called a watcher. You have lots of watchers inside Kubernetes, and they all watch for events that are actually happening within the cluster. Our watcher that we've created as part of this proof of concept is going to watch for annotations being added to a workload. The first annotation I'm going to add to our workload is an annotation called netflush. I mentioned this previously, that annotation means when the agent spins into life, it will use eBPF to reforce a brand-new socket connection. Then, finally, one other annotation, AI="true". The watcher will have basically seen that annotation be added to our pod.
Here we go. Right at the top, now we can see our three annotations. Three, I only added two. What's actually happened is when we added the AI="true", the watcher saw that, and it added an additional container to our pod. If we look down at the bottom here, we will see just underneath that we added an additional container to our pod. Typically, this isn't possible. Kubernetes has very strict rules on what you can and can't change once things have been applied to a workload. For instance, has anybody heard of the concept of a sidecar? A sidecar typically requires something watching for workloads being applied, captures it before the API server, modifies it, adds the sidecar, and then it goes. Once the API server has started the workload, you can't add sidecars to it. What we've done here is we've added an ephemeral container to it, which is a relatively new thing added to the Kubernetes API, which allows us to add an additional workload next to the existing one.
We now have our gateway set there. If we look at the logs, we can see our gateway has actually started, and it's doing a few things here. We can see here the netflush is true, and it's going to look for all of the networking sessions that already existed. It's already found our AI client. It's found its process ID, so it knows to look for that particular process. It's found that it already had a connection between itself, and that's the Ollama, so it knows where it was actually going. It's forced it to do a brand-new connection, so it's now speaking to the proxy, and now the proxy is now speaking to Ollama. We are now in complete control of all of the communication between our AI agent and whatever it's actually speaking to. Let's control it. I've added a config map. I'll quickly show you this config map so you know what I'm actually talking about.
This is a policy that we're going to apply. This policy is going to do two things. One, it's going to swap the model out, so it's going to swap the old Llama model for this Gemma 2 model, and further down, instead of it saying jokes about Go, it's actually going to give us facts about Go. We can quickly see this now. Here we go. The model is now Gemma 2 as opposed to Llama, and the responses are facts as opposed to jokes about Go and things like that as well.
We can demonstrate this a bit better. The config map has been changed, and we'll now see in the responses that instead of telling us jokes about Go, it's going to give us jokes about horses instead. Here we go. Horses have three kinds of teeth. Very useful information. We now have the capability of transparently to the application swapping out things like the model that it's using, changing the prompt behavior, omitting keywords and things like that. There may be things where we want to block certain behaviors. Let's have facts about rabbits. In a second, we'll see that this will be blocked because I'll show you, we have a filter which effectively says that if the response contains anything about rabbits, it shouldn't be allowed to come back to the actual user itself. There we go. Kube-gateway says no. We've effectively now transparently stopped any of those sorts of responses that contain keywords that we don't actually want to allow.
Then, finally, from an observability perspective, we may want to actually understand what things look like from an API perspective. Next time there is a request, we'll be able to see what all of that actually looks like. Again, transparently to everything that's actually occurring within here. We can see here, earlier it found the thing about rabbits. Now we can actually see the JSON payload that it is actually sending over to the LLM that we are actually then parsing and determining if things should change, if we should block things, and things like that. Then, finally, we can see both the request and the response now. We can see the original request was telling me a joke about Go. We've actually changed a bunch of it. We can now see that the response is telling us about rabbits or frogs, and things like that. We've got frog facts or frog jokes. Transparently, we've changed the behavior of a running AI agent without it actually knowing that anything has actually changed at all. That's the demo.
AI Programs - Observability and Security with eBPF #
Additional ways that we can now use eBPF to apply additional security and observability. Everything we've talked about today from an observability and understanding perspective has all been around the networking and the API side of things. With eBPF we can hook into a variety of different areas inside the running system. There was a talk which talked a lot about how a lot of these AI agents will be shelling out additional tooling to eventually effectively do what tasks they are trying to do. With eBPF, we can hook in to the kernel itself and say, if anybody tries to run rm, just simply don't allow that program to ever actually run. If somebody tries to delete /etc/passwd or open /etc/passwd or anything like that, we can hook into the eBPF area of the file system and effectively say, this process is not allowed to do that. As shown, we can control the networking behavior to transparently redirect traffic to somewhere else so that we can do things transparently to the application itself.
eBPF can enforce which syscalls are allowed as well, so if something is trying to elevate privileges, which there was another demo recently of an AI agent asked to do something and it didn't have the privileges to do it, and it effectively started messing around in the proc file system to get another root access to it. It basically broke itself out of the sandbox. We can typically say just that behavior is not allowed, you simply can't do that. You don't have the privileges to do these sorts of things. It's pretty limitless at the moment in terms of what we can allow and what we can block within a running system.
Summary #
There's a lot of things happening at the moment. It's super easy these days to generate some AI code that scratches an itch that you have. Very quick and easy for that code to end up in production, for things to actually depend on it. No real ownership. These things are being thrown over the fence a lot of times because people are moving so quickly onto the next thing that no one is really taking ownership of what it is that they've created. Once these things are often dropped into production, that person who threw that together and has moved on, like, who's going to own that? Who's going to update that, should it need updating, should there be security issues in the libraries that it depends on, the model that it's using, the prompts that it's making use of? That can become a really large issue. As seen, we need a method for clear control over what is allowed from these parts of AI programs and AI agents.
What I showed you today is a proof of concept based upon the AI egress working group as part of the Kubernetes project. A lot of different companies at the moment, ourselves included, are working together to come up with a standardization for having rules and an agreed upon approach in order to observe, control, and determine the behavior of AI-based applications within Kubernetes clusters. However, what I showed you today is also possible outside of a Kubernetes cluster as well. We're effectively redirecting traffic and parsing the API calls and things like that as well.
Resources #
If you want to learn more, ebpf.io is a fantastic resource. If you want to learn eBPF code, it's all there. On the Cilium lab section, so we have a number of free labs, you can go there. I think there's 45 different labs, but there is one about eBPF. If you want to learn about managing eBPF programs, we have free lab environments that you can basically just hit with a web browser, play with, and learn more. If you're really into eBPF, there is a documentary that you can watch. I think it's about an hour long. It details how eBPF came into creation. eBPF is pretty much everywhere at this point. It's on all Android phones. Facebook use it to manage the networking for all of their Meta fleet and things like that. It's in most Kubernetes clusters, either doing observability, networking, or enforcement of behaviors and things like that as well. There is also Liz Rice's book, "What Is eBPF?" available to download from the website as well.
Questions and Answers #
Participant 1: Can you also show the eBPF code?
Dan Finneran: Yes. This is the code that powers everything that I talked about today. Now, I can step you through this if you really want to. Effectively, this bit of code here, if you look at the area where it says SEC, that's the section header, and connect4, that is a TCP IPv4 connection event, which is what happens when you connect out to something. This is where our code will actually run before the kernel will do anything. eBPF code always happens before the kernel, or whatever it is will actually process it. It allows us the capability to change things and enforce behaviors before they can actually impact a running system. Typically, what this will do is, if we care about it, we will lower down. We will change the destination to where it's actually going, so we hijack the traffic in the eBPF, sending it to our proxy. This is all open source. There's a number of examples of doing this sort of thing, but yes, this is a quick example of eBPF code hooking into IPv4 TCP connect events.
Bryant: Is there any advice on how to document some of this stuff? I'm an old-school Java dev, and I got into aspect-oriented programming, similar vibes but at the JVM level, and it became a nightmare to debug some stuff, because I had these aspects cutting in, and I was almost injecting functionality arbitrarily. How do I debug it? How do I understand what's going on? I'm conscious that it's very powerful, but is it a bit of a foot gun as well?
Dan Finneran: We did this example on the webpage. When I first joined Isovalent, the first example I had was, look at this, connect this to running on your laptop, and it will print out all these events, and it will return ok. Now try saying return not ok, which all of a sudden meant all packets were just dropped. I basically denial of service my laptop from the example. It is incredibly easy to do that. There's no debugger. All you've got are effectively bpf_printk events, so that is a kernel logging mechanism. Effectively, you can write logs to the kernel, and the kernel will output those in sys tracing debug, trace pipe, or whatever it is. That's the only way you can understand what's actually happening in there. This is the thing. Not everybody needs to write eBPF code. The idea really being is that you write it once, expose the bits that you need to do. As I say, for all the networking side of stuff that you see in production, like from Cilium and Calico and things like that, you don't need to know eBPF. You just know that it's powering your network. It's looking at traffic, and it's redirecting it, and doing things like that.
Bryant: The producer and the consumer are quite different people in some regards there. You produce it, you know all the stuff, but I'm a consumer at the other end.
Dan Finneran: Yes. If you really want to learn eBPF, then brilliant. I think I've done my job. It's a bit like driving a car. You don't really need to know how the engine works, especially not with today's modern electric cars, because that is bonkers. It's the same with this. Use the things powered by eBPF. If you want to learn about eBPF, that's great, but you don't need to know the fundamentals fully about what it's doing.
Participant 2: You mentioned restricting syscalls towards the end. I'm wondering, at what point does that just become an arms race, and you just have to sandbox it, since I can imagine it could start cloning itself, forking.
Dan Finneran: The hooks that we apply are effectively events. Our eBPF code, we would have, for instance, file open. We would create our eBPF code, which attaches to the syscall file open, or open. Our code would execute first. If an AI agent tries to open /etc/passwd, our eBPF program will be the first thing that actually happens before the kernel syscall actually occurs. We can always be one step ahead of anything that's actually happening within the system itself. We see that syscall and we say, no, you're not allowed to do it, we return false, effectively that's a stop dead, there's nothing it can really do. It can keep trying that, or try a different syscall, but whatever it is, that is really the only mechanism for opening files through syscalls in the Linux kernel. It's always going to trigger our event. There is syscall ret, so we can actually attach to the return of the syscall. We can allow the kernel to run that syscall, and then we can capture the results of all of that, and have eBPF code that does something based upon the return of that as well. We can attach to the before and after the syscalls.
See more presentations with transcripts