{"slug": "presentation-running-ai-at-the-edge-running-real-workloads-directly-in-the", "title": "Presentation: Running AI at the Edge: Running Real Workloads Directly in the Browser", "summary": "James Hall, founder of a 15-year-old tech consultancy, presented a talk on running AI workloads directly in the browser, sharing his experience from projects like porting an SPC700 emulator to JavaScript and embedding an LLM into a web browser before ChatGPT's launch. He highlighted the practical benefits and pitfalls of local AI execution, drawing on his work with AI-based compliance platforms and textile recycling.", "body_md": "## Transcript\n\n**James Hall:** I'm going to give you a little bit of an intro to myself, some background, just to give you some context of the kinds of things and stuff that I've been working on. I'm going to talk about, why bother? What's the point of running stuff locally? I'm going to intersperse all this with a bunch of demos, so you can actually see and feel how some of this is looking at the moment. I'm going to try and explain at a high level how some of this stuff works, and hopefully give you the pointers that you need to know when you should be using it. I'm also going to give you plenty of pitfalls and takeaways. We've got lots of battle scars. We've made lots of mistakes, so hopefully you can learn from mine.\n\n## Background\n\nJust a little bit of background. I started a tech consultancy 15 years ago. We basically build digital products for our customers. We've been given all sorts of weird and wonderful briefs, from street lighting to remote unlocking cars to AI dashboards. There's a big wealth of stuff in there. My background, I've been doing weird and wonderful things with JavaScript for a long while. Just to give you a sense of how much I enjoy bending browsers out of shape. In 2012, I ported a SPC700 emulator. The sound chip inside the SNES, you can actually emulate that. This was very early Emscripten days, so you could actually port C++ to JavaScript. Playing around with lots of Uint8s and massive arrays of numbers. I actually made this work with the then very new and shiny Web Audio API, which, yes, I surprised myself, really. Most people would look at this kind of project and go, what's the point?\n\nIt's really good to find the edges of things, like these kinds of experimentation projects. In 2009, so going even further back, I wrote a library called jsPDF, which is haunting me to this day, and has actually had a massive spike in downloads recently. What you can only assume is people vibe coding invoice generation software, or God knows what. Yes, hundreds of emails a day, lots of open PR requests. I was automatically closing PRs way before people got grumpy with AI. This has led to lots of fun, interesting projects and conversations. I'm glad I wrote it, but I hate PDFs. I'm also an external advisor to an AI recycling project.\n\nThe other advisor is the head of the UK Fashion Textile Association. We're trying to figure out how we can reduce waste from textiles using automatic sorting. We've been very keen on AI and the application of it for a long while. Right from the street lighting projects, where we were doing automatic detection of faults and predictive maintenance. We really like it when we can actually figure out how the math and the actual application maps to the real world. We've been running events to get people out, get people talking about different applications.\n\nMy first exposure to LLMs was actually working for a startup. We made a web browser for them in Kotlin and some React in there. We actually embedded an LLM into the web browser to do automatic tagging and categorization. This was before ChatGPT came out. We had to go to OpenAI and say, this is the app we're building. This is what it does. These are the risk areas that we see. They were very careful about approving each and every use case. Obviously, as soon as ChatGPT was launched, floodgates opened, and they didn't really care anymore, probably because money. It was a very interesting process. We learned a lot doing that. We also launched an LLM-based compliance platform. This won Tech Innovation of the Year. We learned a lot about chatbots. I also know now a lot about why you wouldn't want to build them. Yes, lots of battle scars from this.\n\nWe ended up building lots of systems that are now commonplace. We ended up writing our own LLM evaluation and observability platform as part of the product. We also discovered that users interacting with chatbots, because they're so open-ended, you need to draw them into a railroaded experience. We came up with this idea of checklisting. We had our own version of structured outputs, which was very rudimentary to try and put people down a certain path. We learned a lot doing this. We got picked up by a large iGaming company in the UK. We were working with the Timeform team on blending ML models with LLMs. We effectively took over a proof of concept that Amazon Web Services had done. There, they were not able to confidently predict the order of the horses to any degree. I think their F-score was about 0.5, 0.6, a little better than random.\n\nWe were able to get that to about 0.9 in a couple of weeks by blending lots of different techniques together. We learned a lot through that project. A lot of where this talk's content is coming from is some of the thinking with some of the new organizations we're working with. A lot of it is industrial IoT. Lots of data. How do we make sense of it? How do we plot it? How do we measure what the trends are? That gives you a bit of background on where all this framing is from and where my brain is sort of at.\n\n## Local vs. Cloud\n\nI'd like to talk to you a little bit about the differences between local and cloud. I think a lot of people today will default to cloud because it's the easiest option. It has the strongest models and it is the path of least resistance. A lot of people are picking up these providers, signing their agreements, and sending all of their data to these companies, including their customers' data. The problem with this kind of setup is that end users are increasingly opening up more and more of their personal information without any real thought about the consequences of that. I'm a little bit worried about logging platforms within these organizations. What are the checks and balances for employees looking at data? Can we really trust companies to do what they say they're going to do if they're compelled by the U.S. government to open up logs, for example?\n\nDo you have any control over that? This is where I'm starting to get a little bit worried. This is where these downsides of server-side inference come in. It's not just a privacy concern. There's also low and no internet-connected states. You'd be surprised how, in many countries, the Wi-Fi is very patchy. If you're going on the train, if you're just out of reach of signal. There's also the network latency, which is quite jarring for some applications, especially around real-time video and audio processing. If you're cutting out background noise, for example, you wouldn't want to go from the UK to a data center in Ireland and back and then out to your end user. The cost scales with users, so you become a victim of your own success. Let's say you build something super popular and it relies on the latest and greatest OpenAI model. You get a million users, you've now got a million people to pay for their inference.\n\nI'm going to show you a quick demo of near human-level transcription quality available in JavaScript. The Whisper model was originally produced by the OpenAI team, and they open-sourced the weights. People have been improving and riffing on these kinds of models over the past four years or so. They're getting seriously good. They are able to have correct punctuation. They're able to know place names. They can do complex terminology within particular industries. All this can happen on-device within a web browser as well. We've been seeing examples of this. One of mine and Ian's good friends, actually, Nick Payne, has been building an application, which I'll demo, which uses local transcription models to make it a totally privacy-aware meeting notetaker. One of the demos that I've built to show off some of the capabilities of local models, this was a few years ago, before Claude would be able to one-shot this.\n\nThis actually takes the local model, and it's able to look at the log probes. You can do a piece of mathematics on that to get the probability, so you can actually see the likelihood of each token that's happening. A lot of commercial model providers will actually hide this information in order to stop people from reverse-engineering their frontier model. I think some of the APIs now do expose this. One of the really good advantages of using a local model is that you get access to all this raw data. You can actually see if all of the log probes are pretty flat, there's no standout, you know that there's a risk of hallucination there, because it's literally not able to pick. This is talat. Me and Nick sat, and we reverse-engineered an app called Granola, which is a startup in London. They were using a really interesting Mac API, which allows you to tap into the audio interface.\n\nThey also have a nice neat trick where you can figure out when meetings are starting and ending. Nick took some of that learning and made it into an open-source Taps library, which you can easily hook into the audio interface. This uses a real nice blend of local models. It's an Electron app. All the UI is HTML, JavaScript in React. Then it actually hooks into these local models, which can run natively on the Apple Neural Engine. You've got NVIDIA's Parakeet. You've got a streaming model. You've also got an Alibaba model for summarization. Then we've got voice fingerprinting and identification models as well. These are really good and can be quantized without losing lots of impact, and got to a reasonable size to download.\n\nWhen it comes to local models, there isn't just one option anymore. The only option until a couple of years ago was to bring your own API, so take all the weights, compact them as much as you can. Quantize them. Turn them from 8-bit integers into 4-bit or 2-bit. Send them over the wire and cache them in the browser or in your Electron app or in your desktop application. That used to be the only way. That whole ecosystem has come on a very long way. There are still some problems. One of the problems that the Chrome team are trying to come across and sort out at the moment is the same origin caching limitations. Once we've got the standards ratified for this and ticked off, you can see websites or web applications using one model in one place and you'll be able to reuse it in other places.\n\nThat is the bring your own AI route. For that route, you use WebLLM, which is built on WebGPU. There's also Transformers.js from Hugging Face. They use this ONNX Runtime. Then you have other providers of local inference, like TensorFlow. The built-in AI is pretty new. It was pretty shit until very recently. I was on the early private beta list for this in Chrome when they had a very early version of Gemini Nano. I tried to build a browser extension that would just extract the recipe from a recipe page and get rid of the whole story and everything. You really had to fight it, like the prompting, and you had to really try to get it to do the right thing. It's since come on in leaps and bounds. Those Nano Gemini models are now getting much better. They've also started shipping a bunch of other inbuilt APIs that make a lot of sense.\n\nNot only do you have your prompt API, you've now got a translator, you've got a summarizer, and a language detector. They're trying to get everybody on board. Chrome and Firefox and other organization members are all aligning on the same kinds of APIs. It's super interesting. It seems a shame to not explore these routes when the hardware in your pocket is getting increasingly more advanced. You're having much higher-powered neural processing units. You've got specialized GPUs that are just there for AI now running on your device. It does seem a real shame to just ship everything off to Sam Altman when you could actually weigh up the pros and cons of doing that and find some other route through.\n\nI love Transformers.js. I think this is absolutely magic. There are loads of really cool models here, and there's loads of ways that you can interact with them. It's JavaScript native. It will run just as plain JavaScript on the CPU, but then it also is able to optimize itself. Back when I did like that js-snes-player, there was no optimizations whatsoever. You were extremely limited in the amount of computation you could do within each cycle. As Wasm and other technologies have started to pick up, that really sped up how quickly you could run on the CPU. Now, there's direct access to the GPU through WebGPU, and you can run at near-native speeds now for inference, which is super impressive. Obviously, everything just stays on the device. There are no API calls, no third-party processing. There's new backends now coming out. WebNN is still very early days, but it will be able to run on the very specialized GPUs on your Android and iOS phones.\n\nWhen you're looking at a machine learning model to run on-device, it's very useful to compare the sizes. Some smaller models will be extremely good at writing like a SQL query or something super basic like that, or summarization. As you get bigger and bigger, they're better at more and more complex tasks. In the trends that we're seeing, people are coming up with more and more clever ways to pack these models down, like using variable width quantizing for different areas, for different layers of the model. This is super useful. There's a common phrase in machine learning, but I'd like to remind you of that, which is that all models are wrong, but some are useful. I think this is a really important thing to think about when you're developing applications with AI, is that we should not be trusting the output just on face value. We should be testing and figuring out what are the acceptable bounds in which we find this model useful, and is it safe for our users to also use them? Setting the goalposts up and setting up your evaluation suite at the start, that's where most of the work is. Actually, integrating the model is the easy bit. The difficulty comes with everything that surrounds it, so testing, validation, security.\n\nThere's a bunch of stuff that you can do to optimize in-browser inference. I wouldn't recommend quantizing your own models, unless you really know what you're doing, but you are able to change the precision. For a lot of applications, it doesn't actually lobotomize it as much as you might expect. You could take a 7-gigabyte model down to 2 gigabytes, and you'd only have a modest quality loss. As I mentioned, there's the WebGPU for hardware acceleration, and Wasm for universal CPU fallback, which is super handy. Then the WebNN stuff is coming in the next year or two. You can also fuse kernels together, and you can have custom operators. You can have pre-compiled routes. There are loads of model-specific graph optimizations you can do, and caching. WebLLM is super good at exposing some of these features. You can actually pick the model that you want to use.\n\nYou can actually test out whether it's going to work for your use case. What I'm hoping is that, as people start to use these kinds of things more and more, and the cross-origin caching comes into play, that, actually, you'll find that you might end up with a desktop machine that ships with a Llama 3-point whatever, and you can rely on it being there. At the moment, we're not quite there, but you need to skate to where the puck's going. This is what I'm super excited about. This is the project that gives us that raw access to the NPU acceleration. This gets us to this super-fast speed, even on mobile devices. This is the tipping point that we're looking for when we're going to build these privacy-conscious applications on top of web technologies. One of my customers is working in a highly regulated space. They don't want industrial IoT information going through commercial frontier models.\n\nThere is actually a good cause and lots of interesting experimentation to actually run all this locally. There's a really good, super-fast DB called DuckDB. They have a Wasm build, which is surprisingly quick. A few of our customers have been surprised by how quickly taking large datasets from Redshift, where the analytical workload you'd expect to run fairly quickly, actually seeing better results with a very optimized DuckDB on a local device, which is super surprising for them. If you encode it in Parquet, you can have it run locally in the browser. What this is really good for is trying lots of permutations of queries to find insight within data. You can use an LLM as a mini data scientist, trying lots of different angles, seeing the outputs and iterating very quickly. We've been doing this with virtualized CLI, which I'll talk about.\n\nAnother extremely good use case for local AI is using really small models, like this token classification model. There is a named entity recognition model, which is super small, and it can run on plain text. It can pull out, not using regular expressions or anything, but it can pull out like, this looks like a person's name, this looks like a location, this looks like an address. Once you've pulled all those pieces out, you can swap it out so you're not sending it back to the server, things like that. Or you could even show a warning, saying it looks like you're inputting patient data. When we're doing the compliance applications, because it's highly sensitive areas of work that people are working in, even if you instruct people not to upload sensitive information or documentation, often you'll find users, the first thing they do is copy and paste a load of patient data or some clinical study material into your UI.\n\nThis is when something like this would be super handy, because you can just alert them straight away, saying this data can't be processed for these reasons. We've actually got really good browser support now, which we didn't do a couple of years ago. Finally, Safari has come out the gate with extremely good support. Firefox was second, and Chrome came out with the first. It's getting really good now. Basically, every other browser is secretly Chrome, so that's nice and wide support. These models don't just have to process text. There can be lots of different applications that they can do. This is a super fun example. I think this is a Meta project, actually. It came out of one of Meta Labs projects. You can actually generate music based on just text alone and emotions. You can say, yes, generate me an electro pop tune, and it will churn out a nice little piece of music. It's getting surprisingly good.\n\nAs I've been mentioning, there are these really big benefits of the in-browser inference, but it's not without its pitfalls. You want to measure which workloads make the most sense. For certain workloads, shifting that inference cost from your service to the user's device makes lots of sense, especially video-heavy applications where you might be using WebRTC, which actually is peer-to-peer. You don't want to be bouncing people through a server. There are some really nice examples. You've probably used Google Meet. They actually use a very simple model just to do background removal in the client side, which is super nice. You can actually just figure out for which workloads are you going to do these hybrid approaches? Which ones are you going to do completely local? Which ones are you going to do server-only? There is a bit of a catch at the moment, which is that these larger models are more complex reasoning. The better fit is still the server end. It is really making sure that your workload matches the kinds of models you want to run.\n\n## What Should You Measure?\n\nAs I mentioned, most of the work in deploying a model is measurement and making sure it meets your criteria. What are the things that you need to measure to figure out if you want to launch in-browser or the cloud? You want to look at time to first token. This is for an LLM. How long does it take to spin up the model, warm it up, and generate the first token? Surprisingly for really small models it's super quick locally, and actually that round trip to the server is quite a big chunk of that. Especially if you're sending up like tens, or hundreds of thousands of tokens up, you want to think about the overall latency and download times. If this is a meeting note transcription app that someone's going to use every day, are they happy downloading 500 megabytes? Probably. It's about doing those calculations and making sure that the fit is right.\n\nYou want to be looking at your token throughput. How fast is it actually to output and do completions? Then, the hardest ones are accuracy. You can also use LLM-as-a-judge. I would always recommend that if you're able to do something using dumb code then do that. Not everything has to be AI. I've often seen a lot of AI orchestration or evaluation suites lean towards using yet more AI to test the AI, and it's not always necessary.\n\n## Privacy by Architecture, Not by Policy\n\nOne of the great things you're doing for your users by designing products with AI at the edge is that you're actually enforcing privacy by your architectural decision rather than policy. As I mentioned before, you don't know what's going to happen with, let's say there's a big cloud provider like AWS. Could they be compelled to start tracking information, start surfacing different log files and things like that? You can actually circumvent all of this by making it impossible for that to happen. Why don't you keep the things that you really need to be local, local. I'm just going to talk about the meeting note example again. A lot of meeting note taking apps like Granola, like a few others, they basically send raw audio data from your device straight up to the cloud. Some of these providers have actually had breaches where API keys have been accidentally published.\n\nJust the fact that they're logging and processing this even just for a moment, it's almost like the toxic waste of user data processing. It's like you don't want to be handling this super sensitive information. Similar problems exist with the Meta Ray-Ban glasses. You're recording all the time. It's going to some data center, or the humans are potentially reviewing it to figure out problems, tagging, doing all that stuff. How happy are you that that's happening? How happy are your users that that's happening? When it comes to adding AI to your product, what I would wholeheartedly recommend is to take a little step back and please try not to just add a chatbot to your product. The reason I say that is not because I don't think natural language processing style features aren't cool. I do think they are cool. I think there's many reasons for this.\n\nFirstly, it's quite difficult for users who are not as patient as your internal developers or stakeholders. They are not going to spend five minutes chatting away trying to figure out what this bloody other chatbot is actually good at doing within your product. They've had poor experience in the past, so they might have gone to their courier, DHL, something like that. They've used an online support chatbot and had mixed results. People are getting this chatbot fatigue. I'm not saying don't add some natural language processing features to your product. Absolutely do if it makes sense. Your first go-to shouldn't be, let's bolt on the chatbot.\n\n## Common Pitfalls\n\nYou have responsibility and you have the technical power to figure out like, what is an LLM? What is a model? What is this software that we're building actually good at? Make that choice for your users. Make it super easy. If you're building an AI dashboarding product, the first thing you present them is not ask it anything. That's too open-ended. Users aren't imaginative enough, but probably the power users are. Ninety percent of people just want to get in, get the job done and get on with it. Why don't you find out what it's good at? Why don't you present the suggestions? Why don't you already in the background have been processing these interesting insights and showing examples of what you could produce? Then if you have to, you can then add natural language tweaks and suggestions, \"Let me change this chart. Let me do this. Let me do the other.\" That's for the power users.\n\nFine. Leave it in there. Don't make it like front and center. If it doesn't need to be AI, don't make it AI. I know it sounds stupid, but we end up working with so many customers who at the top of their agenda is we need AI, and actually they don't. This is what's wrong with a well-crafted SQL query or a regex. Just reach for a model when the problem is generally difficult and fuzzy, and not just because it's available. Another pitfall with local first is that that first download is quite painful. You want to have aggressive caching. You want to show progress. Ideally, fold it into the flow so they don't even notice. Instagram, when it came out, everybody was super impressed with it because they couldn't figure out how it was so fast. You would take a picture and you would be choosing your filter and you press go and it was basically there.\n\nThis was back when we had really shit mobile networks. The user experience was super nice. It was really a simple trick. All they did was, as soon as you took the photo and went to the next step, it started uploading the photo, and in the background as you're choosing the filter and press send, all it sent was the text name of the filter to the backend and did the processing remotely. It's a really nice way, if you can hide loading time in some other process, and that perceived loading time is all that matters. Users only care about how they feel about the weight, not the actual weight itself.\n\nA common joke in data science is, you don't need AI, you just need a good SQL query. I'd like to flip that on its head a little bit and a really nice use case for AI is actually to write SQL queries. As I've showed in the demo earlier with the example of DuckDB, you can actually run this full analytical engine in a browser. You can find some interesting angles. We've been doing this by running a fake CLI in-browser and allowing the LLM to write command line calls and arguments and pass SQL and do things like jq selectors to find interesting angles and insights in data using the fewest tokens possible. There's also models that are smaller, faster, and specialized for SQL writing. A quick way to figure out if you should be reaching for in-browser inference is, is this data sensitive or not? Could it potentially include patient data?\n\nCould it potentially include PII? If the answer is yes, there's a really good case for using local inference. Even if the data does end up in the cloud at some point, you could at least redact the PII if you're not expecting to store PII. Again, it's like toxic waste. You don't want it in your log files. You don't want it in your S3 buckets. Unless your job is to process that PII, you'd rather not have it anywhere near your servers. Just don't make it an option. If the data is not sensitive but you have high quality or complexity requirements, then that is probably a good use case to use a cloud API. If you can find a good enough model to do it locally, then do it locally. As always, if it doesn't need to be AI, don't use AI.\n\n## Models in Action\n\nI'm going to show you some models that I think are really cool. We've got an audio multimodal demo here, which I've taken an image. The example image is of some bread, but this is quite a cool little demo. It takes a 2D pixel map, and it is able to infer the depth of those pixels. I've also taken a test image here, which is a nice little QCon logo. This is quite a cool thing where you're actually doing some transformation into WebGL. This is actually coming on really quickly. You're actually able to do super interesting things like frame-by-frame video tracking. This is extremely processor intensive, so sped it up a little bit. It's making a map of all the things that are moving on the screen, and then you can actually build an interactive UI to then just select the flags, and it'll actually follow it around as the camera pans.\n\nSuper useful for blurring things out, highlighting things. Yes, really interesting stuff. I think a really good use case for a local model is blurring out the background, things like that. Often people will be working from home. They've got the kids running around in the background. They've got a load of mess that they haven't tidied up. You can actually do this super quickly now, like do real-time background removal. Why not do that before you send it over WebRTC locally? Because WebRTC is a peer-to-peer video conferencing solution, there's no reason you should be going up to the cloud and back out again, especially if it's just a personal call between two people.\n\nI'm going to show you a few text-based model examples. This is a demo of Chrome's built-in prompt API, and it's actually pretty decent. It wasn't always, as I mentioned. Basically, in one line of code, you can set up a prompt. You can pass in the text, and it can do a completion entirely in-browser. There's no network calls whatsoever. The model's been trained on QCon's website, clearly, because it knows all the great reasons that you should come to QCon. It is also able to do constrained sampling and structured outputs, so you can actually get JSON and things like that from it. Yes, super handy. List out all the best things about attending QCon as a short JSON block, and there you go. You've not given Sam Altman any money, and you've got a bunch of JSON that you can then render into a nice UI. I'd always recommend, if you can, try not to show users walls of text.\n\nIf you're using an LLM, always try and structure it in some way and make some nice UI elements. It just makes it loads easier for people to understand, and also, they've got that wall of text fatigue. This is a product called Duck-UI. You can try this for free. This is entirely in-browser. I've loaded a CSV file of a few years' worth of QCon talks in here, and you could just ask it questions about the data, and it'll generate the SQL in a WebLLM, so that runs on your GPU. Then the query runs in Wasm and returns the data. None of this data is leaving your web browser. These kinds of databases are good for dataset sizes between 100 and 500 megabytes, something like that, which you can fit a surprisingly high amount of data in.\n\n## Measuring What Good Looks Like\n\nOne of the important things with rolling a model out is trying to measure what good looks like. You can come up with and whiteboard as many metrics as you like, but a lot of stakeholders will focus on, does it actually improve a metric that they care about? It might be like time to decision, or it could be the quality of a decision, or how closely it matches what a human would do. Whenever we're doing a project, we'll set out that criteria at the start, and we'd like to save as much of it in a database as we can, so that we can create this eval suite around the model. How do you measure the quality of an LLM output? In regular software that's not using LLM, it's deterministic, you just wrap automated tests around things. With LLMs, much trickier is the output. It's much longer, it can vary, and it gets quite tricky.\n\nInstead of manually reviewing all the reasoning output from an LLM, that's supposed to be a robot version of Judge Rinder, you can actually get another model that is better to actually rank the thinking, and rank whether you think that was a good output or not. You want the judge model to be better than the model that you're judging. You would use a frontier model to evaluate and test the weaker in-browser model. I would wholeheartedly recommend for any AI project, cloud or desktop, to build some kind of eval suite that is very visual and very easy for subject matter experts to understand. If you're doing like banking KYC, or you're doing healthcare records, whatever it is, you need to build a UI that that healthcare person can understand, and they can see what the AI is doing. You don't want this to be an engineering thing.\n\nThis is 80%, 90% of your product. If you're building a very AI heavy product, most of the product is going to be in how it is tested, and how that testing is communicated to people that know much better than you do about whether that's a good quality output or not. For example, let's say you're building a tool that can figure out text on a passport. Maybe it's quite low risk. Maybe this is for checking in to a flight or boarding. You would store the test fixture. This is the test file, the scan. Ideally, this is like the awkward passport scan that's not working so well. Then you would have a JSON blob, which is, this is what we expect, and then you would compare that to the actual. The user interface that you want to produce that's wrapped around this model is something like this. You have, here's your fixture, here's what we expected, and here's actually what happened. The reason you want to make it super visible like this is because you're going to have people pull various levers to improve the quality here.\n\nWhat levers can you pull? Super simple one, change underlying prompts. Obvious for an LLM. You'd be surprised how many product teams we've got into who have no way of knowing if changing a prompt makes their product better or worse. They would just ship it based on whether it felt a bit better, which is super dangerous because it's absolutely random. It's super important that when somebody changes a prompt, you're able to rerun all of your edge case tests and figure out, have I made it better or worse? It sounds obvious, but a lot of people miss this. You can obviously change the underlying models used, so let's beef it up to a bigger model. Let's see what happens. You can also change the harness in which that model is set up, so the workflow. I don't love the term agentic, but like an agentic flow where an AI is actually able to go through multiple steps in order to produce a result.\n\nI think it needs to be as guardrailed as possible. There's an amazing project called just-bash, which is by the Vercel Labs team. They've actually engineered an entirely in-browser Bash environment, which is completely written in TypeScript. You can write new commands in TypeScript, which will run whatever code you want. Then an LLM can chain those commands together using very short, succinct CLI-like syntax, which is super good, because with MCP, you're doing lots of big JSON blobs, very hard to chain together, and you're wasting loads of tokens. With these virtualized Bash environments, you can have an LLM iterate and try things out in order to get to a result, much like Claude Code does. People try all sorts of fancy stuff like indexing entire codebases in vector DBs and doing all this fancy shit, but when it comes down to it, LLMs are actually just really good at just using cat and grep and head and stuff.\n\nWhy not just let it use the really simple, dumb tools that it's really good at doing, but in a completely sandboxed way? The LLM has got no idea that this isn't a real Bash. It doesn't have operating system privileges. It can't access the network, for example. The LLM doesn't care. Instead of doing something super dangerous and connecting the LLM to your real CLI, why not connect it to a fake CLI? Once you've pulled those levers, you can rerun your lovely, nice and visible evals, and your subject matter experts can tell you whether they think it's suitable to ship or not. In my opinion, I think all of the roads lead to madness, which is shipping based on vibes alone.\n\n## Tactics and Key Insights\n\nI'm going to end on a couple of tactics. Although a lot of what I've been talking about is about picking models and a lot more of the technical side, a lot of what actually improves AI apps is super boring. It's like talking to end users, talking to subject matter experts. It's making sure that the platform is reliable, so it retries things when it fails. It's preparing and describing data properly. It's optimizing that end-to-end workflow with feedback, so that eval suite, so you can visually see and test things. Yes, just lots of little tweaks, reviewing and evaluating over and over. What I'd like to go away with today is, if the data is sensitive, go local. If it's somebody's audio or video in their home, then try and apply some pre-processing to make sure you're not uploading content that you don't want to be uploading. Do you want to remove PII before it goes up? Just do that. Make sure you benchmark it on your actual workloads with the actual edge cases that you're going to be seeing in the wild. Have fun and build some cool stuff. Transformers.js is a really good place to start. Just Bash is super fun. We'd recommend giving it a whirl.\n\n**See more presentations with transcripts**", "url": "https://wpnews.pro/news/presentation-running-ai-at-the-edge-running-real-workloads-directly-in-the", "canonical_source": "https://www.infoq.com/presentations/local-ai-browser-inference-privacy/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global", "published_at": "2026-08-31 11:00:00+00:00", "updated_at": "2026-08-31 11:22:04.278110+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-tools"], "entities": ["James Hall", "OpenAI", "ChatGPT", "jsPDF", "SPC700", "Web Audio API", "UK Fashion Textile Association"], "alternates": {"html": "https://wpnews.pro/news/presentation-running-ai-at-the-edge-running-real-workloads-directly-in-the", "markdown": "https://wpnews.pro/news/presentation-running-ai-at-the-edge-running-real-workloads-directly-in-the.md", "text": "https://wpnews.pro/news/presentation-running-ai-at-the-edge-running-real-workloads-directly-in-the.txt", "jsonld": "https://wpnews.pro/news/presentation-running-ai-at-the-edge-running-real-workloads-directly-in-the.jsonld"}}