{"slug": "i-built-chunks-an-on-device-ai-that-sees-my-mac-screen-and-acts-without-stealing", "title": "I Built Chunks: an On-Device AI That Sees My Mac Screen and Acts Without Stealing My Mouse", "summary": "A developer built Chunks, a native macOS menu-bar and notch companion that uses the accessibility tree (AXUIElement) to see the screen and act on apps without moving the user's cursor or sending screenshots to a cloud server. The tool runs Gemma 4 2B and 4B quantized models locally through Ollama for fully offline screen understanding, completing a vision turn in about 2.3 seconds, with consent and policy gates plus a SecretSweeper that scrubs credentials before anything is written. Built for the Hacktoberfest Hack Day × AI in Vizag Gemma track, Chunks did not place but is backed by 710+ unit tests and is open-sourced on GitHub.", "body_md": "*This is a submission for the [MLH x DEV Writing Challenge](https://dev.to/mlh-hackathon)*\n\n**Chunks** is a native macOS menu-bar and notch companion. You talk to it, it looks at your screen, explains what's going on, and can act on your apps for you. And it does all of that without grabbing your cursor or sending your screen to a cloud server.\n\nI built it because the \"computer use\" agents I tried annoyed me. They warp your physical mouse, so you can't touch your laptop while they work. They take 30 to 60 seconds per step. They ship full-screen screenshots off to someone else's backend. Meanwhile, normal AI chat sits in a browser tab and knows nothing about the app you're stuck in.\n\nI wanted something that felt like a friend sitting next to me: it sees what I see, points at the right button, and does the boring clicks only when I say yes.\n\n`AXUIElement`) through a driver I call `AXTDriver`. It finds controls by meaning (` app=\"Notes\" role=button name=\"New Note\"`), climbs the accessibility tree to find something pressable, checks focus before typing, and fires `AXUIElementPerformAction`. My real mouse never moves. A smooth Bezier \"pointer trail\" shows you where it's acting instead.`\"store\": false`. A fully offline path runs `gemma4:e2b-it-qat` and `gemma4:e4b-it-qat`) locally through Ollama on Apple Silicon. A full vision turn finishes in about 2.3 seconds with nothing leaving the laptop.`CoordinateSpace` layer maps that to device pixels, window coordinates and \"Circle-to-Explain\" crops.`SecretSweeper` scrubs API keys, JWTs and passwords before anything is written. A `ConsentGate` asks before any state-changing action, and a `PolicyGate` blocks risky commands. One rule I'm proud of: something Chunks **Stack:** Swift 6, AppKit, SwiftUI, ScreenCaptureKit, ApplicationServices (AXUIElement), AVFoundation, Speech.framework, Ollama, Gemini API.\n\nIt's backed by a dependency-free suite of 710+ unit tests and live OS probes (`--probe-local`, `--probe-ax`, `--probe-semantic`).\n\n60-second demo: [https://drive.google.com/file/d/1gIBZRcIgqB2CSNadLQVepUED_W06cSpv/view?usp=sharing](https://drive.google.com/file/d/1gIBZRcIgqB2CSNadLQVepUED_W06cSpv/view?usp=sharing)\n\nCode: [https://github.com/Rishideep-Mandavilli/chunks](https://github.com/Rishideep-Mandavilli/chunks)\n\nI entered the **Gemma track**, and Gemma is what makes the \"private\" part real. Running Gemma 4 2B and 4B quantized models through Ollama meant I could do screen understanding fully offline on a MacBook. A few things I learned:\n\n`/api/chat` as NDJSON with `think: false` made the difference between \"usable\" and \"waiting\".\nGemini Flash stays as an optional fast cloud mode, but Chunks works end to end without it.\n\nI built Chunks for **Hacktoberfest Hack Day × AI in Vizag** on November 1, 2026, in the Gemma use-case track. Honestly, I didn't present it as well as I wanted, and Chunks didn't place. But I did win the special prize for my LinkedIn post about the day, which was a nice surprise.\n\nWhat I'll remember most is the energy of a room full of people building with open models on a Sunday, and the moment Chunks typed into Notes while my hands were off the trackpad. Next time I'll practise the demo as hard as I practised the code.\n\nWhat's next: a signed release so anyone can try it, and better small-model accuracy on crowded screens.\n\nThanks for reading! If you've fought with computer-use agents too, I'd love to hear what broke for you.", "url": "https://wpnews.pro/news/i-built-chunks-an-on-device-ai-that-sees-my-mac-screen-and-acts-without-stealing", "canonical_source": "https://dev.to/rishideep_mandavilli_a160/i-built-chunks-an-on-device-ai-that-sees-my-mac-screen-and-acts-without-stealing-my-mouse-2eoh", "published_at": "2026-11-03 18:46:13+00:00", "updated_at": "2026-11-03 18:53:51.061913+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "ai-safety", "developer-tools"], "entities": ["Chunks", "Ollama", "Gemma", "Gemini", "Apple", "Rishideep Mandavilli", "GitHub", "Hacktoberfest Hack Day × AI in Vizag"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-built-chunks-an-on-device-ai-that-sees-my-mac-screen-and-acts-without-stealing", "markdown": "https://wpnews.pro/news/i-built-chunks-an-on-device-ai-that-sees-my-mac-screen-and-acts-without-stealing.md", "text": "https://wpnews.pro/news/i-built-chunks-an-on-device-ai-that-sees-my-mac-screen-and-acts-without-stealing.txt", "jsonld": "https://wpnews.pro/news/i-built-chunks-an-on-device-ai-that-sees-my-mac-screen-and-acts-without-stealing.jsonld"}}