I Built Chunks: an On-Device AI That Sees My Mac Screen and Acts Without Stealing My Mouse

A developer built Chunks, a native macOS menu-bar and notch companion that uses the accessibility tree (AXUIElement) to see the screen and act on apps without moving the user's cursor or sending screenshots to a cloud server. The tool runs Gemma 4 2B and 4B quantized models locally through Ollama for fully offline screen understanding, completing a vision turn in about 2.3 seconds, with consent and policy gates plus a SecretSweeper that scrubs credentials before anything is written. Built for the Hacktoberfest Hack Day × AI in Vizag Gemma track, Chunks did not place but is backed by 710+ unit tests and is open-sourced on GitHub.

This is a submission for the MLH x DEV Writing Challenge https://dev.to/mlh-hackathon Chunks is a native macOS menu-bar and notch companion. You talk to it, it looks at your screen, explains what's going on, and can act on your apps for you. And it does all of that without grabbing your cursor or sending your screen to a cloud server. I built it because the "computer use" agents I tried annoyed me. They warp your physical mouse, so you can't touch your laptop while they work. They take 30 to 60 seconds per step. They ship full-screen screenshots off to someone else's backend. Meanwhile, normal AI chat sits in a browser tab and knows nothing about the app you're stuck in. I wanted something that felt like a friend sitting next to me: it sees what I see, points at the right button, and does the boring clicks only when I say yes. AXUIElement through a driver I call AXTDriver . It finds controls by meaning app="Notes" role=button name="New Note" , climbs the accessibility tree to find something pressable, checks focus before typing, and fires AXUIElementPerformAction . My real mouse never moves. A smooth Bezier "pointer trail" shows you where it's acting instead. "store": false . A fully offline path runs gemma4:e2b-it-qat and gemma4:e4b-it-qat locally through Ollama on Apple Silicon. A full vision turn finishes in about 2.3 seconds with nothing leaving the laptop. CoordinateSpace layer maps that to device pixels, window coordinates and "Circle-to-Explain" crops. SecretSweeper scrubs API keys, JWTs and passwords before anything is written. A ConsentGate asks before any state-changing action, and a PolicyGate blocks risky commands. One rule I'm proud of: something Chunks Stack: Swift 6, AppKit, SwiftUI, ScreenCaptureKit, ApplicationServices AXUIElement , AVFoundation, Speech.framework, Ollama, Gemini API. It's backed by a dependency-free suite of 710+ unit tests and live OS probes --probe-local , --probe-ax , --probe-semantic . 60-second demo: https://drive.google.com/file/d/1gIBZRcIgqB2CSNadLQVepUED W06cSpv/view?usp=sharing https://drive.google.com/file/d/1gIBZRcIgqB2CSNadLQVepUED W06cSpv/view?usp=sharing Code: https://github.com/Rishideep-Mandavilli/chunks https://github.com/Rishideep-Mandavilli/chunks I entered the Gemma track , and Gemma is what makes the "private" part real. Running Gemma 4 2B and 4B quantized models through Ollama meant I could do screen understanding fully offline on a MacBook. A few things I learned: /api/chat as NDJSON with think: false made the difference between "usable" and "waiting". Gemini Flash stays as an optional fast cloud mode, but Chunks works end to end without it. I built Chunks for Hacktoberfest Hack Day × AI in Vizag on November 1, 2026, in the Gemma use-case track. Honestly, I didn't present it as well as I wanted, and Chunks didn't place. But I did win the special prize for my LinkedIn post about the day, which was a nice surprise. What I'll remember most is the energy of a room full of people building with open models on a Sunday, and the moment Chunks typed into Notes while my hands were off the trackpad. Next time I'll practise the demo as hard as I practised the code. What's next: a signed release so anyone can try it, and better small-model accuracy on crowded screens. Thanks for reading If you've fought with computer-use agents too, I'd love to hear what broke for you.