Every AI tool I used worked the same way. I'd open a tab, explain who the client was and what we'd agreed, paste in the email, get some text back, and then go do the actual work myself. The model was smart. The setup around it left all the carrying to me.
So I built Moe, an assistant that runs on your Mac the whole time you're using it. This post is about the two decisions that shaped it: how you reach it, and how it does things once you've asked.
#
Reaching it from any app
Moe has no window you have to switch to. It sits in a small panel at the notch, over whatever you're doing. There are three ways to start talking:
- Hold the right Option key, talk, let go.- Press Control-Option-Space to start and stop hands free.- Say "Hey Moe". The wake word runs on a small model on your Mac and listens only for its name.After it answers it keeps listening for a few seconds, so a follow-up doesn't need the key or the name again. Talk over it and it stops. If you change one part of what you asked halfway through, it keeps the rest. The hard part was working out which speech was meant for it. Moe looks at how long ago it spoke, whether it just asked you something, how loud you are compared with the room, and whether the voice matches the person who started the conversation. Someone talking beside you doesn't get answered.
Doing the thing #
Once you've asked, Moe does the work on the Mac. It uses the same accessibility APIs a screen reader uses, and clicks go straight to the target window in the background. Your cursor stays where it is and you can keep typing in another app. Clicking is the last thing it tries, though. Asked to join a meeting, it reads the link off the calendar event and opens it. The order is: data it already has, then the app's own scripting, then a menu item or keyboard shortcut, and only then a click. Reading an app's menu bar directly turned out to matter most. It's faster than looking at the screen, and it works in apps that draw everything on a custom canvas, like audio and 3D tools, because their menus are still ordinary menus. It doesn't ask before every click. It asks when something is about to leave or can't be undone: paying, sending a message or an email, deleting, typing a password. It shows you what's going out and to whom, then waits. Touch the trackpad while it's working and it stops.
It remembers you #
Moe has read the mail, messages and calendar you connected, and it keeps a short list of what it has learned about you in plain Markdown on your disk. Every line says where it came from. I'll write up how that works in another post.
What it runs on #
I didn't build the hard parts. Speech recognition is NVIDIA's Parakeet running on the Mac, and the agent runtime is Nous Research's Hermes Agent. The thinking comes from whatever you already use: a Claude or ChatGPT account, an API key, or a model on your own machine. Moe doesn't resell it or mark it up. It's free on your Mac and needs macOS 15 on Apple silicon. It's in beta, so you'll hit rough edges. If something breaks, tell Moe "that didn't work" and it reaches me. moebot.app