My Paranoid Coding Machine Setup

Brendan Long moved all of his Claude Code AI agents into an unprivileged VM running under an unprivileged user account, using rootless Podman instead of Docker because any user who can run Docker can easily gain root access inside the VM. For GPU access, the agents SSH into an unprivileged gVisor plus nvproxy container through Long's GPU queueing tool, which allowlists Nvidia driver calls to reduce the driver-exploit attack surface. Long also had Claude rewrite the C code of his locally hosted Kokoro-82M text-to-speech service in Rust and sandboxed it with GPU access, keeping the only internet-facing port, exposed via Cloudflare Tunnel, free of privileges and GPU access.

The combination of running AI agents which can install and run anything with supply chain attacks / the Hackening https://www.lesswrong.com/posts/iqgZaJMnmyDCYfnuC/the-hackening-1 <sup 6 fn:6 </sup has made me much more paranoid about my personal machine than I used to be. I’ve been slowly improving this, and finally moved all of my AI agents into a VM today. I’m sharing this to show how far you can take this, even if you have requirements like GPU access or the ability to run something close enough to Docker. This is a follow-up / longer version of a recent shortform 7 fn:7 . Goals goals I run Claude Code in a custom harness https://github.com/brendanlong/clawed-abode <sup 8 fn:8 </sup which currently disables all permission prompts. I’m not really that worried about Claude doing something dangerous on purpose, but I worry about: - Claude breaking something by accident mistyping a command that turns into rm -rf / . - Claude installing malware most likely via a supply chain attack . My biggest worry is kernel exploits giving attackers access to my user account, allowing them to install a keylogger and steal my password manager vault<sup 1 fn:1 </sup . Claude needs the ability to run Docker for testing, and shared access to my GPU for ML work. Non-Goals non-goals To get its work done, Claude needs access to many API keys GitHub, OpenRouter, etc. . These keys are already as limited as they can be, and I limit the amount of API credits each key can access anyway because Claude doesn’t understand the value of money. Agent VM agent-vm I moved Claude agents to an unprivileged user account inside of an unprivileged VM. Any user that can run Docker can easily get root access and root access within the VM makes sandbox escape much easier so agents use rootless Podman https://developers.redhat.com/blog/2020/09/25/rootless-containers-with-podman-the-basics <sup 9 fn:9 </sup instead. GPU drivers allow direct access to hardware, and assigning the GPU to a VM would prevent my computer from being used by humans, so the VM can’t access the GPU directly. GPU Container gpu-container For GPU usage, agents use my GPU queueing tool https://www.brendanlong.com/gpuc-a-single-user-gpu-queue-that-doesnt-require-sudo.html <sup 12 fn:12 </sup over SSH into an unprivileged container using gVisor https://gvisor.dev/ <sup 13 fn:13 </sup + nvproxy. This has an allowlist for Nvidia driver calls, which reduces the attack surface https://gvisor.dev/docs/user guide/gpu/ nvproxy-security <sup 14 fn:14 </sup for driver exploits although imperfectly . Production TTS Containers production-tts-containers I run a Text-to-Speech service for my RSS reader https://lionreader.com/demo/all?entry=text-to-speech <sup 15 fn:15 </sup on this machine, since everything commercially available is either weirdly slow<sup 2 fn:2 </sup or ridiculously expensive. I decided to run Kokoro-82M https://huggingface.co/hexgrad/Kokoro-82M <sup 16 fn:16 </sup locally, but didn’t want to expose a service containing C code<sup 3 fn:3 </sup to the internet. I had Claude rewrite the C code in Rust https://github.com/brendanlong/optimized-kokoro <sup 17 fn:17 </sup , and just in case heavily sandbox the code with GPU access: The only port opened to the internet via Cloudflare Tunnel https://developers.cloudflare.com/tunnel/ 18 fn:18 <sup 4 fn:4 </sup is written entirely in Rust and has no privileges or GPU access. It talks to a separate container with GPU access through gVisor + nvproxy, like my other GPU container over a very simple binary protocol where the only inputs are tokenized phonemes a list of numbers . I’m still unsure if this piece is sufficiently paranoid, but I really like having fast TTS and don’t want to spend $200+/mo for a cloud GPU or to send my users’ data to other people’s gaming machines https://vast.ai/ <sup 19 fn:19 </sup . Gotchas gotchas Segfaults in the agent VM segfaults-in-the-agent-vm This may have something to do with faulty hardware on my machine<sup 5 fn:5 </sup , but switching from host-passthrough to host-model in libvirt fixed it for me. File descriptor leak in gVisor https://github.com/google/gvisor/pull/17011 <sup 20 fn:20 </sup with directfs=false file-descriptor-leak-in-gvisor-with-directfsfalse 20 fn:20 I was using this to avoid running the sandbox as root. Moving the GPU container to rootless Podman with directfs worked instead. Other Security Hardening other-security-hardening - Kernel updates are applied without restarting via Ubuntu Pro https://ubuntu.com/pro <sup 21 fn:21 </sup / Livepatch https://ubuntu.com/security/livepatch <sup 22 fn:22 </sup free for up to 5 machines, although the agent VM counts for one . - Secure Boot is enabled, using Canonical’s signed driver modules. - Several unused kernel modules are disabled, especially network protocols that I don’t use sctp , rds , tipc and n hdlc . - I removed my own account from the docker group to limit malware’s impact on my own account, although a keylogger could still get past this. - A firewall blocks all inbound requests except via Tailscale, and my Tailscale policy is fairly locked-down now: My personal devices can access each other via advertised services and key-only SSH; the agent VM can only access the TTS service. - Package managers proxy installs through Socket Firewall https://socket.dev/features/firewall <sup 23 fn:23 </sup free , which warns or blocks known malware in packages. Next Steps next-steps The biggest remaining security improvement for me is probably running every application in a separate VM https://www.qubes-os.org/ <sup 24 fn:24 </sup , or at least running my browser in a VM although I already use the Snap version of Firefox https://snapcraft.io/firefox <sup 25 fn:25 </sup , which does some sandboxing https://snapcraft.io/docs/explanation/security/security-policies/ <sup 26 fn:26 </sup . I’m also slightly worried about Canonical or another vendor I depend on getting hacked and putting malware in the official repos. I haven’t figured out how to mitigate this yet. Summary summary If you’re sufficiently paranoid, it’s possible to have AI agents running on your machine while sandboxing them to a pretty acceptable level, even if they need GPU access and tools like Docker. 1. I intentionally don’t keep 2FA keys in my password manager vault, so this wouldn’t be entirely catastrophic, but most of my accounts don’t use 2FA and it would still be very annoying to change hundreds of passwords. ↩ fnref:1 2. OpenRouter claims that Kokoro-82M’s latency is <300 ms https://openrouter.ai/hexgrad/kokoro-82m performance <sup 27 fn:27 </sup , but in practice I was seeing several seconds, especially for longer inputs. My local machine using a low-end previous generation GPU responds in ~50 ms and processes at 200× realtime. ↩ fnref:2 3. espeak-ng https://github.com/espeak-ng/espeak-ng <sup 28 fn:28 </sup CVEs that haven’t been fixed in a released version: CVE-2023-49990 https://nvd.nist.gov/vuln/detail/CVE-2023-49990 <sup 29 fn:29 </sup , CVE-2023-49991 https://nvd.nist.gov/vuln/detail/CVE-2023-49991 <sup 30 fn:30 </sup , CVE-2023-49992 https://nvd.nist.gov/vuln/detail/CVE-2023-49992 <sup 31 fn:31 </sup , CVE-2023-49993 https://nvd.nist.gov/vuln/detail/CVE-2023-49993 <sup 32 fn:32 </sup . ↩ fnref:3 4. I chose Cloudflare Tunnel over Tailscale Funnel because it adds additional protections in front of your service, including an API key handled by Cloudflare before hitting a second API key handled by my service. ↩ fnref:4 5. My CPU is known to be flaky in some situations. ↩ fnref:5 6. https://www.lesswrong.com/posts/iqgZaJMnmyDCYfnuC/the-hackening-1 - “The Hackening — LessWrong” 7. https://www.lesswrong.com/posts/kS5efcTP86EDTDbGh/brendan-long-s-shortform?commentId=wb8i468tNgdmmKqhY - “Brendan Long’s Shortform — LessWrong” 8. https://github.com/brendanlong/clawed-abode - “GitHub: brendanlong/clawed-abode” 9. https://developers.redhat.com/blog/2020/09/25/rootless-containers-with-podman-the-basics - “Rootless containers with Podman: The basics | Red Hat Developer” 10. https://tailscale.com/ - “Tailscale | Secure Connectivity for AI, IoT & Multi-Cloud” 11. https://code.visualstudio.com/docs/remote/vscode-web - “Visual Studio Code for the Web” 12. https://www.brendanlong.com/gpuc-a-single-user-gpu-queue-that-doesnt-require-sudo.html - “gpuc: A single-user GPU queue that doesn’t require sudo” 13. https://gvisor.dev/ - “The Container Security Platform - gVisor” 14. https://gvisor.dev/docs/user guide/gpu/ nvproxy-security - “GPU Support - gVisor” 15. https://lionreader.com/demo/all?entry=text-to-speech - “Text-to-Speech Narration - Lion Reader” 16. https://huggingface.co/hexgrad/Kokoro-82M - “hexgrad/Kokoro-82M · Hugging Face” 17. https://github.com/brendanlong/optimized-kokoro - “GitHub: brendanlong/optimized-kokoro” 18. https://developers.cloudflare.com/tunnel/ - “Cloudflare Tunnel · Cloudflare Docs” 19. https://vast.ai/ - “Rent GPUs | Vast.ai” 20. https://github.com/google/gvisor/pull/17011 - “GitHub: google/gvisor 17011” 21. https://ubuntu.com/pro - “Ubuntu Pro | Ubuntu” 22. https://ubuntu.com/security/livepatch - “Livepatch | Security | Ubuntu” 23. https://socket.dev/features/firewall - “Socket Firewall” 24. https://www.qubes-os.org/ - “Qubes OS: A reasonably secure operating system | Qubes OS” 25. https://snapcraft.io/firefox - “Install firefox on Linux | Snap Store” 26. https://snapcraft.io/docs/explanation/security/security-policies/ - “Security policies - Snap documentation” 27. https://openrouter.ai/hexgrad/kokoro-82m performance - “Kokoro 82M - Text-to-Speech API Pricing | OpenRouter” 28. https://github.com/espeak-ng/espeak-ng - “GitHub: espeak-ng/espeak-ng” 29. https://nvd.nist.gov/vuln/detail/CVE-2023-49990 - “NVD - Home” 30. https://nvd.nist.gov/vuln/detail/CVE-2023-49991 - “NVD - Home” 31. https://nvd.nist.gov/vuln/detail/CVE-2023-49992 - “NVD - Home” 32. https://nvd.nist.gov/vuln/detail/CVE-2023-49993 - “NVD - Home”