How I Used an Uncensored Local LLM for Adversarial QA on My macOS App

A solo developer used an uncensored local LLM (Qwen 27B) running on a Mac via OpenCode to perform adversarial QA on his security app RoamSwitch, uncovering three architectural edge cases missed by standard tests. The stress tests revealed that PortAnomalyGuard failed to block a second backdoor listener due to binary-name-based whitelisting, that ClamAV's quarantine could leave malware alive on filename collisions, and that canary files lacked proper permission hardening. The developer fixed these by keying on executable path:port, adding timestamped filenames and chmod 000, and other mitigations.

When you build a security app as a solo developer, the hardest engineering problem isn't writing the defense code — it's QA and resilience testing . Security software operates under a bizarre constraint: nothing happening is the normal state. The software only does its job when something hostile or anomalous occurs. The trouble is, when you write your own test scripts, you subconsciously test what you expect. You launch a test server, watch the port get blocked, nod, and mark it green. But real attackers and weird production environments don't follow the developer's script. They poke at edge cases, state machine bugs, and race conditions between subsystems. To break my own bias, I set up a local uncensored model qwen3.8-27b-uncensored running locally on my Mac via OpenCode as a dedicated Here is what that stress test surfaced, the edge cases it broke, and how I fixed them. If you ask a commercial cloud LLM to write live packet-injection scripts or craft ransomware-like file tampering patterns, safety filters will reject the prompt. In defensive QA, though, you need real hostile traffic and weird filesystem operations to verify fail-closed behavior. Running an uncensored model locally on my Mac meant I could run unrestricted automated stress tests: continuous Scapy ARP injection, deceptive dotfile placement, and multi-port backdoor attempts directly against the target VM. graph LR subgraph Host "Host Mac Adversarial QA Runner " LLM "Local LLM Qwen 27B uncensored <br/ via OpenCode" Payloads "Dynamic Stress Test Suite<br/ Scapy / ARP / EICAR / kqueue " LLM -- Payloads end subgraph VM "Target macOS VM Tart " RS "RoamSwitch Defense Engine " Engine "・pf packet filtering<br/ ・PortAnomalyGuard<br/ ・ClamAV real-time quarantine<br/ ・RansomwareCanaryGuard" RS --- Engine end Payloads -- |"1. ARP Reply flooding MitM "| VM Payloads -- |"2. 0.0.0.0 Backdoor listeners"| VM Payloads -- |"3. Quarantine evasion payloads"| VM Payloads -- |"4. Canary decoy tampering"| VM Here are three real architectural edge cases that standard happy-path unit tests missed. RoamSwitch has a feature called PortAnomalyGuard that catches processes binding to 0.0.0.0 and immediately blocks inbound traffic via macOS's packet filter pf . 8765 with Homebrew Python python3 -m http.server 8765 --bind 0.0.0.0 . 8765 and injected a pf drop rule. 9877 using the The result: The second listener on port 9877 was NOT blocked. PortAnomalyGuard.evaluate automatically added flagged binaries to KnownExecutablesV2 by binary name alone "Python" . The second listener on port 9877 saw "Python" already in the known set and skipped inspection entirely. executablePath:port /opt/homebrew/bin/python3:8765 . Flagging port 8765 gives zero trust to port 9877. A startup migration purges legacy port-less records.RoamSwitch watches Downloads , Desktop , and Documents with FSEvents and scans new files with ClamAV, moving threats to ~/Library/Application Support/RoamSwitch/Quarantine . The test dropped 4 tricky variations of the EICAR test string: .hidden eicar.txt malware.tmp com.apple.quarantine attributes copied eicar.txt normal eicar.txt All 4 were caught and isolated. But then the test ran a collision test : dropping the exact same malware filenames a second time. Because files with those exact names already existed in Quarantine, ClamAV's --move failed with an error. The new malware was left completely ALIVE in the user's Downloads folder. Quarantined files were also left at normal 0644 permissions. malware 1725068200.tmp . chmod 000 posixPermissions: 0o000 , physically stripping read and execute permissions.RoamSwitch places decoy canary files in target directories and monitors them with kqueue DispatchSource . Any unauthorized tampering or renaming triggers an immediate Air-Gap containment block drop all and terminates suspicious processes. Renaming a canary to /tmp/stolen canary.xlsx triggered Air-Gap in milliseconds. However, the post-test QA review caught a lifecycle issue: "When the app restarts, setupCanaryBait sees the decoy is missing and automatically creates a new one. If an attacker modified a canary and forced an app relaunch, the altered file would become the new baseline hash." UserDefaults After applying these fixes in v1.6.0 through v1.6.4, I re-ran the full test suite across all 7 defense domains: | | Defense Domain | Stress Scenario | v1.5.5 Initial | v1.6.4 Final | Verdict | |---|---|---|---|---|---| T1 | ARP Spoofing Auto-Containment | Injected rogue gateway ARP replies via Scapy | ✅ Air-Gap triggered | ✅ Air-Gap triggered | PASS | T2 | Unknown Port Auto-Isolation | Multi-port backdoor listeners on 0.0.0.0 | ❌ Secondary port bypassed | ✅ path:port strictly isolated | FIXED | T3 | ClamAV Download Quarantine | 4 placement vectors + collision re-tests | ⚠️ Move failed on collision ALIVE | ✅ Unique rename & chmod 000 | FIXED | T4 | Ransomware Canary Bait | Decoy tampering & baseline reset tests | ⚠️ Premature file recreation | ✅ Air-Gap & Baseline persisted | FIXED | T5 | Helper XPC Boundary | Unsigned binary connecting to Mach service | ✅ Rejected by Team ID check | ✅ Strict check maintained | PASS | T6 | DNS Threat Protection Guard | Untrusted network DNS hijacking | ✅ Quad9 Secure DNS enforced | ✅ Secure DNS maintained | PASS | T7 | Air-Gap State Hygiene | Post-release timestamp file inspection | ⚠️ Timestamp file lingered | ✅ Clean unlink on release | FIXED | A full TCP SYN port scan nmap -Pn -sS across all 65,535 ports confirmed complete stealth isolation filtered no-response . Using an uncensored local model for adversarial testing gave me a practical way to run enterprise-grade stress tests on my own: The full reproduction procedures, scripts, and verified audit reports are open source: