A Japanese version of this is on Zenn.
I build and sell a macOS network-security app as a solo dev. The landing page says:
Zero Telemetry— no outbound traffic, no data collection
Words are cheap, and that's just a claim. To a user, this is "an unknown solo dev's app that rewrites your firewall with root, saying 'trust me, I send nothing.'" There's no reason to believe it.
And this app sees sensitive stuff, constantly: the gateway MAC and fingerprint, listening ports, the Wi-Fi SSID you're on, FileVault / SIP / firewall state, files it scanned, URLs it checked. If any of that leaked, it would betray exactly the people who picked it for privacy.
So I decided to turn the claim into a measurement. tcpdump and free tools only — wiretap my own app. This post is the method and the places I tripped.
"The MCP server and detection logic are open source — just read them." A few holes:
1. The main app isn't open source. Only the read-only MCP server and detection logic are public; the privileged helper, pf
control, and licensing stay closed (that surface helps attackers). Code alone doesn't cover the whole thing.
2. No App Sandbox. To be upfront: RoamSwitch does not run inside App Sandbox — it controls the firewall and sharing daemons, so it can't. The consequence is that entitlements don't restrict outbound traffic. codesign -d --entitlements
showing "no network-client
key" is not proof it can't send. You have to show behaviour.
3. Regressions. Even if this build is truly silent, a future version could add an analytics SDK, a crash reporter, a "check for news" feature. A one-time check is worthless — this needs to be a procedure you can re-run every release.
So instead of "read the code," I hand people "run this procedure."
The only traffic allowed out is the four paths documented in the whitepaper (§7):
| # | destination | process | when |
|---|---|---|---|
| 1 | lafine.net/api/v1/license/* |
||
| RoamSwitch | license activate / deactivate only | ||
| 2 | lafine.net/updates/appcast.xml |
||
| RoamSwitch | Sparkle (launch + every 24 h) | ||
| 3 | ClamAV mirrors | ||
freshclam (not the app) |
|||
| only with ClamAV installed | |||
| 4 | |||
HEAD to a target URL |
|||
| RoamSwitch | only when the in-app Link Safety sheet is used |
Anything else out of RoamSwitch
/ RoamSwitchHelper
/ RoamSwitchMCPServer
is a fail.
I don't own Little Snitch, so:
brew install wireshark
) — pull destinations and TLS SNI from the pcapLuLu is a fine Little Snitch substitute, and it's by a well-known name in Mac security — which, for a write-up, only helps.
I'd convinced myself that "Sparkle's periodic check is every 24 h, so I have to capture for 25 h straight to span it." I set up for an overnight run.
Then I looked at Info.plist
: SUScheduledCheckInterval = 86400
, and SULastCheckTime
sits in defaults
in the clear. So there's no waiting:
BID=com.tetsuharu.RoamSwitch
defaults delete $BID SULastCheckTime
osascript -e 'quit app "RoamSwitch"'; sleep 2; open -a RoamSwitch
The appcast HTTPS GET is byte-for-byte identical whether it comes from this, the "Check for Updates" menu item, or a fresh first launch.
The only reason to run long is to catch a wall-clock "once a day" telemetry beacon — an unusual design, and there's no analytics SDK in the app anyway (open source, checkable). So a few hours of idle is plenty. Total active work: ~3–4 h, one sitting.
tcpdump tells you "a SYN went to 140.82.112.x." It does not tell you whether RoamSwitch, Chrome, Dropbox, or the OS sent it.
On a daily-driver machine, even idle, there's a flood of non-Apple traffic: Safe Browsing, component updaters, cloud sync, chat keep-alives. Picking RoamSwitch's packets out of that by timestamp and port is genuinely error-prone — you end up waving things away as "probably Chrome."
So you make the machine silent first. A dedicated account or spare Mac is ideal; quieting the same login also works:
launchctl bootout
the third-party agents (reverts on reboot)softwareupdate --schedule off
, analytics off, Spotlight suggestions offThen, before launching RoamSwitch, run tcpdump for 10 minutes and check that every remaining destination is Apple (17.0.0.0/8
, etc.). If a non-Apple destination is still there, something's still running — find it with lsof -nP -iTCP@<IP>
, stop it, start over.
Only once that baseline is clean can you lean on "any non-Apple traffic after this ≈ RoamSwitch."
Don't open this procedure in a browser on the test machine. Chrome especially is a pile of non-Apple traffic (Safe Browsing, a GCM keep-alive). Use a second device, print it, or pre-cache the page and use Safari.
Reproduce the situation RoamSwitch is built for: you join a café / public Wi-Fi and the app locks down automatically.
At max lockdown, RoamSwitch's pf
blocks other apps' egress too, so the capture is even quieter and its own traffic stands out. Crucially, appcast and license still get through — RoamSwitch puts its own traffic in a pf
exception. Part of the point of the test is to confirm that exception is minimal and only points where it's supposed to (i.e. it didn't quietly whitelist a telemetry endpoint for itself).
sudo tcpdump -i any -n -w cap-%Y%m%d-%H%M.pcap -G 3600 -W 72 -Z root \
'(ip or ip6) and not net 127.0.0.0/8 and not net 169.254.0.0/16 \
and not net 224.0.0.0/4 and not port 5353 and not port 53 and not port 67 and not port 68'
sudo bash -c 'while :; do ts=$(date -u +%FT%TZ); \
lsof -nP -iTCP -sTCP:ESTABLISHED +c0 2>/dev/null \
| awk -v ts="$ts" "NR>1{print ts,\$1,\$2,\$9}"; sleep 5; done' | tee lsof.log
log stream --style syslog --predicate 'subsystem BEGINSWITH "com.objective-see"' > lulu.log &
The MCP server only speaks over stdio, so I drive every tool through it and separately confirm its PID never opens an outbound socket.
After the idle hold and the phases (run every feature by hand once; deliberately fire the four known paths), pull destinations from the pcap and cross-reference with the process log by timestamp:
for f in cap-*.pcap; do
tshark -r "$f" -Y 'tcp.flags.syn==1 && tcp.flags.ack==0' -T fields -e ip.dst
done | sort -u
awk '$2 ~ /^RoamSwitch/ {for(i=1;i<=NF;i++) if($i ~ /->/){split($i,a,"->"); print $2,a[2]}}' \
lsof.log | grep -vE '127\.0\.0\.1|::1' | sort -u
Pass criteria:
RoamSwitchHelper
— RoamSwitchMCPServer
— RoamSwitch
— nothing outside lafine.net
(and the URL you hit for path #4)freshclam
, not the appVerdict: PASS. Across a ~2-hour capture (forced Sparkle check, every feature exercised once, DNS Threat Guard toggled, a network transition, security level pinned to Maximum Lockdown):
Zero non-local outbound connections attributed to RoamSwitch, RoamSwitchHelper, or RoamSwitchMCPServer.
The only sockets the MCP server opened were 127.0.0.1:5000
and 127.0.0.1:7000
— the get_exposed_ports
tool checking HTTP headers on locally-listening dev servers. Nothing left the machine.
Here is every HTTPS destination in the whole capture (TLS SNI):
bag.itunes.apple.com gspe35-ssl.ls.apple.com proxy-safebrowsing.googleapis.com
configuration.apple.com init.push.apple.com setup.icloud.com
dns.quad9.net mask.icloud.com swdist.apple.com
gateway.icloud.com mesu.apple.com swscan.apple.com
gsa.apple.com ocsp2.apple.com tether.edge.apple
p117-contacts.icloud.com courier.push.apple.com swallow.apple.com
fbs.smoot.apple.com
lafine.net <- the only non-OS destination
Strip out the OS's own services (iCloud, push, software update, OCSP, Safari Safe Browsing) and one destination remains: lafine.net
— Sparkle fetching appcast.xml
(egress path #2). dns.quad9.net
shows up because the DNS Threat Guard switched the system resolver; the encrypted lookups are mDNSResponder
's, not RoamSwitch's. swallow.apple.com
/ fbs.smoot.apple.com
sit on AWS IPs but are Apple services.
entitlements:
$ codesign -d --entitlements :- /Applications/RoamSwitch.app/Contents/MacOS/RoamSwitchHelper
<dict></dict>
$ codesign -d --entitlements :- /Applications/RoamSwitch.app/Contents/MacOS/RoamSwitchMCPServer
<dict></dict>
Empty — not even com.apple.security.network.client
(which, as noted, wouldn't enforce anything without App Sandbox, hence the behavioural check).
LuLu logged no RoamSwitch-related outbound events for the duration.
It's all in one shell script, in the audit/
folder of the roamswitch-support
repo. ./rs-zerotel-audit.sh all --idle 2h --clam
runs prep, capture, forced Sparkle check, MCP exercise, analysis, a PASS/FAIL verdict, and a shareable FINDINGS.md
. The only manual bits are helper approval, pinning max lockdown, one network transition (a Wi-Fi off/on is enough), and a few menu actions.
Being upfront:
But "a claim" versus "a claim + a procedure anyone can run + an honest statement of the limits" is a very different weight of trust. The more skeptical someone is, the more they want something they can check themselves.
Script and run sheets (EN/JA): github.com/lafine1211/roamswitch-support, audit/