Every Field Was Correct. It Still Wouldn't Connect.

A developer building a self-hosted VLESS+Reality VPN endpoint on an OCI ARM instance using 3x-ui and Claude encountered four configuration traps, including a silent field overwrite by the Generate button and a camouflage target too large for the Reality handshake, where every field was correct yet the connection still failed.

The VPN I share with a few other people ran out of traffic for the month. I had an idle OCI ARM instance sitting there doing nothing, so I decided to stop waiting and build my own node: one VLESS + Reality endpoint, self-hosted, good enough for personal use through Clash Verge. I drove the whole thing with Claude — chatted through the plan, had Claude Code do the Docker deploy, then went back to chat to walk the panel config step by step. The build was an afternoon. The debugging was the rest of the evening, and every minute of it was spent on failures where the configuration was, provably, correct. That's the part worth writing down. Not the happy path — the four traps, and especially the last one, where every field matched on every screen and it still refused to connect. The management panel is 3x-ui https://github.com/MHSanaei/3x-ui , run as a container: docker run -d \ --restart unless-stopped \ -v /etc/x-ui:/etc/x-ui \ -p 443:443 \ -p 2096:2096 \ -p 54321:2053 \ ghcr.io/mhsanaei/3x-ui:latest Three things here are not optional. Persist /etc/x-ui to the host, or every container recreate wipes your config. Set a restart policy unless-stopped or always so the node comes back on its own after a reboot — a proxy you have to SSH in and restart by hand is a proxy you'll abandon. And map only the ports you actually use with -p : the node 443 , the subscription port 2096 here , and the panel. After it's running, docker ps and confirm every one shows up in the PORTS column. The moment it's up, before anything else: log in and change the default credentials. This panel is exposed to the public internet, and default logins are exactly what scanners look for. While you're in there, move the panel off its default 2053 to a random port — that's the 54321:2053 mapping above — and consider changing its base access path. None of this is real security on its own; it just takes you out of the easy-target bucket. On a normal VPS this is one step. On OCI it's two, and missing either one gives you the identical symptom — a plain connection timeout — which makes it genuinely hard to tell which layer is at fault. 0.0.0.0/0 . ufw is separate and just as real: sudo ufw allow 443/tcp the node sudo ufw allow 2096/tcp subscription port, etc. sudo ufw reload Open one and not the other and you'll spend twenty minutes curl -ing from the wrong side of the problem. Open both first. Create a VLESS inbound on port 443 blending in with ordinary HTTPS . Most defaults are fine. Two tabs need care. ⚠️ Trap 1 — the Generate button has a side effect. The Protocol tab has Decryption/Encryption fields; for standard VLESS both should read none . But clicking Generate to create the Reality X25519 key pair also silently overwrites those two fields with an experimental post-quantum encryption string — something like mlkem768x25519plus.native.600s... . This is a newer Xray feature, and Mihomo the core inside Clash Verge can't parse it; it errors out with invaild vless encryption value yes, misspelled that way in Mihomo itself . The fix: after you generate the Reality keys, go back and manually set Decryption and Encryption to none . And don't click Generate again afterward — it re-overwrites the fields and regenerates your key pair, which just adds a fresh variable. On the Security tab, set the security type to Reality and pick a camouflage target — a real TLS 1.3 site your handshake will impersonate. Set uTLS to chrome ; leave short IDs and SpiderX auto-generated. ⚠️ Trap 2 — some targets are literally too large. With www.microsoft.com as the target, every parameter matched and the keys were confirmed identical, but the server log said: REALITY: processed invalid connection ... handshake did not complete successfully This isn't your config. Xray-core reads the target's TLS Certificate record during the Reality handshake and rejects anything over a hardcoded 8192 bytes. www.microsoft.com returns a record around 8273 bytes — just over the line — so the handshake dies. It's a known, reproduced upstream limit Xray-core 6356 https://github.com/XTLS/Xray-core/issues/6356 , 6402 https://github.com/XTLS/Xray-core/issues/6402 , discussion 6387 https://github.com/XTLS/Xray-core/discussions/6387 . The fix is to pick a target with a smaller certificate. www.cloudflare.com works, and the error vanished the instant I switched. On the Add Client page, attach the client to the port-443 inbound you just made, or it's linked to nothing. Then, on the Credentials tab, change Flow from its default None to xtls-rprx-vision to match the server's Reality setup — leave it at None and the connection behaves inconsistently or fails outright. ⚠️ Trap 3 — Clash needs YAML, not the default link. 3x-ui's default subscription link http://ip:port/sub/<id returns the generic Base64 format that clients like V2RayNG expect. Paste that into Clash Verge and you get: the remote profile data is invalid yaml: invalid type: string, expected a YAML mapping Clash/Mihomo needs a YAML subscription, which 3x-ui serves from a different path. In panel settings, confirm the Clash/Mihomo subscription is enabled, check its URI path usually /clash/ , and use: http://server-ip:subscription-port/clash/<subscription-id Import that under Profiles → New → Remote. Only this path returns parseable YAML. Subscription imported, node listed, everything green. Every dial failed: TCP dial ... error: server-ip:443 connect error: REALITY authentication failed We worked it in order — I say "we" because Claude was driving the diagnosis the whole way: nc -zv server-ip 443 from my laptop — TCP connects. Network's fine. curl from the server to the Reality target — clean 200. Target's reachable. docker exec ... netstat -tlnp | grep 443 — the port is listening inside the container. config.json from Everything matched. An assistant that will happily read every value in every file had just confirmed all of them correct — and the connection still failed. The thing that was wrong wasn't in any file. The answer was the version of the running binary. 3x-ui v3.5.0 ships Xray-core v26.7.11 , and that release is deliberately incompatible with Mihomo — a documented breaking change a new default minClientVer , not a local misconfiguration. Multiple people reproduced the exact REALITY authentication failed symptom on v3.5.0, and Mihomo has stated outright it won't support Xray v26.7.11+ 3x-ui 5957 https://github.com/MHSanaei/3x-ui/issues/5957 , 5922 https://github.com/MHSanaei/3x-ui/issues/5922 ; Xray-core 6477 https://github.com/XTLS/Xray-core/issues/6477 . The fix is to downgrade the core to v26.6.27 . Recreating clients, regenerating keys, rebuilding the inbound — none of it matters, because none of it touches the version. There was one more layer to it, and it's the reason the fix looked like it didn't work. After downgrading, the error was still there. What I hadn't accounted for: the panel applies changes through Xray's core API and reports config changes applied through the core API, no restart needed . That hot-reload path does not swap the running Xray process. The old v26.7.11 binary was still serving traffic. A full docker restart 3x-ui loaded the downgraded core, and it connected immediately. So the two facts sit together. The fix was the downgrade. It only took effect on a real process restart — because the panel's "no restart needed" is true for config edits and quietly false for the one thing that actually mattered. When every file says the config is correct and it still fails, stop re-reading the files. The mismatch you can't see is the process, not the config: what version is actually running, and did it actually reload. The panel telling you "no restart needed" is a claim about config edits, not a guarantee about the process — so after any change that matters, restart the container and check the running core yourself. A short checklist, in the order that would have saved me the evening: nc -zv server-ip port — reachability from the client. ufw allow the port. docker exec ... netstat -tlnp — the port is really listening inside the container. none , not the mlkem768... post-quantum string. www.microsoft.com ; www.cloudflare.com is safe. xtls-rprx-vision ; subscription is the /clash/ YAML path, not /sub/ . docker restart the container — don't trust "no restart needed" — then retest.The config was never the hard part. The hard part was trusting, for one evening too long, that a correct file meant a correct process.