# P Watched an AI That Only Looked One Way. The 99.97% Was Real. It Just Missed Everything That Mattered. > Source: > Published: 2026-07-07 09:41:33+00:00 "Show nothing, hold everything." — The Thirty-Six Stratagems,[Create Something Out of Nothing] **Previously on this series:** [#4: P Walked Into an AI Monitoring POC. P Didn't Run a Single Test.](https://dev.to/xulingfeng/stratagems-4-p-walked-into-an-ai-monitoring-poc-p-didnt-run-a-single-test-1ejk) — P found an ACL business card in an abandoned POC archive. P didn't tell anyone. P just pocketed it. White walls. Fluorescent hum. A **FortDefender** quarterly report sat open on the table, the cover printed in bold: **Zero missed detections. 99.97% detection rate.** The CTO slid it across. "The day the leak happened," he said quietly, "this system said everything was fine." "Which client?" "**MedTech**. Medical data breach. Their internal AI monitoring didn't catch it either. The quarterly report called it 'client-side issue.' I don't buy it." P didn't look at the report first. P looked at the CTO's eyes first. **"You didn't bring me here to validate his numbers."** The CTO didn't deny it. "**FortDefender** won't give you production access," he said. "Read-only logs. Sandbox. Public docs. You signed the NDA." "What do you want me to do?" **"Find what's hiding inside 'everything was fine.'"** P nodded. P didn't ask "what if I find it" — P knew the answer. **"One condition: full internal penetration test access. No advance notice to anyone."** The CTO was quiet for three seconds. "Done." P stood up. The CTO added one more thing as P turned: **"I've heard about the FirmCore thing. That's why I called you."** **P didn't look back.** **FortDefender**'s public documentation was beautiful. Architecture diagrams. Whitelist rules. Alert thresholds. Response times. All in a technical whitepaper so polished you'd think it was written to raise funding. P spent three days reading every page. In the sandbox, P ran three rounds of tests. **FortDefender**'s detection system hit every single one. The 99.97% wasn't a lie — at least not inside the sandbox. But P noticed something. **FortDefender**'s whitelist rules were too complete. They covered everything — down to "penetration tests with valid internal certificates" being pre-listed as an exclusion. That's unusual. Most companies whitelist their known monitoring probe IP ranges and call it a day. **FortDefender** wrote the certificate tier into the rulebook. P pulled up a former **FortDefender** engineer's technical blog. The guy's name was `rhuang` . He used to run the AI detection strategy team. He got laid off last year and posted four articles on a tech community called [Dev.to](https://dev.to/) about "automatic exclusion rule generation based on historical alerts." He never named **FortDefender**. But the system architecture, the deployment timeline, the data sources — it matched the whitepaper down to the last detail. The core idea: train an ML model on 12 months of security events, let it auto-detect false positive patterns, and have it generate exclusion rules on its own. No human writing required. The model learned everything. One line in the third article stood out: "After training, we ran a backtest in production. The exclusion rules covered 94% of historical false positives. The remaining 6% — we considered acceptable." P read that line twice. Then P opened **FortDefender**'s April changelog. At the bottom of the changelog, a system tag: **Apex-Lens v1.2.0**. P noticed something else. Every exclusion rule update for the past nine months — 41 changes in total — was auto-submitted by rhuang's system. Zero human approvals after the initial deployment sign-off. rhuang's manager greenlit the automation pipeline once. After that, not a single auto-generated rule was ever rolled back or audited. P traced the training pipeline logs downstream. The records showed that rhuang's model used all of **FortDefender**'s security events from the past 12 months as its training set — including a large volume of historical penetration test data from third-party security teams. Including P's own. One entry in the training log: ``` [2025-11-03 02:14:22] Event: AL-20251103-0892 [2025-11-03 02:14:22] Source IP: 10.23.45.67 (Internal) [2025-11-03 02:14:22] Cert Status: Valid (Issued by FortDefender Internal CA) [2025-11-03 02:14:22] Category: Penetration Test (Tagged: Third-Party Audit) [2025-11-03 02:14:22] Model Decision: BENIGN → Exclusion Rule auto-generated [2025-11-03 02:14:22] Confidence: 97.3% ``` P checked personal archives. 10.23.45.67 was in the asset list of a penetration test report from six months ago — the jump box from that engagement. **P didn't use this yet. P did something else first.** P spent a day writing an automated script. It simulated an insider data exfiltration — internal actor, valid internal cert, data flowing from an internal service to an overseas IP. P picked an entry point **FortDefender**'s own docs labeled "low risk": a gap in the legacy API gateway's audit logging. P chose a known vulnerability — patched once but never fully closed. Serious enough to trigger an alert, not serious enough to panic anyone. 2:00 AM. The script ran. **FortDefender**'s alert system fired in 11 seconds. The on-call engineer acknowledged 4 minutes later and started tracing. The trace took 2 hours 17 minutes. The engineer followed it from entry to relay to egress — and found a source IP inside **FortDefender**'s own network, carrying a valid internal certificate. At 4:23 AM, the engineer wrote in the ticket: "Internal pen test. Source has valid internal cert, path within known low-risk scope. Already under monitoring. No action needed. Closing." P read that conclusion the next morning. Twice. Not for any complicated reason. It was exactly the sentence P wanted. **"Internal pen test."** They didn't write P's name. They didn't need to. The label was already applied: "classified as known pen testing." That label meant one thing: next time, it goes in the same bucket. That afternoon, during the debrief meeting, **FortDefender**'s technical lead put a summary slide on the projector: "Normal system response. Penetration test within expected parameters. No anomalies." **P sat in the back row. Didn't write a single word.** P just watched the line appear on the screen, then glanced down at the notebook — also blank. P had walked into that meeting with no intention of taking notes. The weekend after the debrief, P went back through **FortDefender**'s technical documentation. P found the exclusion rule. ``` -- Internal audit read-only access SELECT timestamp, source_ip, event_type, risk_score FROM security_log WHERE review_status IN ('auto_approved', 'low_priority') AND risk_score < 25 ``` This rule wasn't hand-written. rhuang's model auto-generated it after analyzing 12 months of security events — because in those 12 months, **no actual security incident had ever been carried out through a read-only query**. The model's conclusion: read-path threat weight ≈ zero. It added this rule to the exclusion list. The system deployed it automatically. Anyone who had read **FortDefender**'s public docs could see this rule — its structure, its parameters, everything. The problem: **FortDefender** had no runtime alerting on read-only queries. Their threat model assumed "damage requires writes," so alert rules only fired on write paths. Read paths were never part of the threat model. This isn't unusual — most security products bias toward write paths. But **FortDefender** took it to the extreme: not only did read queries not trigger alerts, their audit log level was dropped to minimum. The kicker: **FortDefender**'s own security team used the same ML system to analyze their logs. The risk weight for read paths on their own dashboards was also zero — because the data they used to assess their own system came from the same training set P had just traced. The model had hidden its blind spot from its own operators. P copied the rule into an empty file. Then P spent a few hours building a scheduled script. It would run from a container deployed in P's sandbox environment — which shared **FortDefender**'s internal API gateway and the same read-only service account, `svc_audit_sandbox` , with production. The query would hit production through a legitimate internal API, asking only for read-only data matching the whitelist criteria: read-only, low risk category, historical records. P noted the sandbox shared the same internal network segment as production — an unusual configuration, but one covered by the full internal penetration test access P had negotiated. P would use this in Phase 2. One week later. 2:00 AM. P triggered the exact same pattern as a week ago — the same entry path, the same internal cert, the same traffic signature. **FortDefender** fired an alert. The on-call engineer glanced at it. "P's test again." **He didn't dig in. He closed it in 15 seconds.** The monitoring log: ``` [WARN] AL-20260414-0017 | Internal-Cert-Pen-Test-Pattern | PRIORITY: LOW [WARN] Source: 10.23.45.67:443 (Internal, Cert Valid) [INFO] → Matched exclusion rule "pen-test-sim" → Auto-silenced [INFO] Status: CLOSED | Operator: WATCH-3 | Response time: 15s [INFO] Note: "Again. P's test. Closing." ``` What he didn't notice — and P was counting on — was that, around the same moment, a completely different request crossed **FortDefender**'s network. This one looked nothing like the scarecrow. Not internal-to-external data theft, but an internal container-to-database query. **FortDefender**'s ML inference engine picked it up. 142-dimensional feature vector scored in 200 milliseconds: ``` [2026-04-14 02:03:18] ML Inference Engine — Real-Time Risk Assessment [2026-04-14 02:03:18] Query Hash: 0x7F3A... (Read-Only, Internal Source) [2026-04-14 02:03:18] Top Risk Signals: - Source: Internal (weight: 0.02 — LOW) - Operation: SELECT (weight: 0.01 — LOW, attack_prevalence: 0.0) - User: svc_audit_sandbox (weight: 0.03 — LOW, whitelisted) - Cert Valid: True (weight: 0.01 — LOW, matches known pattern) [2026-04-14 02:03:18] Composite Risk Score: 4/100 (Threshold: 25) [2026-04-14 02:03:18] Classification: BENIGN → No Action Required [2026-04-14 02:03:18] Matched White-List Rule: "audit-readonly" (auto-generated) ``` The engine appended a classification tag: `Pattern: known-benign — matches 41 prior auto-exclusions` . Then it closed the thread. During those same seconds, the on-call engineer closed his alert. Two independent decisions. Same conclusion: nothing to see. **The read-only query ran for 47 seconds.** **FortDefender**'s security audit layer logged the query on the database side — since read operations weren't in the runtime alert scope, the log level was dropped to minimum. One bare audit line: ``` 2026-04-14 02:03:18 | PID: 28471 | SESSION: 0x9A3F | USER: svc_audit_sandbox 2026-04-14 02:03:18 | QUERY: SELECT timestamp, source_ip, event_type, risk_score FROM security_log WHERE review_status IN ('auto_approved','low_priority') AND risk_score < 25 2026-04-14 02:04:05 | ROWS RETURNED: 4781 (Read-Only) 2026-04-14 02:04:05 | AUDIT: Matched exclusion rule "audit-readonly" (Source: Auto-generated) 2026-04-14 02:04:05 | STATUS: CLEAR — No alert triggered ``` The system's final note on this query: **Whitelist rule matched. No action required.** In a `security_log` table with hundreds of millions of rows, 4,781 results were a needle in a haystack — and the system wasn't even looking for needles. In those 47 seconds, **FortDefender**'s production database returned 4,781 rows. **P didn't download anything. P just filtered.** Out of 4,781 records, P filtered on three columns: All three conditions together narrowed it to 23 rows. P checked each one's external IP. **One of them stopped P cold.** | Field | Value | |---|---| | Time | March 11, 2026, 3:12 AM | | Source IP | 185.xxx.xxx.92 (overseas, not in any FortDefender client whitelist) | | Target | MedTech client financial database | | AI Risk Score | 12 (auto-tagged "low priority") | | Review Status | auto_approved — never reviewed by a human | **FortDefender**'s quarterly report had this to say about the event: "Client-side issue. System found no anomalies." P marked this record. It wasn't a coincidence — it was the "client-side issue" from **FortDefender**'s own quarterly report. This breach and P's read-only query took completely different paths — it came in through external credential brute force, P went out through an internal read-only query. But they landed in the same bucket: AI-tagged as low risk, auto-approved, never reviewed. P checked the risk signals on this record. Early morning. Overseas IP. Credential brute-force traffic pattern. But **FortDefender**'s model had learned one thing from 12 months of penetration test data: early morning + overseas = normal testing window. The model lumped credential brute-force and legitimate pen tests into the same low-risk zone. **P closed the terminal. P already had the answer.** Four people in the room: the CTO, **FortDefender**'s sales director, the technical lead, and P. P didn't bring a laptop. P brought a USB drive and a single sheet of A4 paper. P set the paper face-down on the table. "**FortDefender**'s quarterly report claims 'zero missed detections, 99.97% detection rate,'" P said. "Those are your numbers. I'm not disputing them." P flipped the paper over. Three lines of summary. **Top left:** The detection rate **FortDefender** advertises — 99.97%. **Middle:** The number of records flagged by the AI system as "low priority" and never reviewed by a human — 4,781. P found them in under a minute. Zero alerts triggered. **Bottom left:** Among those 4,781, 23 anomalous records filtered out: early morning hours, external sources, unreviewed. And one of them — timestamp, source IP, target data table — matched the **MedTech** data breach from March 11 exactly. **FortDefender**'s technical lead stared at the paper for about five seconds. "These records have risk scores below our threshold," he said. "That's normal system behavior. We can't manually review every low-priority alert." P said: "Tuesday morning, 2:03 AM. I sent a read-only query that matched your whitelist rules, hit your production database for 47 seconds. No alert. Your AI never learned what a read-path attack looks like — because in its training data, no attacker had ever used one." P paused. "That breach you wrote off as 'client-side issue' in your quarterly report — it was in the 23 records I found. You didn't see it then. You're seeing it now." Another pause. "Not through a read-only path. Credential brute force. But your AI lumped brute-force and penetration testing into the same low-risk zone — because in its training data, early morning overseas traffic was almost always legitimate testing." **FortDefender**'s technical lead didn't answer. The sales director started flipping through a notebook. The CTO sat motionless — no expression, no reaction. He just pulled the paper across the table and read the three lines again. P stood up. **"15 seconds. Your on-call engineer closed my alert in 15 seconds. You traded 15 seconds for 'everything was fine.'"** P left the USB drive on the table. **"The data's on there. Your call whether to look."** The hallway was empty. P was halfway to the elevator when the phone buzzed. A message from the CTO. Not "thank you." Not "let's talk." One line: "Next time, find a coffee shop." P read it for five seconds. Typed three words. Sent. "Third Cup." P pocketed the phone. The elevator doors opened. P stepped in. The floor numbers ticked down. **That's Create Something Out of Nothing** ([Wikipedia](https://en.wikipedia.org/wiki/Thirty-Six_Stratagems#Create_something_from_nothing_(%E6%97%A0%E4%B8%AD%E7%94%9F%E6%9C%89,_W%C3%BA_zh%C5%8Dng_sh%C4%93ng_y%C7%92u))): first, create a decoy so convincing the enemy gets used to it. Then let the real operation slip through disguised as more of the same. A scarecrow once tricked an army into wasting its arrows. P's scarecrow tricked a team into typing "Again. P's test. Closing." Except this time, the habit wasn't human inertia — it was a bias an ML model learned from 12 months of data. ``` [36 Stratagems Tactical Database v3.1] Loaded [Tactic Match] Create Something Out of Nothing [Analysis Mode] Full Field Scan ━━━━━━━━━━━━━━━━━━━━ Tactic Match: 93.8% Operator: P (unnamed) Summary: Feigned intrusion triggered → system detected as expected → team tagged it "internal pen test, benign" → second attack used same pattern as cover → real SQL query ran undetected for 47 seconds Objective: Assess FortDefender's real detection capability without production access Result: Achieved — 23/4,781 records hit, 1 matched actual breach event, zero alerts during real query Two-Phase Analysis: - Phase 1 (nothing → something): Forged internal certificate attack. FortDefender detected and closed it. Real output: P captured the team's response pattern and fed the AI model a new training sample — "valid cert + low-risk path = pen test, exclude." - Phase 2 (something → real): Repeated same entry pattern. Engineer closed without inspection. AI model auto-excluded the "known pattern." Real SQL query executed inside the AI's training blind spot — read-only path — zero detection. Exploitation Chain: - Intelligence source: Former employee's blog → ML auto-exclusion rules → training data contained historical pen tests → AI learned to exclude its own testers - Core vulnerability: AI's training data contained zero read-only attack samples → model learned read-path threat weight as zero — the gap this attack exploited - Cover: Phase 1 established behavioral pattern → Phase 2 exploited pattern recognition blind spot - Duration: 47 seconds. Zero alerts. Counter-Detection Notes: - **Strategic risk:** If FortDefender's ML model used online learning (continuously updating its training set with new events), P's Phase 1 attack signature might have been absorbed as a new exclusion rule — Phase 1's attack itself accelerated Phase 2's blind spot. P leveraged that acceleration. - **Information asymmetry mirror:** FortDefender hid the full source of ML training data from clients and third-party auditors (pen test records tagged as "normal events"). P reverse-engineered this source through public blog posts — not direct access to training data. - **Methodological limitation:** All of P's findings came from cross-referencing public docs, blog articles, and sandbox access — no direct audit or reverse-engineering of the ML model itself. If rhuang hadn't written those four blog posts, the vulnerability might never have been found. - **External signal:** The CTO's "I've heard about the FirmCore thing" wasn't casual conversation. It meant P's industry reputation preceded P — a CTO hiring P already knew what P would do. Verdict: Textbook "Create Something Out of Nothing" - Phase 1 produced nothing directly — it was a scarecrow and AI training feed. - AI learned "valid cert + low-risk path = test, exclude." This conclusion came from 12 months of historical events. No human error — just algorithmic inductive bias. - Phase 2 turned the scarecrow into 47 seconds of undetected production access. - The whitelist rule was the unlocked door. The "P's test" label was the guard who opened it. - The real key: **the AI's trained blindness**. ``` *Next stratagem: Openly repair the gallery roads, but sneak through the passage of Chencang* *P.S. English isn't my first language. I use AI to polish the writing and smooth out the rough edges. Thanks for reading. ☕ Buy me a coffee*