✓ Human-authored analysis; AI used for formatting and proofreading.
I traced a production incident back to a Secrets Manager secret that had rotation enabled. The dashboard showed rotation configured. The schedule said 30 days. The compliance report said compliant.
The rotation Lambda had been deleted four months ago.
The credential hadn't changed in 120 days. The audit said 30. Nobody noticed because every tool we had checked whether rotation was enabled not whether it was working.
The gap between configured and functioning is important for cloud security. The gap is a structural problem with how the entire cloud security tool market thinks about detection.
Pick your tool such as Prowler, Checkov, ScoutSuite, Cloud Custodian, AWS Trusted Advisor. They all work the same way under the hood.
A security researcher publishes a finding, AWS releases a best practice or CIS writes a benchmark item. Then the tool adds a check.
S3 bucket is public? CIS benchmark 2.1.5. Add a check. RDS instance unencrypted? CIS benchmark 2.3.1. Add a check. Root account without MFA? Check. CloudTrail disabled? Check.
This is reactive detection. The issue has to be discovered and documented before the tool can find it. The coverage of every tool on the market is bounded by the security community's published knowledge.
If no researcher has written up the issue, no tool checks for it. Tier 1 is the known knowns. CIS benchmarks, AWS best practices, security advisories. Every tool covers it. Public S3 buckets, unencrypted databases, overprivileged IAM roles. Well understood, documented and checked. This is table stakes.
Tier 2 is the known unknowns. Issues that security teams know could exist but don't have tooling to check. Your team probably knows that DNS records can dangle after deleting infrastructure. But does your scanner check 10 different resource types for dangling DNS? Probably not. You know rotation should rotate. But does any tool verify that the rotation Lambda still exists? Teams understand these risks conceptually. They just can't automate the detection.
Tier 3 is the unknown unknowns. Issues that haven't been conceptualized as a class. Nobody had systematically cataloged "things that appear configured but don't function" as a structural pattern that repeats across services. The rotation ghost is one instance. DMARC with p=none is another where the record exists, the auditor marks it compliant, the policy does nothing. A WAF associated with a CloudFront distribution but with no rules is another. SPF with +all is another where the record exists and explicitly authorizes every server on earth.
Each individual instance might be known to a specialist somewhere. The structural pattern, the archetype was never cataloged across services. Nobody was looking for it as a class.
Every tool on the market operates at Tier 1. Some teams manually poke at Tier 2. Nobody systematically covers Tier 3.
The root of the problem is the question these tools start with: "What issues have been reported?"
That question's answers are bounded by what's been discovered. It's necessarily incomplete and always behind.
A different question: "What are all the states this configuration can be in, and which states cause harm?"
That question's answers are bounded by what's possible by the configuration surface itself. It doesn't depend on whether a researcher has written a blog post. It doesn't depend on whether CIS has a benchmark item. It depends only on the configuration options the service exposes.
I've been working through this question systematically for AWS services. For Route 53 alone, I enumerated 195 distinct failure modes. Existing tools cover maybe 5-10, the ones that appear in CIS benchmarks. The other 185 aren't in any benchmark, any scanner, or any published guide. They exist because Route 53's configuration surface allows those states.
Some examples from that enumeration that no tool checks for:
CNAME pointing to a terminated Elastic Beanstalk environment. Beanstalk CNAMEs are reclaimable. An attacker creates a new environment with the same CNAME. Your domain serves their content.
NS record delegating to name servers that don't serve the zone. The delegated zone was deleted. The parent NS records remain. An attacker who controls those name servers or registers the name server domain if it expired controls ALL DNS record types for the delegated subdomain. They can issue certificates via DNS validation. They can receive your email via MX records.
MX record pointing to a decommissioned mail service. Email sent to your domain routes to a server that doesn't exist or worse, one that an attacker now controls. They receive your password reset emails.
They're configuration states that exist in production infrastructure right now. The configurations allow these states. Nobody checks for them.
These failures follow structural patterns that manifest across every service.
A ghost reference is a structural defect that appears everywhere:
Same archetype. Different service and manifestation. Same structural shape.
If you find one ghost reference, your decommissioning process missed it. If the process missed one, it almost certainly missed others. But your scanner only checked the service with data. I've identified 12 of these archetypes across AWS services. The ghost reference archetype alone has over 100 controls spanning dozens of services. The "false protection" archetype are things that appear configured but provide no protection spans Route 53 (DMARC p=none), Secrets Manager (rotation enabled, Lambda deleted), CloudFront (WAF associated, no rules), and more.
You build a CloudWatch alarm. It monitors your RDS instance for high CPU. When it fires, it publishes to an SNS topic. The SNS topic delivers to your Slack channel via webhook.
Someone deletes the SNS topic.
The alarm still works perfectly. CPU spikes. The alarm transitions to ALARM state. The alarm action tries to publish to the SNS topic. The topic doesn't exist. The publish fails silently. Your Slack channel is quiet. Your database is on fire.
Every alarm you've ever built such as RDS threshold, Lambda throttle alert, ECS task count monitor, API Gateway 5xx alarm is useless if the notification chain is broken. The detection works. The notification doesn't. There is no tool that checks it.
This one finding affects every service in your account simultaneously. One deleted SNS topic breaks alerting for everything.
The CISO buys a scanner. It runs and reports 47 findings. The team fixes 47 findings. The scanner runs again. Green dashboard. Compliant.
Few months later, ransomware. The attacker got in through a dangling DNS record that the scanner never checked or a rotation Lambda that was deleted or a notification chain that was silently broken. The alarm fired during the intrusion, nobody was notified.
The CISO looks at the green dashboard. Looks at the ransom demand. Asks the question: "What am I paying for?"
The tool did its job. It checked its list and everything on the list was fine. The breach didn't come through the list. It came through the gap between the list and reality. The tool can't report what it doesn't check. The CISO can't defend a budget for a tool that showed green while the building burned.
The ROI of a security tool is supposed to be: breaches prevented. But you can't measure a breach that didn't happen. So the tool's value gets measured by what is visible such as findings found, fixes, compliance score, dashboard color.
This creates a perverse incentive. The tool finds lots of Tier 1 issues (public S3 buckets, unencrypted volumes, missing MFA). The team fixes them. The dashboard goes green. Everyone feels secure. Budget renewed.
But the tool never checked the thousands of failure modes beyond its benchmark list. The green dashboard is a statement about the list, not about the infrastructure. "Everything on our list is fine" is a different statement from "our infrastructure is secure." The first is true. The second is unknown. The dashboard doesn't distinguish between them.
The executive sees green and hears secure. The tool means list-compliant. The ransom demand comes from the gap between those two meanings.
The value isn't just what you found. The value is being able to say with precision what you have not checked yet. The unknown becomes known because you enumerated it.
An executive can't act on "we might have issues we don't know about." That's vague. That's every security tool's implicit admission. But an executive CAN act on "we have verified 1,200 of 4,400 possible failure modes. Here are the 3,200 we haven't checked yet, grouped by archetype. The ghost-reference archetype alone has 113 controls, we've verified 30 of them. Here are the 83 remaining, the services they cover, and the snapshots needed to check them."
That's not a green dashboard hiding unknowns. That's a coverage map with explicit gaps. The executive sees what's verified and unverified. Can also see what it takes to close the gap. The ROI is measurable: the ratio of verified surface to total surface. The unverified surface is specific. A list of failure modes that haven't been checked yet.
Three numbers change the executive conversation:
Total failure surface. 4,400+ enumerated failure modes across 112 AWS services. This is the denominator. No other tool has this number because no other tool derived their checks from the configuration surface.
Verified surface. The failure modes that have been checked based on available snapshots. This grows as the operator generates more snapshots. It starts small (one service, maybe 150 failure modes) and expands organically through the archetype system. Each finding guides the operator to check related services.
Unverified surface. The gap. Specific, enumerated, grouped by archetype and service. Not "there might be issues we don't know about" but "here are the 3,200 specific failure modes we haven't checked yet, here's what it takes to check them, and here's which archetypes have the most unverified surface."
The executive who sees these three numbers can answer the board's question: "Are we secure?" with an honest answer: "We've verified 1,200 of 4,400 possible failure modes. Our ghost-reference coverage is 90%. Our confused-deputy coverage is 100%. Our false-protection coverage is 60%. Here's the plan to close it. We can't tell you we're 100% secure. We can tell you exactly what we've checked and what we haven't."
That's a different conversation from "the dashboard is green."
The green dashboard tool has a budget problem every year. The CISO can't justify renewal because the tool's value is invisible. It checked a list, the list was fine, and if nothing bad happened, was it because of the tool or because nobody attacked?
The coverage percentage is a measurable metric that improves over time. Quarter over quarter, the verified surface grows. The executive can see progress: "Q1 we covered 40% of the failure surface. Q2 we covered 65%. Q3 we're targeting 80%." The investment has a visible trajectory. The gap is quantified. The remaining risk is specific.
When a breach DOES happen, the conversation is also different. With the green dashboard tool: "The tool showed green. How did we get breached?" With a coverage map: "The breach came through a failure mode in our unverified surface. Specifically, archetype X, service Y. We hadn't generated the snapshot for that service yet. Here's the plan to close that gap."
The first conversation ends a vendor relationship. The second conversation drives expansion.
I'm building this approach into Stave, an open-source CLI tool that verifies cloud configurations using local snapshots. The archetype system groups findings by structural defect class and tells you which other services to check when you find one instance.
If you find a ghost reference in your S3 configuration, Stave tells you: here are the other services where this archetype manifests. Here are the snapshots you need to generate. Here's the command to run. Because if your process missed this one, it missed others. The catalog currently covers 4,400+ failure modes across 112 services, with 670+ compound chains that compose findings across resources. The goal is to shift cloud security from "check the list" to "check the surface." The list will always be incomplete. The surface is the surface.