✓ Human-authored analysis; AI used for formatting and proofreading.
If you're an attacker who's just landed in an AWS account, the most expensive thing about your future is detection. Every API call you make leaves a trail in CloudTrail, gets forwarded to a SIEM, becomes evidence the team uses to boot you out and reconstruct what you touched. The cost of your campaign rises linearly with the number of events on record.
So you make one call first:
aws cloudtrail stop-logging --name org-audit-trail
The trail still exists. The console still shows it. aws cloudtrail describe-trails
still returns it. The S3 bucket is still configured as the destination. To anyone glancing at the configuration, the audit infrastructure looks healthy.
Only aws cloudtrail get-trail-status
reveals that IsLogging
is false
. New events stop landing in the bucket. The audit clock s. Whatever the attacker does next happens unobserved, and remains unobserved until somebody notices.
This pattern is established enough to have its own MITRE ATT&CK identifier: T1562.008 (Disable or Modify Cloud Logs). It appears in cloud incident write-ups across Mandiant's catalogue, AWS Security Bulletin guidance, and pen-test reports against organisations whose detection budget went into SIEM and not into CloudTrail-config monitoring.
A stopped trail is structurally healthy. Every other configuration field is normal:
{
"Name": "org-audit-trail",
"S3BucketName": "acme-cloudtrail-archive",
"IsMultiRegionTrail": true,
"IsOrganizationTrail": true,
"EventSelectors": [...],
"InsightSelectors": [],
"KmsKeyId": "arn:aws:kms:us-east-1:...",
"LogFileValidationEnabled": true
}
The only field that flips is the one get-trail-status
returns separately:
{
"IsLogging": false,
"LatestDeliveryTime": "2026-01-01T00:00:00Z",
"StartLoggingTime": "2025-09-15T00:00:00Z",
"StopLoggingTime": "2026-01-01T00:00:00Z"
}
Two timestamps and a boolean. That's the entire signal. Detection requires reading the right API — GetTrail
and DescribeTrails
both return the configuration but not the running state.
The audit blind window is the difference between a 30-day incident response and a 6-week one. Concrete things the attacker does inside that window:
AdministratorAccess
, an access key for offline use, a Lambda function with a privileged execution role that the attacker invokes via API Gateway.None of these leave a CloudTrail event. All of them fail audit review because there is no audit. The investigator's first question: "what did this principal do between Jan 1 and Jan 8?" has no answer.
Operators who built the CloudTrail configuration usually checked their work. The bucket is encrypted, the trail is multi-region, log file validation is on, the bucket policy denies external principals. Every static configuration field passes audit review.
The mistake is treating "trail exists and is configured" as equivalent to "trail is running." They are not the same. The IAM permission that creates the trail (cloudtrail:CreateTrail
) is different from the permission that toggles its state (cloudtrail:StopLogging
, cloudtrail:StartLogging
). Roles permitted to do one routinely have permission to do the other — incident-response runbooks frequently include "start logging if it stopped" without restricting the inverse.
Every CloudTrail trail must haveIsLogging: true
,
and at least one trail in the account must be
multi-region.
In Stave's observation schema, the trail's running state is a first-class field on the asset:
{
"id": "arn:aws:cloudtrail:us-east-1:111122223333:trail/org-audit-trail",
"type": "aws_cloudtrail_trail",
"properties": {
"trail": {
"name": "org-audit-trail",
"is_logging": false,
"is_multi_region_trail": true,
"stopped_at": "2026-01-01T00:00:00Z",
"stopped_by": "arn:aws:sts::111122223333:assumed-role/incident-actor/session"
}
}
}
The collector that produces this observation calls GetTrailStatus
for every trail, picks up the IsLogging
boolean, and surfaces it directly. The diagnostic fields (stopped_at
, stopped_by
) carry forensic context the responder uses next; the predicate only reads the booleans.
id: CTL.CLOUDTRAIL.STOP.DETECT.001
name: CloudTrail Trails Must Be Actively Logging in All Regions
severity: critical
unsafe_predicate:
any:
- field: properties.trail.is_logging
op: eq
value: false
- field: properties.trail.is_multi_region_trail
op: eq
value: false
any
— either the trail is stopped, or the trail is single-region (which leaves the other regions blind even when the configured one is running). Severity is critical
because the trail being stopped is the configuration that determines whether every other finding in the account is investigable; the impact multiplies across the rest of the security posture.
This is a presence check, not a reachability question. The collector observes two booleans; the predicate is a two-leaf disjunction. There is no search space. There is no witness to enumerate.
Z3 earns its complexity when the question is "what specific request does this configuration admit?" — that's reachability over a permission set. The question here is "is the audit infrastructure running?" — a flat read on a status field.
A self-contained example is in the repo: stave/examples/cloudtrail-stop-logging/
:
go run ./examples/cloudtrail-stop-logging before
Captured stdout:
=== before (trail stopped) ===
status: NON_COMPLIANT total_assets=1 violations=1
CTL.CLOUDTRAIL.STOP.DETECT.001 fired on 1 asset(s):
- arn:aws:cloudtrail:us-east-1:111122223333:trail/org-audit-trail severity=critical exposure_score=100.00
assertion: fires=true (expected) ✓
After aws cloudtrail start-logging
runs:
=== after (trail re-started) ===
status: COMPLIANT total_assets=1 violations=0
CTL.CLOUDTRAIL.STOP.DETECT.001: no findings
assertion: fires=false (expected) ✓
The before-fixture carries stopped_by
— assumed-role/incident-actor/session
— which is the forensic breadcrumb the responder uses next. CloudTrail itself recorded the StopLogging
event before stopping; EventBridge can route that event to the SOC even after the trail goes silent, so the first unauthorised stop is detectable. The risk is detection latency. A six-hour latency is enough window for a determined attacker.
The remediation is a single command:
aws cloudtrail start-logging --name org-audit-trail
A follow-up question every IR runbook should answer: what was logged in the audit-blind window? The answer is "nothing through this trail." Other data sources fill in the gap to varying degrees:
None of these are a substitute for the trail. Together they shorten the audit-blind window's effective length.
Three layers, in order of leverage:
Service Control Policy denying StopLogging for any non-bootstrap principal. The strongest layer:
{
"Sid": "DenyCloudTrailStopLogging",
"Effect": "Deny",
"Action": [
"cloudtrail:StopLogging",
"cloudtrail:DeleteTrail",
"cloudtrail:UpdateTrail",
"cloudtrail:PutEventSelectors"
],
"Resource": "*",
"Condition": {
"ArnNotLike": {
"aws:PrincipalArn": [
"arn:aws:iam::*:role/CloudTrailAdministrator"
]
}
}
}
The deny applies to every principal except the explicit admin role. An attacker who lands on a compromised Lambda or compromised user has no path to StopLogging
; the SCP rejects the call before it reaches CloudTrail's API.
EventBridge alarm on StopLogging events. The trail records the
StopLogging
event before going silent. Route that specific event to the SOC channel with a high-priority page:
{
"source": ["aws.cloudtrail"],
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
"eventSource": ["cloudtrail.amazonaws.com"],
"eventName": ["StopLogging", "DeleteTrail"]
}
}
The alarm catches the first unauthorised stop. Every moment the attacker waits is a moment the page rings.
** stave apply in CI** against the post-deploy observation snapshot. The example shipped with this article is the template. PRs that cause a trail's
is_logging
to go false produce exit code 3.cloudtrail:StopLogging
,
cloudtrail:DeleteTrail
,
cloudtrail:UpdateTrail
,
cloudtrail:PutEventSelectors
for any non-bootstrap principalStopLogging
, DeleteTrail
events from the management trail and pages the on-call SOCstave apply
runs in CI against post-deploy observations; PRs that introduce a stopped trail failThe attacker's first action is StopLogging
. The defender's first action should be making StopLogging
hard to call without paging the on-call. The window between the call and the page is the budget the attacker has to spend; closing the window to seconds is how the rest of the security posture stays investigable.
The control above asks one question: is the trail running? It's a presence check on is_logging
and is_multi_region_trail
. The CEL predicate is two booleans ANDed together. That's the headline-grabbing case where the attacker who calls StopLogging
to evade detection.
The Bybit / Safe{WALLET} writeup (March 2025) makes a quieter point: "modifying objects in an S3 bucket can often be hard to detect because, by default, CloudTrail logging for S3 object activity is disabled." That is a different gap. Management events log CreateBucket
, PutBucketPolicy
, DeleteBucket
. Data events log GetObject
, PutObject
, DeleteObject
. Most production trails enable management events and stop there — data events are high-volume, expensive, and opt-in.
The trail in data-events-before/
is a normal production CloudTrail:
trail_name: org-management-trail
is_logging: true
is_multi_region_trail: true
include_global_service_events: true
log_file_validation_enabled: true
event_selectors:
- read_write_type: All
include_management_events: true
data_resources: [] # ← empty
Five booleans say "logging is healthy." The compliance dashboard shows green. The CEL control above (CTL.CLOUDTRAIL.STOP.DETECT.001
) reports no findings — because, by its definition, there is nothing wrong: is_logging=true
and is_multi_region_trail=true
. The check passes.
But the account contains three production buckets — company-frontend-prod
(serves JavaScript via CloudFront), customer-data-prod
(PII), and financial-records
(financial data) — and none of them appear in any trail's data_resources
. An attacker who modifies app.js
in the frontend bucket leaves no CloudTrail record. The security team's dashboard stays green.
The earlier section "Why Z3 Doesn't Help Here" is honest about the original control: presence checks have no search space, no witnesses to enumerate. CEL is the right shape.
The data-events question is different. It asks: is there a sensitive bucket whose object operations are not covered by any trail's data_resources? That has the shape Z3 is built for — quantifier over buckets, predicate combining tag classification and trail coverage. The prover at
examples/cloudtrail-stop-logging/z3prove/
runs two queries:Query 1 — coverage gap. ∃ bucket b: b is sensitive ∧ no trail covers b.
=== data-events-before (mgmt logging on, data events off) ===
trails observed: 1
- org-management-trail is_logging=true data_resource_patterns=0
buckets observed: 4
- company-frontend-prod environment=production data_classification=public sensitive=true
- customer-data-prod environment=production data_classification=pii sensitive=true
- financial-records environment=production data_classification=financial sensitive=true
- company-frontend-dev environment=development data_classification= sensitive=false
--- S3 Data Event Logging Gap ---
verdict: SAT
witness: arn:aws:s3:::company-frontend-prod
(sensitive=true, classification=public, environment=production,
data_event_coverage=NONE)
rationale: 3 of 3 sensitive buckets fall outside every trail's data_resources
Three of three sensitive buckets are uncovered. Z3 picks one as the witness. The prover doesn't care which; any unconvered sensitive bucket discharges the existential.
Query 2 — compound: write access without audit. This is the conjunction that makes the gap matter: write_admitted ∧ ¬covered. The write_admitted
half is discharged against an IAM policy with a prefix wildcard; the ¬covered
half is discharged against the trail's event selectors. The compound is the supply chain attack path Bybit took: modify production JavaScript, leave no CloudTrail record, $1.5B redirected.
Don't enable data events on every bucket. That's cost-prohibitive and probably useless. Enable data events on the buckets that contain PII, serve production frontends, or hold financial records. The remediated fixture is a one-statement change:
event_selectors:
- read_write_type: All
include_management_events: true
data_resources:
- type: AWS::S3::Object
values:
- arn:aws:s3:::company-frontend-prod/
- arn:aws:s3:::customer-data-prod/
- arn:aws:s3:::financial-records/
Three ARN prefixes. Z3 reports UNSAT on both queries:
=== data-events-after (data events scoped to sensitive buckets) ===
--- S3 Data Event Logging Gap ---
verdict: UNSAT
rationale: every sensitive bucket is covered by at least one trail's data_resources
--- Compound: Write Access + No Audit Trail ---
verdict: UNSAT
rationale: no logging gap → compound trivially UNSAT
The teaching beat: management events answer "is logging running?" Data events answer "is logging covering the buckets that matter?" The first question is one boolean. The second is a quantifier over a sensitivity-tagged bucket inventory and a coverage relation. Different question, different prover.
The CEL control above is correct as written. It catches the attacker who runs StopLogging
. The data-event question is the one a heuristic scanner can't ask — it requires reasoning over which sensitive buckets fall outside which trail's data_resources
patterns, across the join of two asset types. Z3 closes that gap.
The example at stave/examples/cloudtrail-stop-logging/ is a self-contained Go program that loads two fixture snapshots, runs pkg/stave.Apply, asserts that CTL.CLOUDTRAIL.STOP.DETECT.001 fires on the trail-stopped fixture and is silent on the running one, and exits zero when both assertions hold. Stave detects this pattern and 31 other H1-grounded scenarios from local AWS configuration snapshots, with no cloud credentials.