First Action After Compromise: Blind the Audit

A developer detailed how attackers can blind AWS audit trails by calling `cloudtrail stop-logging`, which halts event delivery while leaving the trail's configuration looking healthy. The post highlights that detection requires checking `get-trail-status` for `IsLogging: false`, and recommends monitoring this field as a first-class asset property to close the audit blind window.

✓ Human-authored analysis; AI used for formatting and proofreading. If you're an attacker who's just landed in an AWS account, the most expensive thing about your future is detection. Every API call you make leaves a trail in CloudTrail, gets forwarded to a SIEM, becomes evidence the team uses to boot you out and reconstruct what you touched. The cost of your campaign rises linearly with the number of events on record. So you make one call first: aws cloudtrail stop-logging --name org-audit-trail The trail still exists. The console still shows it. aws cloudtrail describe-trails still returns it. The S3 bucket is still configured as the destination. To anyone glancing at the configuration, the audit infrastructure looks healthy. Only aws cloudtrail get-trail-status reveals that IsLogging is false . New events stop landing in the bucket. The audit clock pauses. Whatever the attacker does next happens unobserved, and remains unobserved until somebody notices. This pattern is established enough to have its own MITRE ATT&CK identifier: T1562.008 Disable or Modify Cloud Logs . It appears in cloud incident write-ups across Mandiant's catalogue, AWS Security Bulletin guidance, and pen-test reports against organisations whose detection budget went into SIEM and not into CloudTrail-config monitoring. A stopped trail is structurally healthy. Every other configuration field is normal: { "Name": "org-audit-trail", "S3BucketName": "acme-cloudtrail-archive", "IsMultiRegionTrail": true, "IsOrganizationTrail": true, "EventSelectors": ... , "InsightSelectors": , "KmsKeyId": "arn:aws:kms:us-east-1:...", "LogFileValidationEnabled": true } The only field that flips is the one get-trail-status returns separately: { "IsLogging": false, "LatestDeliveryTime": "2026-01-01T00:00:00Z", "StartLoggingTime": "2025-09-15T00:00:00Z", "StopLoggingTime": "2026-01-01T00:00:00Z" } Two timestamps and a boolean. That's the entire signal. Detection requires reading the right API — GetTrail and DescribeTrails both return the configuration but not the running state. The audit blind window is the difference between a 30-day incident response and a 6-week one. Concrete things the attacker does inside that window: AdministratorAccess , an access key for offline use, a Lambda function with a privileged execution role that the attacker invokes via API Gateway.None of these leave a CloudTrail event. All of them fail audit review because there is no audit . The investigator's first question: "what did this principal do between Jan 1 and Jan 8?" has no answer. Operators who built the CloudTrail configuration usually checked their work. The bucket is encrypted, the trail is multi-region, log file validation is on, the bucket policy denies external principals. Every static configuration field passes audit review. The mistake is treating "trail exists and is configured" as equivalent to "trail is running." They are not the same. The IAM permission that creates the trail cloudtrail:CreateTrail is different from the permission that toggles its state cloudtrail:StopLogging , cloudtrail:StartLogging . Roles permitted to do one routinely have permission to do the other — incident-response runbooks frequently include "start logging if it stopped" without restricting the inverse. Every CloudTrail trail must have IsLogging: true , and at least one trail in the account must be multi-region. In Stave's observation schema, the trail's running state is a first-class field on the asset: { "id": "arn:aws:cloudtrail:us-east-1:111122223333:trail/org-audit-trail", "type": "aws cloudtrail trail", "properties": { "trail": { "name": "org-audit-trail", "is logging": false, "is multi region trail": true, "stopped at": "2026-01-01T00:00:00Z", "stopped by": "arn:aws:sts::111122223333:assumed-role/incident-actor/session" } } } The collector that produces this observation calls GetTrailStatus for every trail, picks up the IsLogging boolean, and surfaces it directly. The diagnostic fields stopped at , stopped by carry forensic context the responder uses next; the predicate only reads the booleans. id: CTL.CLOUDTRAIL.STOP.DETECT.001 name: CloudTrail Trails Must Be Actively Logging in All Regions severity: critical unsafe predicate: any: - field: properties.trail.is logging op: eq value: false - field: properties.trail.is multi region trail op: eq value: false any — either the trail is stopped, or the trail is single-region which leaves the other regions blind even when the configured one is running . Severity is critical because the trail being stopped is the configuration that determines whether every other finding in the account is investigable; the impact multiplies across the rest of the security posture. This is a presence check , not a reachability question. The collector observes two booleans; the predicate is a two-leaf disjunction. There is no search space. There is no witness to enumerate. Z3 earns its complexity when the question is "what specific request does this configuration admit?" — that's reachability over a permission set. The question here is "is the audit infrastructure running?" — a flat read on a status field. A self-contained example is in the repo: stave/examples/cloudtrail-stop-logging/ : go run ./examples/cloudtrail-stop-logging before Captured stdout: === before trail stopped === status: NON COMPLIANT total assets=1 violations=1 CTL.CLOUDTRAIL.STOP.DETECT.001 fired on 1 asset s : - arn:aws:cloudtrail:us-east-1:111122223333:trail/org-audit-trail severity=critical exposure score=100.00 assertion: fires=true expected ✓ After aws cloudtrail start-logging runs: === after trail re-started === status: COMPLIANT total assets=1 violations=0 CTL.CLOUDTRAIL.STOP.DETECT.001: no findings assertion: fires=false expected ✓ The before-fixture carries stopped by — assumed-role/incident-actor/session — which is the forensic breadcrumb the responder uses next. CloudTrail itself recorded the StopLogging event before stopping; EventBridge can route that event to the SOC even after the trail goes silent, so the first unauthorised stop is detectable. The risk is detection latency. A six-hour latency is enough window for a determined attacker. The remediation is a single command: aws cloudtrail start-logging --name org-audit-trail A follow-up question every IR runbook should answer: what was logged in the audit-blind window? The answer is "nothing through this trail." Other data sources fill in the gap to varying degrees: None of these are a substitute for the trail. Together they shorten the audit-blind window's effective length. Three layers, in order of leverage: Service Control Policy denying StopLogging for any non-bootstrap principal. The strongest layer: { "Sid": "DenyCloudTrailStopLogging", "Effect": "Deny", "Action": "cloudtrail:StopLogging", "cloudtrail:DeleteTrail", "cloudtrail:UpdateTrail", "cloudtrail:PutEventSelectors" , "Resource": " ", "Condition": { "ArnNotLike": { "aws:PrincipalArn": "arn:aws:iam:: :role/CloudTrailAdministrator" } } } The deny applies to every principal except the explicit admin role. An attacker who lands on a compromised Lambda or compromised user has no path to StopLogging ; the SCP rejects the call before it reaches CloudTrail's API. EventBridge alarm on StopLogging events. The trail records the StopLogging event before going silent. Route that specific event to the SOC channel with a high-priority page: { "source": "aws.cloudtrail" , "detail-type": "AWS API Call via CloudTrail" , "detail": { "eventSource": "cloudtrail.amazonaws.com" , "eventName": "StopLogging", "DeleteTrail" } } The alarm catches the first unauthorised stop. Every moment the attacker waits is a moment the page rings. stave apply in CI against the post-deploy observation snapshot. The example shipped with this article is the template. PRs that cause a trail's is logging to go false produce exit code 3. cloudtrail:StopLogging , cloudtrail:DeleteTrail , cloudtrail:UpdateTrail , cloudtrail:PutEventSelectors for any non-bootstrap principal StopLogging , DeleteTrail events from the management trail and pages the on-call SOC stave apply runs in CI against post-deploy observations; PRs that introduce a stopped trail failThe attacker's first action is StopLogging . The defender's first action should be making StopLogging hard to call without paging the on-call. The window between the call and the page is the budget the attacker has to spend; closing the window to seconds is how the rest of the security posture stays investigable. The control above asks one question: is the trail running? It's a presence check on is logging and is multi region trail . The CEL predicate is two booleans ANDed together. That's the headline-grabbing case where the attacker who calls StopLogging to evade detection. The Bybit / Safe{WALLET} writeup March 2025 makes a quieter point: "modifying objects in an S3 bucket can often be hard to detect because, by default, CloudTrail logging for S3 object activity is disabled." That is a different gap. Management events log CreateBucket , PutBucketPolicy , DeleteBucket . Data events log GetObject , PutObject , DeleteObject . Most production trails enable management events and stop there — data events are high-volume, expensive, and opt-in. The trail in data-events-before/ is a normal production CloudTrail: trail name: org-management-trail is logging: true is multi region trail: true include global service events: true log file validation enabled: true event selectors: - read write type: All include management events: true data resources: ← empty Five booleans say "logging is healthy." The compliance dashboard shows green. The CEL control above CTL.CLOUDTRAIL.STOP.DETECT.001 reports no findings — because, by its definition, there is nothing wrong: is logging=true and is multi region trail=true . The check passes. But the account contains three production buckets — company-frontend-prod serves JavaScript via CloudFront , customer-data-prod PII , and financial-records financial data — and none of them appear in any trail's data resources . An attacker who modifies app.js in the frontend bucket leaves no CloudTrail record. The security team's dashboard stays green. The earlier section "Why Z3 Doesn't Help Here" is honest about the original control: presence checks have no search space, no witnesses to enumerate. CEL is the right shape. The data-events question is different. It asks: is there a sensitive bucket whose object operations are not covered by any trail's data resources? That has the shape Z3 is built for — quantifier over buckets, predicate combining tag classification and trail coverage. The prover at examples/cloudtrail-stop-logging/z3prove/ runs two queries: Query 1 — coverage gap. ∃ bucket b : b is sensitive ∧ no trail covers b . === data-events-before mgmt logging on, data events off === trails observed: 1 - org-management-trail is logging=true data resource patterns=0 buckets observed: 4 - company-frontend-prod environment=production data classification=public sensitive=true - customer-data-prod environment=production data classification=pii sensitive=true - financial-records environment=production data classification=financial sensitive=true - company-frontend-dev environment=development data classification= sensitive=false --- S3 Data Event Logging Gap --- verdict: SAT witness: arn:aws:s3:::company-frontend-prod sensitive=true, classification=public, environment=production, data event coverage=NONE rationale: 3 of 3 sensitive buckets fall outside every trail's data resources Three of three sensitive buckets are uncovered. Z3 picks one as the witness. The prover doesn't care which; any unconvered sensitive bucket discharges the existential. Query 2 — compound: write access without audit. This is the conjunction that makes the gap matter: write admitted ∧ ¬covered . The write admitted half is discharged against an IAM policy with a prefix wildcard; the ¬covered half is discharged against the trail's event selectors. The compound is the supply chain attack path Bybit took: modify production JavaScript, leave no CloudTrail record, $1.5B redirected. Don't enable data events on every bucket. That's cost-prohibitive and probably useless. Enable data events on the buckets that contain PII, serve production frontends, or hold financial records. The remediated fixture is a one-statement change: event selectors: - read write type: All include management events: true data resources: - type: AWS::S3::Object values: - arn:aws:s3:::company-frontend-prod/ - arn:aws:s3:::customer-data-prod/ - arn:aws:s3:::financial-records/ Three ARN prefixes. Z3 reports UNSAT on both queries: === data-events-after data events scoped to sensitive buckets === --- S3 Data Event Logging Gap --- verdict: UNSAT rationale: every sensitive bucket is covered by at least one trail's data resources --- Compound: Write Access + No Audit Trail --- verdict: UNSAT rationale: no logging gap → compound trivially UNSAT The teaching beat: management events answer "is logging running?" Data events answer "is logging covering the buckets that matter? " The first question is one boolean. The second is a quantifier over a sensitivity-tagged bucket inventory and a coverage relation. Different question, different prover. The CEL control above is correct as written. It catches the attacker who runs StopLogging . The data-event question is the one a heuristic scanner can't ask — it requires reasoning over which sensitive buckets fall outside which trail's data resources patterns, across the join of two asset types. Z3 closes that gap. The example at stave/examples/cloudtrail-stop-logging/ is a self-contained Go program that loads two fixture snapshots, runs pkg/stave.Apply, asserts that CTL.CLOUDTRAIL.STOP.DETECT.001 fires on the trail-stopped fixture and is silent on the running one, and exits zero when both assertions hold. Stave detects this pattern and 31 other H1-grounded scenarios from local AWS configuration snapshots, with no cloud credentials.