{"slug": "logspecter-schema-aware-secret-scanner-for-cloud-logs", "title": "LogSpecter – Schema-aware secret scanner for cloud logs", "summary": "LogSpecter, a schema-aware secret scanner for cloud logs, detects leaked API keys by combining regex, Shannon entropy, and JSON-structure analysis to identify the responsible identity, API call, and JSON path, achieving throughput of up to 55.0 MiB/s with a flat memory ceiling of about 43 MiB on 1 GiB files. The tool, benchmarked on 8 cores with CPython 3.13, supports AWS CloudTrail, GCP Cloud Logging, Kubernetes audit logs, Azure Activity Log, and Elastic Common Schema, and is designed to reduce false positives from long but low-diversity strings.", "body_md": "**Schema-aware secret scanner for cloud logs.** Regex finds candidates; Shannon entropy and a\nheuristic layer decide whether they are real keys; a JSON-structure layer tells you *who* leaked\n*what* through *which field*. Streams tens of gigabytes with a flat memory ceiling.\n\n``` bash\n$ logspecter scan cloudtrail-2026-08-30.json.gz --stats\n\n  CRITICAL   openai-api-key   AWS IAM User (Alice) → action: AssumeRole\n                              → requestParameters.headers.Authorization\n                              cloudtrail-2026-08-30.json.gz:81421 @byte 24118904\n                              sk-p******** kAyS (len=64)   conf 1.00 / H 5.19\n```\n\nNot \"a suspicious string on line 800\". The actual identity, the actual API call, the actual JSON path.\n\nMost log scanners are a pile of regexes. That fails in production for three reasons, and LogSpecter attacks each one directly.\n\nA rule like \"base64 string of 32+ chars\" fires on every pagination cursor, every trace ID, every base64-encoded JSON blob in your logs. LogSpecter runs a second stage on every candidate:\n\n| Check | What it kills |\n|---|---|\n| Shannon entropy + charset-normalised entropy | low-diversity strings that merely look long |\nCharset coverage (unique chars ÷ achievable unique chars) |\n`aaaa…` , `ababab…` , and it does not penalise 64-char hex hashes |\n| Repeat / sequential runs | `xxxxxxxx` , `abcdefgh` , `987654321` |\n| Natural-language likeness (bigram + vowel ratio) | `SpringBootApplicationConfigurationLoader` |\n| Base64 decode-back | strings that decode to readable text or JSON — encoded data, not keys |\n| Placeholder & vendor-doc detection | `AKIAIOSFODNN7EXAMPLE` , `changeme` , `<your-api-key>` , `${VAR}` |\n| Keyword proximity | a 32-char hex blob only counts near `key` /`secret` /`hmac` |\n\nEvery decision is recorded on the finding, so you can audit *why* something was reported or\ndropped:\n\n```\n\"evidence\": [\"entropy=5.61/6.00(base64url)\", \"charset_coverage=0.85\",\n             \"non-linguistic\", \"keyword-nearby\", \"also-matched:authorization-header-bearer\"]\n```\n\nEntropy thresholds are **per rule**, not global. `Authorization: Basic`\n\nintentionally disables the\ndecode-back check (Basic auth *is* base64 text); database URLs relax entropy to catch weak human\npasswords while relying on the `scheme://user:pass@host`\n\nshape for precision.\n\nLogSpecter parses records with [orjson](https://github.com/ijl/orjson) (Rust-backed) and\nrecognises the schema it is looking at:\n\n- AWS CloudTrail — IAM identity,\n`eventName`\n\n, region, source IP, account - GCP Cloud Logging —\n`principalEmail`\n\n,`methodName`\n\n, resource, project - Kubernetes audit — user, verb,\n`objectRef`\n\n- Azure Activity Log — identity claims,\n`operationName`\n\n, result - Elastic Common Schema, Logback / Log4j2 JSON (including MDC)\n\nNested JSON-inside-a-JSON-string is expanded too, because `requestParameters`\n\nand MDC fields are\nfull of it. You get a precise path (`protoPayload.request.credential`\n\n) instead of a line number,\nplus the actor and action needed to actually respond to the incident.\n\nThe input layer plans **line-aligned byte ranges** without reading the file (a few seeks and small\ntail reads), then hands one range per worker. Compressed files and stdin go through a\nproducer/consumer path with a bounded submission window. Resident memory is a function of chunk\nsize, never of file size:\n\n| Scan | Wall time | Throughput | Main RSS | Peak worker RSS |\n|---|---|---|---|---|\n| 256 MiB, 1 worker | 20.7 s | 12.4 MiB/s | 45 MiB | — |\n| 256 MiB, 8 workers | 5.1 s | 49.9 MiB/s | 42 MiB | 34 MiB |\n| 1 GiB, 8 workers | 18.6 s | 55.0 MiB/s | 43 MiB | 35 MiB |\n\nSame memory for 1 GiB as for 256 MiB. Measured with `GetProcessMemoryInfo`\n\n/\n`/proc/self/status`\n\nand reported by `--stats`\n\n— not a claim, an output field.\nNumbers from `logspecter benchmark`\n\non 8 cores / Windows / CPython 3.13, scanning a synthetic mix\nof CloudTrail records, application logs, and high-entropy-but-harmless noise. Reproduce with\n`logspecter benchmark --size 1GB -j 8`\n\n; pure-Python throughput is CPU-bound, so expect it to track\nyour single-core speed times the worker count.\n\n**How it gets there in pure Python** (the part that took the most work)\n\nA naive \"for each line, for each rule, run the regex\" loop benchmarks at **3.4 MiB/s**. Three\nchanges took it to 12.4 MiB/s per core:\n\n-\n**Bytes end to end.** Rules compile to`bytes`\n\npatterns, so there is no per-chunk decode, match offsets*are*file offsets, and`\\b`\n\n/`\\w`\n\nget predictable ASCII semantics. -\n**Occurrence-driven scanning instead of line iteration.** Each rule's regex AST is statically analysed for literals that*must*appear in any match (`\\b((?:AKIA|ASIA)[A-Z0-9]{16})\\b`\n\n→`AKIA|ASIA`\n\n). Those literals are located with`bytes.find`\n\n(~3.7 GiB/s) and the regex runs only on the lines that contain them. A monotonic cursor per literal means an absent literal is scanned once, not once per line — getting this wrong cost a 50× slowdown before it was fixed. -\n**Anchored matching.** The analyser also computes how many bytes of the match may precede the literal.`_live_`\n\nin`(?:sk|rk)_live_…`\n\nis always at offset 2, so instead of`search()`\n\n-ing a whole line the scanner tries`match()`\n\nat one exact position. This is what removed most of the remaining cost: \"literal present but regex does not match\" is the single most common case in real logs (every CloudTrail S3 record contains`\"key\":`\n\n).\n\nLiterals are merged into prefix-tree regexes (`key|keystore|kms`\n\n→ `k(?:ey(?:store)?|ms)`\n\n) so\nCPython's `INFO`\n\nfirst-character-set optimisation applies: an 8 MiB buffer with no match at all is\nrejected in 2.5 ms. That gives the fast path for clean or binary data.\n\nThe analyser only emits a literal when it can *prove* it is mandatory; otherwise the rule falls\nback to a full scan. `tests/test_prefilter.py`\n\nasserts, for every built-in rule, that the prefilter\nnever rejects an input the regex would have matched, and that anchor windows always contain the\nreal match offset.\n\n```\npip install logspecter\n```\n\nFrom source:\n\n```\ngit clone https://github.com/logspecter/logspecter\ncd logspecter\npip install -e \".[dev]\"\n```\n\nRequires Python 3.10+. Runtime dependencies: `typer`\n\n, `rich`\n\n, `PyYAML`\n\n, `orjson`\n\n.\n\n```\n# a file, a directory, a compressed archive\nlogspecter scan /var/log/app.log\nlogspecter scan /var/log/ --recursive\nlogspecter scan cloudtrail-2026-08-30.json.gz\n\n# a pipe\nkubectl logs deploy/api --since=1h | logspecter scan -\naws logs tail /aws/lambda/api --format short | logspecter scan -\n\n# CI gate: fail only on new critical leaks\nlogspecter scan ./logs --baseline .logspecter-baseline.json --fail-on critical\n\n# machine-readable output\nlogspecter scan ./logs -f json -o findings.json\nlogspecter scan ./logs -f csv  -o soc2-evidence.csv\nlogspecter scan ./logs -f sarif -o results.sarif   # GitHub code scanning\n```\n\nExit codes: `0`\n\nclean, `1`\n\nfindings at or above `--fail-on`\n\n(default `high`\n\n), `2`\n\nbad input.\n\n| Flag | Effect |\n|---|---|\n`-j, --workers N` |\nprocesses; default `min(8, cpu)` , `1` disables multiprocessing |\n`--chunk-size 4MB` |\nthe memory knob — resident data ≈ 2 × chunk × workers |\n`--min-entropy 4.5` |\nraise the global entropy floor (precision over recall) |\n`--min-confidence 0.8` |\ndrop low-confidence findings |\n`--aggressive` |\nenable noisy entropy-only rules (recall over precision) |\n`--pack aws --tag github` |\nnarrow the rule set |\n`--no-structured` |\nskip JSON parsing entirely; fastest, loses cloud context |\n`--show-secrets` |\nprint plaintext (off by default — reports are redacted) |\n`--stats` |\nthroughput, memory, and the full noise-reduction breakdown |\n\n```\nlogspecter rules list                        # 64 built-in rules across 7 packs\nlogspecter rules show aws-secret-access-key  # pattern, entropy gate, prefilter\nlogspecter rules validate ./my-rules.yaml    # lint custom rules\nlogspecter selftest                          # 64 positive + 25 negative samples\nlogspecter benchmark --size 1GB -j 8         # throughput and memory on your box\n```\n\nRules are plain YAML. A rule with the same `id`\n\nas a built-in one overrides it, which is the\nrecommended way to retune thresholds for your environment.\n\n```\nversion: 1\npack: acme\n\nrules:\n  - id: acme-internal-token\n    name: ACME Internal Service Token\n    severity: critical\n    confidence: high\n    pattern: '\\bacme_(?:live|prod)_([A-Za-z0-9]{40})\\b'\n    capture: 1\n    tags: [acme, internal]\n    entropy:\n      min_entropy: 4.4\n      min_normalized: 0.72        # entropy ÷ log2(charset size)\n      min_length: 40\n      min_charset_coverage: 0.6   # unique chars ÷ achievable unique chars\n      reject_encoded_text: true\n\n  - id: acme-mdc-secret\n    name: Secret in ACME MDC field\n    severity: high\n    pattern: '\\A\\s*(\\S{12,4096})\\s*\\Z'\n    capture: 1\n    json_keys: [acme_token, acme_signature]   # only applied to these JSON keys\n    entropy:\n      min_entropy: 3.5\nlogspecter rules validate acme.yaml\nlogspecter scan ./logs --rules acme.yaml\n```\n\nFull field reference: [ docs/rules.md](/Jeffy123-zhu/logspecter/blob/main/docs/rules.md).\n\n``` python\nfrom logspecter import engine\nfrom logspecter.rules import load_ruleset\nfrom logspecter.scanner import ScanOptions\n\nconfig = engine.ScanConfig(ruleset=load_ruleset(), options=ScanOptions())\nresult = engine.scan([\"/var/log/app.log\"], config, workers=4)\n\nfor group in result.groups:\n    f = group.representative\n    print(f.severity.value, f.rule_id, f.context_summary(), f\"×{group.occurrences}\")\n\nprint(result.stats.throughput_mb_s, result.stats.peak_rss_max_process)\n```\n\n`logspecter selftest`\n\nruns the bundled corpus with **every** rule enabled:\n\n```\n检出率 64/64  ·  负样本零误报 25/25\n```\n\n64 positive samples (one per rule, generated from a fixed seed — no real credentials in the repo)\nand 25 negative samples drawn from the shapes that actually fool regex-only scanners: UUID request\nIDs, git SHAs, base64-encoded JSON cursors, camel-case class names, template placeholders,\n`AKIAIOSFODNN7EXAMPLE`\n\n, ISO timestamps, CSS colours, service-account token paths, and unkeyed\nSHA-256 digests. Each of these is a distinct rejection reason in the entropy layer, and each is a\nregression test.\n\n`--stats`\n\nreports the funnel on your own data, so the numbers are yours rather than ours:\n\n```\n降噪  正则候选 1,245 → 熵值/上下文层拦下 16 条（1.3%）\nlogspecter/\n├── ingest.py      byte-range planning, mmap window reads, gz/bz2/xz/stdin streaming\n├── engine.py      chunk scheduling, bounded-window multiprocessing, line-number prefix sums,\n│                  fingerprint aggregation\n├── prefilter.py   regex AST → mandatory literals + prefix widths, trie merging, screen tree\n├── scanner.py     the detection pipeline\n├── entropy.py     Shannon entropy and the heuristic gate\n├── rules.py       YAML loading, validation, bytes compilation\n├── cloud.py       cloud log schema detection and context extraction\n├── structured.py  orjson parsing and JSON flattening\n├── report/        Rich console, JSON, CSV, SARIF\n└── rules/*.yaml   built-in rule packs\n```\n\nTwo details that are easy to get wrong and are worth knowing about:\n\n**Line numbers under multiprocessing.** Workers only know their offset in the file, so they report\na chunk-local line number plus that chunk's total line count. The parent computes a prefix sum\nover chunk line counts and rewrites the findings. No pre-pass over the file, exact `file:line`\n\n.\n\n**Report readability.** One leaked key repeated 50 000 times is one finding, not 50 000 rows.\nFindings are aggregated by `SHA-256(rule_id ‖ secret)[:16]`\n\nwith an occurrence count and a few\nsample locations. Overlapping rules on the same value collapse to the most specific one\n(`openai-api-key`\n\nbeats `authorization-header-bearer`\n\nbeats `sensitive-json-key-value`\n\n), with the\nothers preserved in the evidence chain.\n\nReports are redacted by default: masked value plus real length, never plaintext, unless you pass\n`--show-secrets`\n\n. Baseline files store only fingerprints. The scanner makes no network calls.\n\nIf you find a vulnerability, please open a private security advisory rather than a public issue.\n\n```\npip install -e \".[dev]\"\npytest              # 360 tests\nruff check .\nlogspecter selftest\n```\n\nNew rules need a positive sample in `src/logspecter/samples.py`\n\n; the test suite fails if any rule\nlacks one, and `tests/test_prefilter.py`\n\nwill tell you if your pattern defeats the prefilter.\n\nChinese documentation: [README.zh-CN.md](/Jeffy123-zhu/logspecter/blob/main/README.zh-CN.md).\n\nApache-2.0.", "url": "https://wpnews.pro/news/logspecter-schema-aware-secret-scanner-for-cloud-logs", "canonical_source": "https://github.com/Jeffy123-zhu/logspecter", "published_at": "2026-09-03 00:06:01+00:00", "updated_at": "2026-09-03 00:21:55.529173+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools"], "entities": ["LogSpecter", "AWS CloudTrail", "GCP Cloud Logging", "Kubernetes", "Azure Activity Log", "Elastic Common Schema", "orjson", "CPython 3.13"], "alternates": {"html": "https://wpnews.pro/news/logspecter-schema-aware-secret-scanner-for-cloud-logs", "markdown": "https://wpnews.pro/news/logspecter-schema-aware-secret-scanner-for-cloud-logs.md", "text": "https://wpnews.pro/news/logspecter-schema-aware-secret-scanner-for-cloud-logs.txt", "jsonld": "https://wpnews.pro/news/logspecter-schema-aware-secret-scanner-for-cloud-logs.jsonld"}}