# I Built an Offline AWS Auditor. It Caught My Mistakes, Then Made Its Own.

> Source: <https://dev.to/okoroaforvic/i-built-an-offline-aws-auditor-it-caught-my-mistakes-then-made-its-own-2j25>
> Published: 2026-10-18 22:11:09+00:00

*This is a submission for the [Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle](https://dev.to/challenges/hacktoberfest-week2-2026-10-12)*

My final-year project was a serverless fintech backend on AWS: 48 Terraform resources and 10 IAM policies, all written by me, a cybersecurity student who thought he knew what he was doing.

I was the target user. A small team, no security engineer, permissions written under deadline. So before offering this to anyone else, I ran ZeroLeak on my own plan with the Wi-Fi off. It found things I'd missed, and it handed me two "fixes" that would have broken my own app. More on that below.

Would I pay for it? Not for the tool, which is free and open source. I'd pay for the reviewed report, because I know exactly how confidently wrong my own tool can be. I only know that because I was the customer first.

The people I actually want to help look a lot like the person who wrote that project. Meet Ada. She's the CTO of a six-person fintech startup. (She's made up, but if you've worked at a startup, you've met her.) Her team ships on AWS fast, and nobody's job is security. She knows their permissions are messy, and she can't say which role is the dangerous one.

She has three options: hire a security firm she can't afford, paste her AWS setup into a cloud chatbot and hope nobody minds, or ignore it until something breaks.

I just graduated in cybersecurity, and I want to be her fourth option. This post is about the tool I built to make that offer, and about the three times it told me everything was fine when it wasn't.

Picture an AWS account as an office building. IAM policies are the keycards: they decide who can open which doors. Most cloud breaches don't involve anyone breaking a door down. Someone simply holds a keycard that opens far more than it should. In a policy, a `*` is a master key.

**ZeroLeak AI** reviews those keycards entirely on a laptop. No cloud AI, no API keys, and no internet once it's set up. You give it one policy, or a whole Terraform plan (the blueprint a team uses to build their AWS setup), and two very different reviewers look at it:

The guard then re-checks the detective's suggestion. The report lists exactly which permissions the fix removed, and everything exports to a PDF Ada can hand to her team or her investors.

*Test 04 after codifying Rule 4: the guard now fails it, and Gemma explains why.*

Here's what I'd offer Ada:

My pilot price is **$100 for up to 10 policies**, with the first few reviews discounted in exchange for honest feedback. I don't have a paying client yet. This post is part of finding the first one.

The first `docker compose up` needs internet once to download the app and the model. After that, the whole audit, PDF included, runs with the network off. Telemetry is disabled (`gatherUsageStats = false`). You can test the privacy claim yourself by switching off your Wi-Fi.

*Page one of a Test 04 report. On this run the gate refused Gemma's fix, and the report says so up front (more on why below).*

**A deterministic rule gate (OPA / Conftest Rego) plus a local open-weight model (Google Gemma 2 2B)**

**🔒 No cloud endpoints. No API tokens. No vendor DPA. $0 cloud spend.**

| **~20 ms** rule gate per policy | **52.2 s** mean Gemma 2 2B on CPU | **0 bytes** leave the machine | **10** hand-crafted benchmark policies | **$0** cloud spend | 

[Architecture](https://github.com/vicGrey/zeroleak#-5-stage-pipeline-architecture) • [Discovery Loop](https://github.com/vicGrey/zeroleak#-the-discovery-loop-breaking-the-circular-verification-trap) • [Benchmark](https://github.com/vicGrey/zeroleak#-benchmark-10-hand-crafted-policies) • [Real Plan](https://github.com/vicGrey/zeroleak#-case-study-a-real-terraform-plan) • [Quickstart](https://github.com/vicGrey/zeroleak#-quickstart) • [Wi-Fi Off Test](https://github.com/vicGrey/zeroleak#%EF%B8%8F-the-wi-fi-off-test) • [Limits](https://github.com/vicGrey/zeroleak#%EF%B8%8F-engineering-integrity-and-boundaries)

IAM policies are a map of an organization's infrastructure: account IDs, role naming conventions, ARN hierarchies, bucket names. Auditing them with a cloud LLM API means sending that map to a third party.

For freelance cloud security consultants and DevSecOps contractors working under NDAs, and for regulated teams (HIPAA, PCI-DSS, SOC 2), that's often not allowed without a signed…

```
git clone https://github.com/vicGrey/zeroleak.git
cd zeroleak
docker compose up   # then open http://localhost:8501
```

MIT licensed. The baseline wildcard rules are adapted from my final-year project, [aws-serverless-zero-trust-governance](https://github.com/vicGrey/aws-serverless-zero-trust-governance). Everything else (the Gemma pipeline, the combination rule, the Terraform plan parser, and the reports) was built during the challenge week.

Gemma 2 2B runs locally in Ollama at temperature 0.1 (low randomness, so it gives near-identical answers on every run). Each policy gets one structured prompt that returns both the explanation and the fixed JSON, and a sanitizer repairs the JSON when the model cuts off on a long policy. Conftest/OPA runs the Rego v1 rules, Streamlit is the interface, WeasyPrint makes the PDFs, and Docker Compose runs the app next to an Ollama container that downloads `gemma2:2b` on first start. A small parser pulls every IAM policy out of `terraform show -json`.

I picked the 2B model on purpose. It's a ~1.6 GB download that runs in 3 to 4 GB of RAM on an ordinary laptop, and that's the kind of machine a freelancer actually owns. The cost is speed: **52.2 seconds per policy on average (39.0s to 67.3s)** on a CPU. Newer and bigger Gemma releases exist. The model is one tag in Ollama, so you can swap it, but I haven't tested a larger one here.

I tested it on 10 policies I wrote, each targeting a known IAM mistake:

| # | What the policy does wrong | Guard (rules) | Detective (Gemma) | 
|---|---|---|---|
| 01 | Nothing (clean control) | 0, correct | 2, false positive | 
| 02 | Wildcard action | 1 | 1 | 
| 03 | Wildcard resource | 1 | 1 | 
| 04 | PassRole + Lambda escalation | **0, missed** | **1, caught** | 
| 05 | Over-broad DynamoDB access | 2 | 1 | 
| 06 | Same risk, different JSON format | 2 | 2 | 
| 07 | `NotAction` loophole | 2 | 1 | 
| 08 | Wildcard `sts:AssumeRole` | 1 | 1 | 
| 09 | EC2 instance-profile escalation | 1 (wildcard only) | 2 (incl. escalation) | 
| 10 | Prompt injection hidden in a label | 2 | resisted | 

The guard caught almost everything, in milliseconds. The detective earned its 52 seconds on the two rows where the danger was a *combination* of permissions the guard had no rule for (and gave me my first false positive on row 01, flagging standard logging as a hazard).

Getting to a table I could trust was the hard part.

**1. The guard wasn't looking.** Test 04 lets someone create a Lambda function and hand it a more powerful role than their own. My first pipeline said: original policy, 0 violations; Gemma flags the danger and writes a fix; fixed policy, 0 violations. Two green checkmarks, "Remediated and Verified."

But the guard's checklist only covered master keys. It had no line for this combination, so it waved through the dangerous original *and* the fix. A broken fix would have passed just the same. Emma, a friend who freelances on Upwork, pointed out that the verification was circular.

The fix was a loop: when the detective finds something new, a human writes it into the guard's checklist.

``` python
import rego.v1

deny contains msg if {
    some s1 in get_statements
    s1.Effect == "Allow"
    has_action(s1, "iam:PassRole")
    some s2 in get_statements
    s2.Effect == "Allow"
    has_action(s2, "lambda:CreateFunction")
    msg := "[passrole_lambda_escalation] PassRole with lambda:CreateFunction allows privilege escalation."
}

# helpers in policies/iam_policy.rego
```

Now the original fails (1 violation), and in my logged run the fix passed (0). The green check finally means something.

It also works against me. On a later run, Gemma kept both actions but scoped them to a single function and a single role, `role/role-name`, a placeholder ZeroLeak flags before deployment. The gate still refused it (RULE GATE: FAILED, 1 remaining), because my rule checks which actions are present, not which resources they touch. Whether that fix is safe depends on what `role-name` can do. That's a call for a human, not for the rule or the model.

Test 09 has the same gap until someone writes its rule too: nothing in my ruleset tests for PassRole + RunInstances, so any "verified" there only means "passed the wildcard rules."

**2. My own real project.** I ran ZeroLeak on that same plan, with the Wi-Fi off. It took **102.29 seconds**. Of the 10 policies, 5 passed, 2 failed, and 3 couldn't be read at all.

Those 3 pointed at things that don't exist until deployment (a bucket, a database table, a login pool), so they show up as "unknown" in a plan. A scanner that treats unknown as "nothing to flag" calls them clean. ZeroLeak now lists them as **NOT EVALUATED (KNOWN AFTER APPLY)**, so the client gets a to-do list instead of false comfort.

*My final-year project's plan: 48 resources, 10 policies, 3 not evaluated at plan time. Runtime varies by machine and model state; this rerun took 88s, while the logged run took 102.29s.*

The 2 failures were worse. Gemma rewrote both, the guard passed both rewrites, and both were wrong. One removed `logs:PutLogEvents` from a logging role, the one permission it existed for. That's like fixing a cashier's keycard by taking away access to the till. The other scoped `config:PutEvaluations` to a fake example ARN, but that action [doesn't support resource-level permissions](https://docs.aws.amazon.com/service-authorization/latest/reference/list_config.html), so `*` was actually correct. The guard passed both because it only looked for a literal `*`, and any other text satisfied it.

The report badge used to say "RE-VERIFICATION: PASSED". It now says **"RULE GATE: PASSED (Workload & ARN validity not checked)"**, which is the honest version.

**3. The final pressure test.** Three fresh policies, none used in development.

First, Gemma flagged a policy combining `iam:PassRole`, `iam:CreateAccessKey` and `iam:PutUserPolicy`. That one is more of a sanity check than a discovery: either of the last two is enough to escalate privileges on its own, and standard scanners like Cloudsplaining already flag them.

Second, `arn:aws:s3:::*/*` (every file in every bucket) slipped past the baseline guard because it isn't a bare `*`, so the rules now catch wildcard ARNs too.

Third, on a perfectly fine read-only policy, Gemma confidently added a Condition: `"s3:BucketOwner": "arn:aws:iam::123456789012:user/my-user"`. In AWS S3, a valid key is `s3:ResourceAccount` (which takes a 12-digit account ID, not an IAM user ARN), not `s3:BucketOwner`. An IAM condition on a key AWS doesn't define never matches, so the Allow would silently fail and reads would be denied. The guard passed it because it only looked for wildcards.

So ZeroLeak now shows "preserved intact" only when the guard and Gemma agree. When they don't, the report says **Manual Review Recommended** and labels any fix as advisory (no copy-paste fix without manual verification).

*So, why would Ada pay a person? Because the tool can be confidently wrong in ways that look right. ZeroLeak does the private, tireless first pass, and a human who knows what each permission is for does the part that matters.*

The prompt for this challenge asks why open innovation matters for what I built. Here's the specific answer, not the general one.

For Ada, three reasons:

A big hosted model would probably be faster and smarter than a 2B model on a laptop. I'm trading that for privacy and cost, and for a rule gate that doesn't depend on the model being right.

`Resource` and require an `iam:PassedToService` condition.
I started this project thinking the AI was the product. I'm finishing it knowing the product is the judgment: knowing when a green checkmark was actually looking for the problem.

**Best use of Gemma:** Gemma 2 2B runs fully locally through Ollama as the detective in the pipeline. It flagged two risky permission combinations (tests 04 and 09) that the rule gate had no rule for. Every audit runs behind a deterministic rule gate, and because the model runs locally, no client data leaves the machine.