A client sends you an AWR report under NDA. I built a second opinion that never leaves your laptop

A developer built Second Opinion, an open-source command-line tool and Python library that performs first-pass triage of Oracle AWR and Statspack performance reports entirely on a local machine using the Gemma 4 open model via Ollama, so NDA-covered client data never leaves the laptop. On 18 held-out lab reports with injected faults, the tool identified the root cause in 17 of 18 cases, versus 10 of 18 for Gemma 4 given the raw report and 7 of 9 for Oracle's ADDM on the AWR subset, with every quoted figure verified against the source report. A fine-tuned Qwen3-8B variant ranked causes correctly in 16 of 18 runs, up from 11 of 18 untuned.

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle https://dev.to/challenges/hacktoberfest-week2-2026-10-12 Every Oracle DBA who freelances knows this email. "The system was slow yesterday between 2 and 4. AWR attached. Can you take a look?" Attached is a 200-page HTML report covering every wait event, every statement and every segment the database touched. The client is paying by the hour, and the report is covered by an NDA. It holds their SQL text, table names and host names. The first hour on a report like that is always the same triage. Is the problem parsing, I/O, locking, redo, or one bad statement? Load Profile, Top Events, SQL ordered by gets, back to the events, then the instance activity stats. It's skilled work, but it's routine. And the obvious shortcut is off the table: you can't paste a client's AWR report into a hosted chatbot. So I built Second Opinion . It's a first-pass AWR and Statspack reviewer that runs entirely on your own machine, with an open model. Once the model is loaded, it takes under a second on my desktop GPU to rank the likely root causes, show the numbers behind each one, and draft a short note for the client. Every number in that note is checked against the report before you see it. In short: on 18 held-out lab reports with faults I injected myself, Second Opinion named the cause in 17 of 18 . Oracle's own ADDM got 7/9 of the AWR ones, and Standard Edition can't run ADDM at all. Gemma 4 given the raw report got 10/18 . Neither model ever quoted a number that wasn't in its input. I also fine-tuned Qwen3-8B on Tinker to rank the causes with no rules at all. Untuned it got 11/18; tuned, 16/18, with the top cause right on every run, and it runs locally in Ollama. Second Opinion is a command-line tool and a small Python library: bash $ second-opinion awrrpt 1 17 18.html AWR, as awrrpt.sql writes it $ second-opinion sp 10 11.lst Statspack, any release $ second-opinion pasted.txt the sections a colleague pasted into a ticket It reads the report and runs the checks a DBA runs in their head. Gemma 4 then writes up what those checks found: Statspack report · ORCLCDB 12.2.0.1.0 · 13-Oct-26 16:58:47 to 17:05:47 7.0 min, DB time 27.71 min High Hard Parse Rate Indicates Inefficient SQL Usage Likely causes, most important first: Hard parsing literal SQL, no binds Note for the client The Statspack report indicates a significant performance issue primarily driven by hard parsing. The evidence shows that 100.0% of parses are hard parses, with 8811.8 hard parses per second. This high rate is contributing to the high wait time on library cache: mutex X, which accounts for 39.5% of DB time. To address this, first, review the application code to ensure all SQL statements utilize bind variables instead of literals. ... Findings computed from the report, not by the model - Hard parsing literal SQL, no binds score 1.0 : Most parses are hard parses... hard parses per sec=8811.8, parses per sec=8812.8, hard parse pct=100.0, shared pool event=library cache: mutex X, shared pool event pct db time=39.5 Numbers checked: 3; every number in the note was found in the report. Model: gemma4:e4b. 0.6 s, all local. That's a real run on one of the held-out reports. It isn't perfect: the note's second suggestion, about indexing, has nothing to do with hard parsing. That's exactly why the output is a draft for a DBA, not something to send unread. Who pays for this: A two-minute narrated walkthrough: a real report, the review, the Sentry traces, and the Tinker before/after. Second Opinion gives a first-pass review of an Oracle AWR or Statspack report. It runs on your own machine with a local open model, Gemma 4 via Ollama, so the report never leaves the box. This is a real run on one of the held-out lab reports reports/test/hard parse-statspack- : bash $ second-opinion statspack.txt Second Opinion: statspack.txt Statspack report · ORCLCDB 12.2.0.1.0 · 13-Oct-26 16:58:47 to 13-Oct-26 17:05:47 7.0 min, DB time 27.71 min High Hard Parse Rate Indicates Inefficient SQL Usage Likely causes, most important first: Hard parsing literal SQL, no binds Note for the client The Statspack report indicates a significant performance issue related to SQL execution. The primary finding is a high rate of hard parses, with 100.0% of parses being hard parses. This is evidenced by the high hard parses per sec of 8811.8 and the associated wait event library cache: mutex X consuming 39.5% of … My first attempt was the obvious one: give Gemma 4 the AWR report and ask what's wrong. An AWR report for an 18-second window is about 18,000 tokens of text, and a real one-hour report is far bigger. On my held-out set, Gemma reading the raw report found the cause in 10 of 18. It reads alarming-looking wait events as problems on healthy systems, and it misses every full table scan, because the evidence for a full scan is spread across three sections SQL ordered by gets, instance activity, segments by table scans . So the model never reads the report. The work is split three ways: awr.py , statspack.py , text.py turn AWR HTML, AWR text, Statspack from any release, or a few pasted sections into one small dict: load profile, top events, time model, a few instance activity stats, top SQL per category, and segments. Statspack and AWR text are fixed-width, so column names are read from the header above each dashed rule. Pasted text often has its whitespace mangled, so rows are read token by token: a name, then numbers. On all 32 lab runs, the text parser gives exactly the same findings as the HTML parser. rules.py , about 150 lines are the checks I'd run by hand: hard parsing, full scans from logical reads, or from db file scattered read when the scans come from disk , row locks, commit storms, NOCACHE sequences, one CPU-heavy statement, and Resource Manager CPU caps. Each fires with the numbers that made it fire. A materiality gate stops a large percentage of an idle database from becoming a finding: "74% of DB time is log file sync" means nothing at 0.1 active sessions. A DBA can live with a model that writes clumsy prose. A model that invents a statistic, in a note going to a client, is a disaster. So every number in Gemma's note is checked against the numbers it was given, allowing for rounding and thousands separators. If any number isn't there, the note goes back once with the offending numbers listed. If a number still can't be found after the retry, it is printed in a warning, not quietly dropped. On the held-out set, neither Gemma reviewer quoted a number that wasn't in its input, and Second Opinion's retry never fired. You can't evaluate a diagnosis tool on reports where nobody knows the answer, so I built a fault lab. There are two disposable Oracle databases in Docker on my workstation: | Database | What it produces | |---|---| | Oracle AI Database 26ai Free 23.26.3 | AWR + ADDM Diagnostics Pack included in Free | | Oracle Database 12.2 Standard Edition 2, control management pack access=NONE | Statspack only | lab/faults.py runs a workload with a known fault for 5–7 minutes: literal SQL, a missing index, sessions fighting over the same rows, commit-per-row, a NOCACHE ORDER sequence, a cartesian join, or a healthy light load. It takes a snapshot before and after, then saves the AWR, ADDM or Statspack report a DBA would get. The rules were written while looking at a 14-run dev set. Then I wrote a held-out set : the same causes, but different SQL, different tables and different session counts, plus two runs with two faults at once. I scored it once. Held-out, 18 runs. A reviewer passes if the injected cause is in its top two. With two faults at once, both have to be in its top three. | Reviewer | AWR 9 | Statspack 9 | |---|---|---| | Oracle ADDM | 7/9 | not available on SE2 | | Gemma 4, raw report | 5/9 | 5/9 | | Rules alone | 9/9 | 8/9 | | Second Opinion | 9/9 | 8/9 | What the misses taught me: where column. resmgr:cpu quantum ; that's real for that database, but an artefact of Free. Lab faults are tidy. To see how it copes with what people actually paste, I collected ten AWR and Statspack reports posted publicly on the OraFAQ forums and Jonathan Lewis's blog. Each has an expert's diagnosis in the thread, and all ten are text, from Oracle 9i to 11g, and often only a few sections. Two things broke at first: After that, the rules matched the expert's diagnosis on 7 of the 10 . The other three were two excerpts that left out the SQL sections the diagnosis depended on, and one log-switch problem just under my threshold. I wrote those two checks after reading these cases, so this is a set of worked examples, not a test score. | Thread | Report | Expert's call paraphrased | Second Opinion | Match | |---|---|---|---|---| | Statspack report: suggestion https://www.orafaq.com/forum/mv/msg/124437/345935/ | Statspack | Full scans of a big table scattered reads | full scans | ✓ | | statspack report https://www.orafaq.com/forum/mv/msg/122717/337645/ | Statspack | Scattered reads on top: the SQL is full-scanning | full scans | ✓ | | Upgrade Argh https://jonathanlewis.wordpress.com/2011/09/26/upgrade-argh/ | AWR text | Small tables' scans turned into direct path reads after an upgrade | full scans | ✓ | | High System I/O wait class https://www.orafaq.com/forum/mv/msg/185603/573992/ | AWR text | log file sync dominates slow redo writes vs. too many commits debated | commit storm, other | ✓ | | Fast now, fast later https://jonathanlewis.wordpress.com/2017/07/24/fast-now-fast-later/ | Statspack | Single-block reads grow as the data grows | other: db file sequential read dominates | ✓ | | Network waits from AWR https://www.orafaq.com/forum/mv/msg/187833/585763/ | AWR text | Nothing to fix: network waits are a tiny share | healthy | ✓ | | Reading AWR https://www.orafaq.com/forum/mv/msg/179660/547263/ | AWR text | Mostly CPU, no instance-level problem | healthy | ✓ | | NOLOGGING in 11gR2 https://www.orafaq.com/forum/mv/msg/184526/568790/ | AWR text | Log switches too frequent checkpoint incomplete, ~19% | healthy | ✗ | | Analysing Statspack 13 https://jonathanlewis.wordpress.com/2013/01/07/analysing-statspack-13/ | Statspack excerpt | CPU-bound on logical I/O from scans; needs the SQL sections | healthy | ✗ | | Analysing Statspack 5 https://jonathanlewis.wordpress.com/2007/04/19/analysing-statspack-5/ | Statspack excerpt | Repeated single-row deletes each doing a short full scan | healthy | ✗ | The three misses are honest misses. Two are blog excerpts where the answer lives in SQL sections that weren't quoted, and a tool that only sees the Top 5 events can't find it there either. The third is a real configuration problem that sits below my 25% threshold. The rules in Second Opinion are hand-written, and I wrote them knowing which faults I'd inject. So the fair question is whether a small open model could learn that triage itself from the parsed numbers, with no rules at all. Tinker let me try that for about the price of a coffee. Data. I left two generators running overnight against both lab databases: 524 short runs with random faults one at a time, or two at once a quarter of the time , random session counts and 90–300 s windows. They use their own tables, SQL and sequences, so nothing overlaps the held-out set. A label is a fault that actually reached the report. I dropped 52 runs where the injected fault never showed. In one, for example, a literal-SQL flood had aged the CPU-heavy statement out of the shared pool before the snapshot, so no report could show it. That left 425 runs to train on and 61 to validate on 47 generated plus my 14 dev runs . The 18 held-out runs stayed untouched. Task. The input is the parsed report, trimmed to what triage needs. The output is JSON ranking the likely causes. The base model and the tuned one get the same prompt, with Qwen3's thinking turned off. Training. LoRA rank 32 on Qwen3-8B, 3 epochs, 81 steps, with a checkpoint saved after each epoch. Training used 3.66M tokens, about $1.61 at list price. With the early test run and the sampling for evaluation, the whole experiment cost roughly $2 of a $10 credit. I picked the epoch on the validation set: | Qwen3-8B, validation 61 runs | Pass | Top cause right | False causes | |---|---|---|---| | Base, untuned | 40 | 31 | 136 | | Tuned, epoch 1 | 55 | 61 | 0 | | Tuned, epoch 2 | 51 | 57 | 4 | | Tuned, epoch 3 | 57 | 61 | 1 | Then I scored the base model and epoch 3 once on the 18 held-out runs: | Qwen3-8B, held-out 18 runs | AWR | Statspack | Top cause right | False causes | |---|---|---|---|---| | Base, untuned | 5/9 | 6/9 | 11/18 | 38 | | Tuned | 8/9 | 8/9 | 18/18 | 0 | "Pass" is the same rule as the main table: the cause in the top 2, or both causes in the top 3 when two were injected. A false cause is a label that wasn't injected. I don't count cpu cap as a false cause, because Free really does throttle itself, and the training labels include it when the report shows it. The base model's problem is easy to see: it names three causes every time, so it is never sure and often wrong. After tuning it names one or two, and the first one is right on every held-out run. Its two misses are both the full-scan + commit-storm pair, where it names the commit storm and drops the full scan. This doesn't replace the rules. The rules plus Gemma pipeline is still the tool, at 17/18 with a written note and every number checked. But a tuned 8B model with no rules reached 16/18 on the same runs. That tells me the triage isn't only in my head, and a model can learn it from lab data I can make as much of as I like. So it went into the tool as a second, independent read. second-opinion report.html --triage-model second-opinion-qwen3 adds the tuned model's own ranking under the review, checks the numbers in its evidence the same way, and says whether it agrees. On the held-out set it catches the review's one miss: on the 12.2 Statspack sequence run, the rules say "committing too often", and the tuned model says "sequence contention", which is right. It also disagrees on the healthy AWR run with real deadlocks. A disagreement isn't a verdict; it's a reason to read both before writing to the client. Running it locally. The tuned model only counts for this use case if it runs offline, so I exported the adapter to Ollama. That was harder than expected: What works is merging the adapter into the exact qwen3:8b weights Ollama already has and stopping at 8-bit lab/tune/export.py . The local copy passes 16/18 too, and it agrees with Tinker's top cause on 17 of 18 runs, at about 0.5–0.7 s per report. The adapter is on Hugging Face: pyaroslav/second-opinion-qwen3-8b-lora https://huggingface.co/pyaroslav/second-opinion-qwen3-8b-lora , in PEFT and GGUF form. Where it differs, it misreads the healthy AWR run as row locks the run with the real deadlocks , and it gets one full-scan + commit-storm pair that the Tinker copy missed. Tracing an agent whose whole selling point is "nothing leaves your laptop" is a bit of a contradiction. So the first design decision was what Sentry may see. With SENTRY DSN set, each review becomes one gen ai.invoke agent transaction with these child spans: gen ai.execute tool for parsing; gen ai.chat per model call, including the retry. What leaves the machine: span names, durations, the model name, token counts, which causes the rules found, and how many numbers failed the check. There are no prompt or response attributes, send default pii=False , and tracing is off unless you turn it on. Sentry's own agent view confirms it: Setup. I wrapped both reviewers in the benchmark as agents, "second-opinion" and "raw-baseline", and ran all 18 held-out reports through each. Performance tokens, latency, cost . The agents dashboard made the cost of the naive approach very concrete: | Agent | Tokens per review | Median latency local GPU | Found the cause | |---|---|---|---| | raw-baseline whole report to Gemma | 16K–24K | 2.3 s | 10/18 | | second-opinion parse + rules + Gemma | 1.2K–1.4K | 0.7 s | 17/18 | The dollar cost is $0 either way, because it runs locally. But about 15 times fewer tokens is what lets a 4B-parameter model run on a consultant's laptop at all, instead of needing a 32K context and a bigger GPU. What I found, and how I debugged it. The first version of that table said 5.0 s for Second Opinion and 6.7 s for the raw baseline. Then I compared it with the traces from running the CLI on the same reports, and the CLI reviews took 0.65–0.92 s : Same model, same prompt, but a 7× difference in latency. The waterfall put almost all the time in the gen ai.chat span, so the problem wasn't my code. In the benchmark, the two reviewers alternate, and they asked Ollama for different context sizes 16K for mine, 32K for the raw report . I checked Ollama's load duration directly. With the same context size, a warm call takes about 0.1 s. Alternating 16K and 32K forces a ~4.2 s model reload on every single call . The benchmark had been timing model reloads, not reviews. The fix is one line: a single context size for every call. After the fix, the same benchmark in Sentry alternates between 1.9–2.8 s for the raw baseline and 0.64–0.72 s for Second Opinion: Accuracy didn't change, but every latency figure in this post did, and without the traces I would have published the wrong ones. For this job, "open" is the product, not a nice-to-have: I didn't want to put a client's report anywhere near this, so the first "user" was a blind reviewer. A separate Claude instance, with no access to my code, labels or results, was told to act as a senior DBA. It was given six lab reports it had never seen, renamed report-1 to report-6 , and asked for a by-hand triage: ranked causes, the numbers that convinced it, and what it would check next. I also renamed one sequence whose name gave the answer away. Then I scored its read and Second Opinion's against the injected faults, with the same pass rule as everywhere else: | Report | Injected | Blind reviewer | Second Opinion | |---|---|---|---| | AWR | full scan | full scan, CPU cap ✓ | full scan, CPU cap, CPU SQL ✓ | | Statspack | NOCACHE sequence | commits, sequence ✓ | commits, sequence ✓ | | Statspack | healthy | healthy ✓ | healthy ✓ | | AWR | commit storm | commits, small redo logs ✓ | commits ✓ | | Statspack | row locks + literal SQL | hard parse, row locks ✓ | hard parse, row locks ✓ | | AWR | CPU-heavy SQL + literal SQL | commits, CPU SQL ✗ | hard parse, commits, CPU cap ✗ | What I learned: So the "user" is a model playing a DBA, not a paying consultant, and six reports is a small sample. A real DBA's blind pass is the next step.