# I let a local Gemma babysit a PR queue overnight. It fast-tracked 31 of 71 with replies, and I woke up to just the other 40.

> Source: <https://dev.to/iamwasih/i-let-a-local-gemma-babysit-a-pr-queue-overnight-it-fast-tracked-31-of-71-with-replies-and-i-woke-3oc>
> Published: 2026-10-26 06:37:42+00:00

"As the ecosystem grew, maintainers started to face a massive flood of activity and burnout from the rise of low-effort PRs. […] In an era where AI tools make low-effort PRs easier than ever to generate, maintainers face unprecedented volume and noise."

([hacktoberfest.com](https://hacktoberfest.com/llms-full.txt), on why pull requests no longer count toward rewards this year)

AI made pull requests almost free to write. Reading them didn't get any cheaper. The pile grows faster than the handful of volunteers who have to read it.

So I flipped the question. Could a small model on the maintainer's own laptop help them *read* the pile, without ever speaking for them?

**Nightshift** reads the pile overnight. While you sleep, a local Gemma model reads every new PR and issue in your repos and sorts each one: duplicate, spam or slop, out of scope, needs tests or info, trivial and safe, or needs a human. Where a reply makes sense, it drafts one. In the morning you get one page.

**It never acts alone.** It doesn't post, label, close or merge anything by itself:

I didn't test it on fixtures. I pointed it, read-only, at three of this challenge's partners' own repos: `temporalio/temporal`, `timescale/timescaledb` and `mastra-ai/mastra`. Over two scheduled nights and three manual runs it made **112 triage decisions on 111 real items** ([STATS.md](https://github.com/AbdulWasih05/nightshift/blob/feat/dashboard/docs/STATS.md), generated from the database). That's a small sample, so the bad news comes first.

**I labelled night 7's 71 items by hand, and the model wasn't good enough to trust on its own:**

None of it reached GitHub, because nothing does until I approve the exact text. Night 7 is why that rule exists.

The strongest alternative is [GitHub Agentic Workflows](https://github.github.com/gh-aw/blog/2026-01-13-meet-the-workflows/). An AI engine runs in Actions on every new issue, gets no write permission, and separate [safe-outputs](https://github.github.com/gh-aw/reference/safe-outputs/) jobs apply what it asks for. Nightshift uses the same idea: the part that thinks can't write.

gh-aw wins if you already live in Actions: nothing to install, it acts in the repo, and GitHub maintains it. Nightshift is different in four ways:

Rule bots like `actions/stale` stay the baseline for anything a rule does better. The model only gets what rules can't do: slop, near-duplicates and missing information.

A 1:52 walkthrough: the morning report, one card from evidence to approval, the honest result, and the Sentry trace. The footage is real screen capture; the voiceover is synthetic.

**Live dashboard: [https://nightshift-dashboard.onrender.com](https://nightshift-dashboard.onrender.com)** (Render's free tier, so the first load after a quiet spell can take about a minute). The public page shows live numbers from the database. The review page sits behind Auth0, limited to me, because it can post.

Each card on the review page leads with one next step: Review, Ask for info, Close as duplicate of #N, or Quick merge check. Under it are chips built only from GitHub facts, like diff size, failing CI, missing tests, or the duplicate it cites. The model's own reasoning is folded away. Two buttons, **Right** and **Wrong**, are how I labelled all 71 items.

Two replies have gone out this way, both on my own repo: [#5](https://github.com/AbdulWasih05/nightshift/issues/5#issuecomment-6373303034), approved as drafted, and [#6](https://github.com/AbdulWasih05/nightshift/issues/6#issuecomment-6373389268), which I edited first because the model claimed #5 was already fixed. It wasn't. Both show "via nightshiftdemo", the GitHub App behind Token Vault.

The maintainer's night shift. A local agent reads your PR and issue pile overnight, buckets each item (duplicate, spam/slop, out of scope, needs tests or info, trivial and safe, needs a human), drafts a reply, and leaves you one morning digest.

**It never posts, labels, closes or merges anything without your explicit approval.** Partner repos are read-only by construction: the write allowlist is a frozen source constant (`packages/core/src/allowlist.ts`), and the collector's GitHub client rejects every non-GET request.

Runs on a laptop with Gemma 3 via Ollama. $0.00 per night. Contributor data stays on your machine.

Built for the DEV Hacktoberfest Open-Source AI Challenge, Week 3 ("Slow and Steady"). Challenge window: Oct 19 to Oct 25, 2026 (PDT). Repo created 2026-10-21. Any commits after publishing are listed under "Post-deadline changes" below.

Work in progress. See `docs/LOOP.md` for the build log and `docs/DECISIONS.md` for every cut and change.

A TypeScript monorepo: `packages/core` (collector, triage, duplicate search, digest, eval, approval gate), `packages/workflows` (Temporal), `apps/cli`, `apps/dashboard` (Next.js) and `sidecars/tabpfn` (Python).

Every number in this post comes from a file a script writes from the database, never by hand: [STATS.md](https://github.com/AbdulWasih05/nightshift/blob/feat/dashboard/docs/STATS.md), [NIGHTS.md](https://github.com/AbdulWasih05/nightshift/blob/feat/dashboard/docs/NIGHTS.md), the [digests](https://github.com/AbdulWasih05/nightshift/tree/feat/dashboard/docs/digests) and [docs/eval](https://github.com/AbdulWasih05/nightshift/tree/feat/dashboard/docs/eval). Every cut, failure and reversal is dated in [DECISIONS.md](https://github.com/AbdulWasih05/nightshift/blob/feat/dashboard/docs/DECISIONS.md).

```
02:00  Temporal schedule fires nightlyTriage
 ├─ collect       read-only GitHub client, 200 items/repo cap
 ├─ claimBacklog  whatever an earlier night didn't finish
 ├─ embed         nomic-embed-text → pgvector (Tiger Data)
 ├─ triageChunk×N Gemma 3 4B → 1B fallback → strict parser + rules
 └─ finally       digest → finish the night → unload the models
morning  /review: read, label, approve, Post (Auth0 Token Vault)
```

I built it in small units, each with an acceptance check written before any code, run against the real repos, then read by a reviewer agent that had never seen the code. That reviewer found a blocker in most units, and many of the stories below started there ([LOOP.md](https://github.com/AbdulWasih05/nightshift/blob/feat/dashboard/docs/LOOP.md)).

Gemma 3 4B returns one JSON object per item: bucket, confidence, reason, evidence and an optional draft. **My parser is the source of truth, not the model.** Anything unparseable becomes "needs a human" and is never dropped; across 112 decisions there were zero parse failures. The 1B model steps in only when 4B errors or times out, so the smaller model never quietly overrules the bigger one. 4B answered 109 items, 1B answered 3.

Early on it called a 414-line concurrency fix "trivial and safe" at 0.95 confidence. I didn't fix that with a better prompt. I fixed it with rules in the parser: "trivial" means under 40 changed lines, a duplicate must cite a candidate it was actually shown, and spam needs two tells. Those rules changed **38 of 112 answers**, almost all of them "trivial" calls on big PRs, the largest at 6,246 lines.

Why 4B and nothing bigger? The laptop has a 4 GB MX130, and even 4B only fits 61% of its layers on the GPU. I capped both models at 3 threads so the laptop stays usable overnight. The cost is speed: a median of 92 seconds per item.

This runs on a 4-core laptop that reboots, sleeps and gets borrowed for other work. Nobody staged these failures; they happened during real nights.

**The laptop rebooted 40 minutes into night 5.** Everything was down until the next day. When I brought the stack back, the night resumed from Temporal's saved state: nothing was lost and nothing was triaged twice, because each chunk of items is an activity that records its progress.

**One item took 1,956 seconds.** I first blamed my own dashboard builds. The Sentry section below shows the laptop was asleep, and DECISIONS.md keeps both entries.

The bug I'm least proud of was in my own plumbing, not the model. When Ollama went down mid-night, my first version filed every item as "needs a human" and reported the night as fine. Now an outage stores nothing, the activity fails, and Temporal waits until Ollama is back.

Both scheduled nights ended "partial": night 5 through a reboot, an outage and my own builds; night 7 because the laptop slept from 02:30 to 10:01 and I then stopped it to use the machine. Unfinished items carry over to the next night. I chose not to keep the laptop awake, so nights simply pause while it sleeps.

The collector only sees the last 24 hours, but duplicates are usually older. So a backfill keeps up to 90 days of each repo, about 5,500 items, in Tiger Cloud: Postgres with pgvector and full-text search, shared by the laptop and the dashboard.

I measured before picking thresholds. Vector similarity alone is useless here: in one repo, 89% of items have a near neighbour at cosine 0.80, because everything in a repo sounds alike. A genuine duplicate pair, temporal#12211 and #11497, sits at 0.96, while related-but-different pairs sit around 0.87–0.89. So the gate is vector-only at 0.92 or above, or vector plus shared title terms, or a strict keyword match. That last rule came from a real miss: the model once paired two temporal issues just because both titles said "task queue user data".

Reading and writing use two different GitHub clients. The reader refuses anything but reads, and the writer is bound to one repo. The first review found a hole in my own allowlist: it checked the parameters, not the URL actually sent. It now checks the resolved URL, and that bypass is a regression test.

A reply goes out only when three things hold. A human approved it, and the approval is bound to a hash of the exact text shown, so what's posted is what was read. A unique index allows one post per GitHub item, claimed before the network call, so a retry can't double-post. And at that moment Auth0 Token Vault hands the dashboard a short-lived GitHub token; none is stored in the app.

Each item is an `invoke_agent` span, with a `gen_ai.chat` span for each model attempt, carrying tokens, latency, model and bucket, and never text. An allowlist decides what leaves the laptop, and a test enforces it.

That 1,956-second item is where the trace paid for itself:

| IST, Oct 23 | Sentry trace | Windows event log | 
|---|---|---|
| 13:05:26 | 4B chat starts |  | 
| 13:06:05 |  | system enters sleep | 
| 13:37:17 |  | resume | 
| 13:37:18 | 1B chat starts |  | 
| 13:38:01 | 1B chat ends, row saved |  | 

They agree to the second. The model wasn't slow; the laptop was asleep. The trace also showed something the event log couldn't: the sleep changed *which model answered*. The 4B call was cut off on wake, and the 1B fallback started one second after the wake. The trace shows *when* the call ended, not why. The likeliest cause is that its 240-second timer counted the suspended half hour. Either way, a timeout that can't tell a hung model from a sleeping laptop hands work to the weaker model for no reason. Fixing that is next, and I'd never have seen it without the trace.

TabPFN produces a "likely to stall" flag, trained on stored PR history. The data was harder than the model: the backfill picked items by *update* date, so older PRs were survivors that skewed everything. I caught it by plotting merge rate by month before training, and now train only on PRs from inside each repo's window.

The triage call itself is a Mastra Agent with structured output. I use Mastra for nothing else, so I'm not entering its category.

Three things Nightshift can only do because the model is open and runs locally.

**1. Nothing goes to a model vendor.** Every triage call goes to Gemma through Ollama on the laptop. Items live in a Postgres I control, and Sentry gets an allowlisted set of metadata, never text ([`telemetry.ts`](https://github.com/AbdulWasih05/nightshift/blob/feat/dashboard/packages/core/src/telemetry.ts)). For a private repo, that's the difference between allowed and not allowed.

**2. Free calls change how it behaves.** All 112 decisions cost $0.00: 131,474 tokens in and 16,469 out ([STATS.md](https://github.com/AbdulWasih05/nightshift/blob/feat/dashboard/docs/STATS.md)). At Claude Sonnet 5.5's list price ($2 in, $10 out per million tokens, [checked 2026-10-25](https://platform.claude.com/docs/en/about-claude/pricing)) the same tokens would cost $0.43, so the money isn't the argument. Behaviour is: with no meter running, the agent can spend 92 seconds on an item, re-read anything that changes, and replay old nights against a new prompt before it ships.

**3. The model is mine to fence.** I can see and control every layer: the prompt, the model, and the parser between them. That's why rules can overrule the model where it's weak. On night 5 alone, it called 12 items trivial and the 40-line rule overruled 10 of them, including a 717-line PR ([night 5 digest](https://github.com/AbdulWasih05/nightshift/blob/feat/dashboard/docs/digests/2026-10-22-night5.md)).

Instead of 71 tabs, night 7 gave me one page ([digest](https://github.com/AbdulWasih05/nightshift/blob/feat/dashboard/docs/digests/2026-10-24-night7.md)). Here's how it held up against my own labels:

| Pile | Items | Right, by my labels | What you do | 
|---|---|---|---|
| Needs you | 40 | 40 of 40 | read it | 
| Drafted replies | 25 | 2 of 25 | read the draft, then approve, edit or skip | 
| Trivial and safe | 6 | 4 of 6 | a quick merge check | 

On night 7, when it said "needs you", it was right all 40 times. But it only sends 62% of what really needs me there; the rest lands among the drafts, mostly as false duplicates. So the safe way to read the morning is: trust the top pile, and read every draft. The digest's "est. 113 min" is labelled as an estimate. I haven't timed myself, so I don't claim minutes saved.

| Night 7, n = 71 | Correct | 
|---|---|
| Gemma 3 4B + rules (prompt v1) | **46 (64.8%)** | 
| Always "needs a human" | **65 (91.5%)** | 

That's one night, on repos where most items really do need a maintainer (65 of 71, by my labels), so always-escalate is a hard baseline to beat. The more useful view is where it was confidently wrong:

| Bucket the model chose | Precision | 
|---|---|
| Duplicate | **0 of 13** | 
| Needs tests or info | 2 of 12 | 
| Trivial and safe | 4 of 6 | 

**All 13 duplicate calls failed the same way:** related, not the same ([full list with my reasons](https://github.com/AbdulWasih05/nightshift/blob/feat/dashboard/docs/STATS.md)). Five were siblings in a series, like a mastra run of Word, then PowerPoint, then OneDrive toolsets. Eight were a different bug in the same corner, like "cancel overwrites a finished run" against its exact reverse. The search found the right neighbourhood every time; the 4B model just couldn't tell a neighbour from a twin, and said so at up to 0.98 confidence. My citation rule passed all 13, because each cited a real candidate. It checks *that* a duplicate is cited, not *whether* it's the same thing.

Nothing was harmed: all 13 were on read-only partner repos. On my own repo, each would have reached me as a suggestion, not a closed issue.

**Other honest misses:**

**What's next:** prompt v2 makes the model say what's *different* between two items before it may call them duplicates, an idea borrowed from Microsoft hve-core's [backlog triage](https://microsoft.github.io/hve-core/docs/agents/backlog/triage). It's the only change, so the next night's difference can be traced to it. It ships only if it beats v1 on these same 71 labelled items. I didn't run it before publishing: replaying 71 items takes about two hours on this laptop (71 at a 92-second median), and I'd rather publish a measured v1 than an unmeasured v2.

`timescale/timescaledb` itself. Used for search, not time series. Shared by the laptop and the dashboard.`render.yaml` blueprint