My local AI summary had valid sources. It still connected the wrong facts.

A developer's local Gemma-based standup tool, SlothPulse, produced recaps with valid source IDs that still linked the wrong facts, prompting a redesign of its generation contract to an object with one required key per source. In one five-commit experiment the array-based schema still omitted evidence after a retry, and a later run attached an error investigation to a commit even though both IDs existed, so Python now derives blocker and next-step sections from a single E object with a shared reference and rejects extra commit-based next steps when an error is present. The author notes the tool's generated Markdown labels a KeyError: 'result' as user-reported with an unverified cause, and that a second invalid response stops without appending a digest.

My local AI summary had valid sources. It still connected the wrong facts. Companion engineering notes for my Hacktoberfest Week 3 entry on DEV https://dev.to/tengbyte/slothpulse-i-wrote-the-code-gemma-can-draft-the-standup-2bhp . The submission includes the short demo and setup; this post follows the source-validation failures in more detail. After a day of writing code, reconstructing that day from Git commits is an unappealing second writing task. SlothPulse https://github.com/TengByte/slothpulse uses local Gemma to turn those commits and a traceback into a standup draft, with a source receipt beside each item. The first version produced a tidy developer recap. It also described an unspecified user-reported error as a failure in the recap tool itself. The output had the headings I asked for. The relationship between the facts was unsupported. I built the CLI for Hacktoberfest’s “Slow and Steady” theme. The product loop is short: preview the input, generate locally, review the Markdown. The engineering lesson was in deciding what the model was allowed to connect. Three checks that look similar until they fail There are at least three separate questions to ask about this output: 1. Is it parseable? Does the response satisfy the JSON structure? 2. Did it account for the supplied sources? Are required commits and reported errors represented? 3. Did it preserve the relationship? Does an error’s follow-up cite that error rather than an unrelated commit? None of these checks establishes whether the model’s interpretation is correct. They give the person reviewing the draft a shorter route back to the input. Arrays made missing evidence easy to express My first structured plan used arrays of items with prose and source IDs. In local runs it could omit a blocker or leave a commit uncovered. One five-commit experiment still failed after its retry. The JSON could be well formed while the recap skipped evidence I intended it to cover. I changed the generation contract to an object with one required key for every source. For a single commit and error, the response shape is: { "completed": {"C1": "A summary of this commit"}, "blockers": { "E1": { "summary": "A summary of this reported error", "next step": "One suggested investigation" } }, "next steps": } The schema is created from the collected evidence. Python checks the exact key sets and assigns references from those keys. The model supplies the prose; it does not choose whether a required receipt exists. A second invalid response stops without appending a digest. The tradeoff is repetition: one summary per commit can read awkwardly, and the eight-commit cap can miss part of a busy day. The window and those limits are explicit in the saved note. A known ID can still be the wrong ID A later plan exposed the more useful bug: an investigation of an error was attached to a commit. Both source IDs existed. A validator that only asked “is this ID in the input?” had nothing to object to. The current contract keeps an error’s summary and its suggested investigation inside the same E object. Python derives the blocker and next-step sections from that object and gives them the same reference. When an error is present, extra commit-based next steps are rejected. With no errors, the model may suggest up to two follow-ups citing supplied commits. That removes one opportunity to mix sources. It does not prevent a bad interpretation within the object. The captured Gemma output says a KeyError: 'result' occurs while downloading job results. The trace supports a missing dictionary key in a worker function; it does not establish the complete application behavior or root cause. The generated Markdown labels the error user-reported and its cause unverified. A real commit exposed the remaining gap After the synthetic benchmark, I ran SlothPulse twice on this project’s six actual commits. Both calls passed the source checks. Both summarized “Record passing Python compatibility checks and submission handoff” as “Recorded passing Python compatibility checks and submitted.” The entry was still a draft. The receipt was valid; the completion claim was wrong. An AI-assisted comparison of the six summaries to their sources identified that one status correction, which is preserved separately from the raw output. This is not a human editing-effort measurement. The full trial https://github.com/TengByte/slothpulse/tree/main/evidence/real-repo-2026-10-21 took 13.619 seconds with the model initially unloaded and 8.720 seconds on the immediate repeat, before review time. Local inference also needed a concrete check Calling localhost was not enough to establish the execution boundary: Ollama can expose cloud-backed models through its local API. Before sending project context, SlothPulse inspects model metadata and rejects remote or unverifiable models. The tested server runs with OLLAMA NO CLOUD=1 , and the HTTP client bypasses environment proxies. The user can preview the sanitized context before inference. Common credential patterns are masked, but this is best-effort pattern matching, not a complete secret scanner. Reports and digests can still contain private project details. The trust boundary includes the local server and the person deciding where those files go. What the measurements establish I ran three synthetic scenarios three times each against local gemma:latest on an Apple M5 with 16 GB RAM. Median external process times were 7.668 seconds for five commits plus an error, 6.495 seconds for commits alone, and 3.926 seconds for an error without a repository. All nine final runs passed the documented structural and workflow checks on their first generation attempt. This small, ordered batch measures workflow behavior and latency. It does not score factual accuracy or human time savings. The unit suite separately covers 23 boundary cases, and passes on Python 3.8 and 3.14 in CI. The synthetic fixture, exact sanitized input, validated plan, terminal cast, and raw measurements are in the repository. The 48-second video https://raw.githubusercontent.com/TengByte/slothpulse/main/demo-assets/demo.mp4 replays a real captured call and its wait, with explanatory cards around it. What I would evaluate next The next useful evaluation is held-out, misleading source material: ambiguous commit subjects, traces whose application is unknown, and instructions embedded in otherwise ordinary input. A human would grade whether each sentence is supported, whether uncertainty survived, and how much correction the draft needed. For this version, the gain is a shorter route from a generated sentence to the evidence it was supposed to summarize. I can point to the association the code enforces and to the interpretation it leaves for review. That distinction matters even for a small daily note. Interactive recorded demo https://slothpulse.chicangel12.chatgpt.site · Code and setup https://github.com/TengByte/slothpulse · Measurements and limits https://github.com/TengByte/slothpulse/blob/main/BENCHMARK.md · Edited build log https://github.com/TengByte/slothpulse/blob/main/BUILD LOG.md SlothPulse is a personal project. Codex assisted with implementation, tests, demo preparation, and this draft. The application’s inference uses local Gemma. The demonstration repository and its authors are synthetic; its key-shaped value is deliberately fake.