Building explain-ci: The AI Was the Smallest Part

A developer built explain-ci, a GitHub Action that parses failed CI job logs and posts a plain-English root-cause explanation as a PR comment using a single LLM API call. The engineer reports that the model was the smallest part of the work, with most effort going into reliability features like never failing the pipeline, editing its own comment on re-runs, and scrubbing credentials before log text leaves the runner.

CI goes red. Before I can think about the actual fix, I'm expanding the failing job and scrolling through three hundred lines of dependency resolution to find the one line that matters. Then the next person on the repo does the same thing, from scratch. So I built explain-ci https://github.com/gopalcnepal/explain-ci : a GitHub Action that reads the failed job's log and posts a plain-English explanation as a PR comment. By the time anyone opens the PR, it's already there. That's a real run. Here's what CI printed: E AssertionError: assert == 'washers' E Right contains one more item: 'washers' And what the comment said: the function used < where it needed <= , so items sitting exactly on the threshold were skipped. The traceback tells you the assertion failed. It never tells you why. The model worked that out from the log alone — it never saw the source file. I expected this to be the interesting bit. It isn't. GitHub's job logs carry structure markers — group , endgroup , error . Find the first error, walk backwards to see which step was running, slice the log into three sections. That's plain string handling, no model involved. Then those sections go to the provider with a short prompt asking for two things: root cause and suggested fix. That's the whole feature. One API call and some parsing. The one decision worth anything is what you don't send. Not the whole log — each section is capped, and the cap keeps the tail , because errors cluster at the end while the top is setup noise. That holds whether the job printed 200 lines or 200,000. Everything that took real time was about making the thing well-behaved enough that you'd leave it installed. It must never fail your pipeline. An action that explains failures has no business causing one. If the GitHub API hiccups or the provider returns a 500, it emits a warning and exits 0. Your build status should reflect your build, not my action's bad day. It edits its own comment. Re-run a failing job four times and you don't want four bot comments. Each one carries a hidden marker, so a re-run finds the previous comment and updates it in place. Only the newest run comments. Scoped per workflow — a repo with five workflows shouldn't get five explanations. Credentials get scrubbed first. Log text goes to a third party. GitHub masks the secrets it registered, but tokens minted or echoed during a job — an OIDC session key, a git remote carrying a password, a verbose curl dumping its auth header — reach the log unmasked. So they're stripped before anything leaves the runner. None of that is AI work. All of it is the difference between a demo and something you'd actually install. The stale-run check was supposed to stop old runs commenting on new commits. It queried the endpoint that returns runs from every workflow in the repo, so if any other workflow had run more recently, the current run decided it was stale and said nothing. In a single-workflow repo it looked fine. In a real one it silently did nothing. The second is more interesting, because I caused it while fixing something else. My first pass at secret redaction flagged any long run of characters as a possible key. Deep file paths match that trivially: php /home/runner/work/myorg/myproject/src/main/java/com/example/Service.java - / REDACTED .java It over-redacted rather than leaked, so nothing was exposed — but it removed the failing file path, which is exactly the context the model needs. My tests missed it because every path I'd invented happened to contain dots and underscores, which break the match. A real path didn't. Both bugs have the same shape: the code was tested against what I imagined the input looked like, not against a real one. explain-failure: needs: test if: always runs-on: ubuntu-latest steps: - uses: gopalcnepal/explain-ci@v1 with: api key: ${{ secrets.OPENAI API KEY }} It has to be its own job — the failed job's log isn't queryable through the API until that job has finished. Bring your own key. It speaks the OpenAI-compatible API, so OpenAI, Gemini, Claude, Groq, Mistral and friends all work, and base url points it at a self-hosted Ollama if your logs can't leave your network. It's free and MIT. If you try it and it gets something wrong, I'd like to hear about it.