Ahnii!
https://jonesrussell.github.io/blog/content-autopilot/ covers how this blog's Content Autopilot workflow mines, curates, and publishes without a human reading the draft first. Three things broke or needed hardening after that pipeline went live: how it pays for Claude, why the auth silently died, and the two gates bolted on afterward to catch a bad draft before it ships.
The produce step originally authenticated with ANTHROPIC_API_KEY
, a pay-as-you-go API account billed per token. That account ran dry, and the daily cron started failing on the very first Claude call. Swapping to CLAUDE_CODE_OAUTH_TOKEN
fixed it in one line: this env var runs the headless CLI against a Claude subscription instead.
env:
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
GH_TOKEN: ${{ secrets.CROSS_REPO_TOKEN }}
claude setup-token
mints this token locally. It's a short-lived credential meant for one CLI session, not a service account, and that mismatch is exactly what caused the next failure.
Subscription tokens rotate. This one got revoked, and the workflow had no step watching for a failure()
outcome. Four daily runs failed in a row before anyone noticed, because a cron job with no output channel just... stops. Nobody was checking the Actions tab every morning.
The fix wasn't a longer-lived token. It was making failure loud:
- name: Alert on failure
if: failure()
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPO: ${{ github.repository }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
gh label create autopilot-failure --repo "$REPO" --color d73a4a \
--description "Content Autopilot run failed" 2>/dev/null || true
body=$(printf 'Content Autopilot run failed: %s\n\nMost common cause: CLAUDE_CODE_OAUTH_TOKEN expired or was revoked (subscription tokens rotate). Re-sync the secret from a fresh token, or run `claude setup-token` for a long-lived one.\n\n@jonesrussell' "$RUN_URL")
existing=$(gh issue list --repo "$REPO" --label autopilot-failure --state open --json number --jq '.[0].number // empty')
if [ -n "$existing" ]; then
gh issue comment "$existing" --repo "$REPO" --body "$body"
else
gh issue create --repo "$REPO" --title "Content Autopilot is failing" \
--label autopilot-failure --body "$body"
fi
This runs on any step failure in the job, not just the auth step. It opens one issue, reuses it on repeat failures instead of spamming a new one per day, and GitHub's default notification settings turn that issue into an email. The permissions
block needed issues: write
added alongside the existing contents: write
and id-token: write
for this to work.
Auth failures are loud and easy to fix. A boring, over-hedged, cliche-riddled post is a quieter failure, and nothing upstream of publish was checking for it. The next addition borrows the Self-Refine technique from the AI-writing research: have the model critique its own output once, then revise, before anything ships.
Critique and then revise the blog post at <path> and its social copy.
Rubric: (1) voice matches docs/blog-style.md and the essay reference;
(2) every claim is backed by a concrete fact, number, or reference;
(3) sentence lengths vary (no runs of same-length sentences);
(4) zero phrases from the site's banned AI-cliche list;
(5) intro scope line is not the phrase 'This post covers'.
The step is continue-on-error: true
. If the critique call itself fails or times out, the workflow falls back to publishing the un-refined first draft rather than blocking the whole run over a quality pass.
A revise pass helps, but it's still the same model marking its own homework. The step after it is deliberately dumb: scripts/slop-check.mjs
runs no API call at all. It strips frontmatter, code fences, and markdown syntax, splits what's left into sentences, and scores three signals.
const mean = totalWords / lens.length;
const variance = lens.reduce((a, n) => a + (n - mean) ** 2, 0) / lens.length;
const stdev = Math.sqrt(variance);
const burstiness = stdev / mean; // human ~0.6-1.2, AI slop < 0.4
Human writing varies sentence length a lot; AI writing tends to cluster around one length. The script also counts em dashes per thousand words (over 20 reads as machine-written) and matches a growing list of stock corporate-blog phrases pulled straight from the site's own style guide. It fails the draft on a hard burstiness floor of 0.32. Three or more banned-phrase hits alone are enough too, and so is any two of the three signals firing together. A failed check holds the draft: nothing gets committed, and the source issue stays open for the next day's run to try again.
Build order matters here. The slop gate runs before the Hugo build gate, so a flagged draft never even reaches hugo --gc --minify
, let alone git push
.
Two posts have shipped through this pipeline since it went live, tracked by issue number in data/autopilot-ledger.json
. None of the hardening above came from planning ahead. Each piece exists because a specific run failed in a specific way: out-of-balance API credit, then a revoked token nobody caught, then no evidence-based reason to trust an unreviewed draft's prose. Unsupervised doesn't mean untested. It means every failure mode has to turn into a gate before the same thing is allowed to happen twice.
Baamaapii