{"slug": "porting-the-self-fixing-bug-workflow-to-a-bot-with-no-sentry-no-railway-no-ci", "title": "Porting the Self-Fixing Bug Workflow to a Bot With No Sentry, No Railway, No CI Runner", "summary": "A developer ported TribeMarkets' self-fixing production bug workflow to python-discord-scheduler-bot, a Python Discord bot for scheduling Rainbow Six Siege gaming sessions that runs as a systemd service on a mini PC, replacing Bugsink, Sentry, Railway, and a GitHub Actions CI runner with a rotating app.log file and a single git clone. The port required re-deriving every guardrail from the original design, including fingerprinting log lines by stripping quoted values and numbers before hashing and scoping occurrence counts to the current release's short git SHA. The checkpoint mechanism tracks the log file's inode plus a byte offset and walks both rotated backups on an inode mismatch to avoid silently dropping records between scheduled runs.", "body_md": "[← All technical posts](https://patrickdesjardins.com/blog)\n\n# Porting the Self-Fixing Bug Workflow to a Bot With No Sentry, No Railway, No CI Runner\n\nPosted on:\n\nI previously wrote about [TribeMarkets' self-fixing production\nloop](https://patrickdesjardins.com/blog/building-tribemarkets-with-ai-architecture-and-self-fixing-production):\na scheduled GitHub Actions workflow that reads grouped errors from Bugsink and,\nunder a long list of guardrails, prepares a draft pull request for a\nrepeatable production bug. I wanted the same closed loop for a much older,\nmuch less glamorous project: [python-discord-scheduler-bot](https://github.com/MrDesjardins/python-discord-scheduler-bot),\na Python Discord bot that schedules gaming sessions for a Rainbow Six Siege\ncommunity, running as a systemd service on a mini PC under my desk.\n\nThe problem is that almost none of [TribeMarkets'](https://tribemarkets.com/) infrastructure exists here.\nThere is no Bugsink or Sentry. Errors go to a rotating `app.log` file written\nby Python's own `logging` module. There is no Railway. The bot runs directly\non a box I SSH into. There is no GitHub Actions runner with a clean, disposable\ncheckout on every run. There is one git clone, and it's the same one the bot\nprocess reads its code from. Every guardrail from the original design had to\nbe re-derived for a much more fragile execution environment, and two of the\nbugs I hit while building it were more interesting than the feature itself.\n\n## Sourcing incidents from a log file instead of an error tracker\n\nBugsink hands you pre-grouped issues with occurrence counts for free. A raw\n`app.log` doesn't group anything, it's just lines. The bot already logs in a\nconsistent format (see `deps/log.py`), so the first job is turning that into\nsomething fingerprintable:\n\n```\nLOG_LINE_RE = re.compile(\n    r\"^(?P<ts>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3}) - (?P<level>\\w+) - (?P<message>.*)$\"\n)\n\ndef normalize_message(message: str) -> str:\n    \"\"\"Collapse variable data (ids, quoted values, numbers) so recurring errors fingerprint the same.\"\"\"\n    normalized = QUOTED_RE.sub(\"<val>\", message)\n    normalized = NUMBER_RE.sub(\"<n>\", normalized)\n    return normalized.strip()\n\ndef fingerprint(message: str) -> str:\n    return hashlib.sha256(normalize_message(message).encode(\"utf-8\")).hexdigest()[:20]\n```\n\nStripping quoted values and numbers before hashing means `\"User '12345' not found in guild '998877'\"` and `\"User '99' not found in guild '11'\"` collapse to\nthe same fingerprint. Occurrence counts are then scoped to the current release\n(the short git SHA), so a fix that lands in a new deploy doesn't inherit a\nstale count from before the fix. It starts filing incidents fresh only if the\nsame bug is actually still happening.\n\nBecause `app.log` rotates (`RotatingFileHandler`, `backupCount=2`), a naive\n\"read from where I left off\" approach breaks the moment a rotation happens\nbetween two scheduled runs. The checkpoint is the file's inode plus a byte\noffset, and on a mismatch the script walks both rotated backups to make sure\nnothing between runs gets silently dropped:\n\n``` php\ndef read_new_log_records(log_path: Path, state: LogState) -> list[LogRecord]:\n    current_stat = log_path.stat()\n    records: list[LogRecord] = []\n    if state.checkpoint_inode is not None and state.checkpoint_inode != current_stat.st_ino:\n        backups = [log_path.with_suffix(log_path.suffix + f\".{n}\") for n in (1, 2)]\n        matched_index = next(\n            (i for i, b in enumerate(backups) if b.exists() and b.stat().st_ino == state.checkpoint_inode),\n            None,\n        )\n        if matched_index is not None:\n            matched_backup = backups[matched_index]\n            with matched_backup.open(\"r\", encoding=\"utf-8\", errors=\"replace\") as handle:\n                handle.seek(state.checkpoint_offset)\n                records.extend(parse_log_lines(handle.read()))\n            for newer_backup in reversed(backups[:matched_index]):\n                if newer_backup.exists():\n                    records.extend(parse_log_lines(newer_backup.read_text(encoding=\"utf-8\", errors=\"replace\")))\n        state.checkpoint_offset = 0\n    ...\n```\n\nOnce a fingerprint repeats at least `AUTO_FIX_MIN_OCCURRENCES` times in the\ncurrent release, it becomes a durable GitHub issue, not a pull request yet,\njust a tracked record that survives even if the automation never manages a\nconfident fix.\n\n## No CI runner means the machine validating the patch is the machine running the bot\n\nThis is where the port stopped being a straight copy. On TribeMarkets, GitHub\nActions gives every run a disposable VM: clone, patch, test, throw away. Here,\nthe only place with the repo, the dependencies, and a working test setup is\nthe production box itself, the same directory `gametimescheduler.service`\nruns the bot from.\n\nMy first version did the obvious, wrong thing: `git checkout -b`, run the\nvalidation suite in place, `git checkout --detach HEAD` in a `finally` block.\nIt worked in manual testing. It was also a production incident waiting to\nhappen, and a code review of my own commit caught it before it shipped:\n\nCritical fix: `validate_patch`/commit/push were running `git checkout -b`,\n`make unit-test`, and `git checkout --detach HEAD` directly in `repo_root`,\nwhich is the same directory `gametimescheduler.service` runs the bot from. A\nrestart during validation (or a concurrent `deployment/update.sh`) could\npick up an unmerged candidate patch, and the `finally` block left the\ncheckout in a detached HEAD instead of back on `main`.\n\nTwo independent things could go wrong at once. A systemd restart mid-validation\nwould read whatever half-applied patch was on disk at that instant, and a\nconcurrent `deployment/update.sh` pull would race the branch switch outright.\nThe fix is a disposable `git worktree`, built from the exact commit currently\ndeployed, so `repo_root` is never written to, only ever read for the log\nfile, the current commit, and source context:\n\n``` php\ndef create_worktree(repo_root: Path, branch: str) -> Path:\n    \"\"\"Check out ``branch`` in a disposable `` git worktree`` instead of `` repo_root`` itself.\"\"\"\n    base = Path(tempfile.mkdtemp(prefix=\"log-autofix-wt-\"))\n    worktree_path = base / \"wt\"\n    run_command(repo_root, [\"git\", \"worktree\", \"add\", \"-b\", branch, str(worktree_path), \"HEAD\"])\n    venv_source = repo_root / \".venv\"\n    if venv_source.is_dir():\n        (worktree_path / \".venv\").symlink_to(venv_source, target_is_directory=True)\n    return worktree_path\n```\n\nPatch application, `black --check`, `mypy`, and `make unit-test` now all run\ninside `worktree_path`. Cleanup is best-effort and never raises, since it\nalways runs from a `finally`:\n\n``` php\ndef remove_worktree(repo_root: Path, worktree_path: Path, branch: str) -> None:\n    subprocess.run([\"git\", \"worktree\", \"remove\", \"--force\", str(worktree_path)],\n                    cwd=repo_root, capture_output=True, check=False)\n    subprocess.run([\"git\", \"worktree\", \"prune\"], cwd=repo_root, capture_output=True, check=False)\n    subprocess.run([\"git\", \"branch\", \"-D\", branch], cwd=repo_root, capture_output=True, check=False)\n```\n\nThe unit test for this isn't just \"does the worktree exist.\" It asserts that\n`repo_root` is on the exact same branch, at the exact same HEAD, with an empty\n`git status --porcelain`, both before and after a commit happens inside the\nworktree:\n\n``` php\ndef test_create_worktree_never_touches_the_live_repo_root(tmp_path: Path) -> None:\n    repo = tmp_path / \"repo\"\n    _init_repo(repo)\n    original_head = subprocess.run(\n        [\"git\", \"rev-parse\", \"HEAD\"], cwd=repo, check=True, capture_output=True, text=True\n    ).stdout.strip()\n\n    worktree = create_worktree(repo, \"automation/log-autofix/test123\")\n    try:\n        (worktree / \"marker.txt\").write_text(\"patched content\\n\", encoding=\"utf-8\")\n        _run_git(worktree, \"add\", \"marker.txt\")\n        _run_git(worktree, \"commit\", \"-q\", \"-m\", \"patch\")\n        assert (repo / \"marker.txt\").read_text(encoding=\"utf-8\") == \"main content\\n\"\n    finally:\n        remove_worktree(repo, worktree, \"automation/log-autofix/test123\")\n```\n\nThat's a test whose entire job is proving a negative: that the thing running in production is unaffected by the thing being validated next to it.\n\nA second consequence of running on a shared box instead of an ephemeral\nrunner: a manual debugging run can now overlap a scheduled timer fire, or two\ntimer fires can overlap if `make unit-test` runs long. Both would race on the\nsame state file and worktree bookkeeping, so the whole `process()` call is\nwrapped in a non-blocking file lock that lives outside the repo (so it never\nneeds a `.gitignore` entry):\n\n```\nlock_path = Path(tempfile.gettempdir()) / f\"log-autofix-{config.repo_root.name}.lock\"\ntry:\n    with FileLock(str(lock_path)).acquire(timeout=0):\n        report = process(config)\nexcept FileLockTimeout:\n    print(\"log-autofix: another run is already in progress; skipping\", file=sys.stderr)\n    return 0\n```\n\n## Systemd instead of a cron-shaped GitHub Actions schedule\n\nThere's no Actions YAML here. The equivalent is a plain `.service` + `.timer`\npair, deliberately *not* auto-installed by the normal deploy script, since\nturning this on requires secrets to exist on the box first:\n\n```\n# systemd/gametimescheduler-log-autofix.timer\n[Timer]\nOnBootSec=10min\nOnUnitActiveSec=15min\nPersistent=true\n```\n\n`deployment/update.sh` only keeps the unit files in sync after they already\nexist under `/etc/systemd/system/`. Enabling the automation for the first\ntime is a manual, one-time step documented in `systemd/README.md`, with the\nsame triage-only (`AUTO_FIX_ENABLED=false`) mode as the original design so I\ncould watch it file issues for a while before trusting it to open PRs.\n\n## Two bugs that only showed up against the real thing\n\nThe rest of the guardrails carried over almost unchanged from the TribeMarkets\ndesign: secrets are scrubbed before anything reaches a model, the model must\nreturn a structured plan with an explicit `can_open_pr` boolean and a\nconfidence score, patches are restricted to an allowlist of directories and\nmust touch at least one test file, and every automatic PR is opened as a draft\nthat the workflow never merges. What's worth writing down are two failures\nthat only surfaced once this ran against production instead of against mocks.\n\n**The script never actually read its own secrets.** It documented reading\n`GITHUB_REPOSITORY`, `GITHUB_TOKEN`, and `NVIDIA_API_KEY` from `.env`, the same\nconvention `bot.py` uses, but nobody had called `load_dotenv()`. It worked\nfine from an interactive shell where those variables happened to already be\nexported, and failed silently everywhere else, including under the systemd\ntimer. Found by running it manually on production right after enabling the\ntimer:\n\n``` python\nfrom dotenv import load_dotenv\n\nROOT_DIR = Path(__file__).resolve().parents[1]\nif str(ROOT_DIR) not in sys.path:\n    sys.path.insert(0, str(ROOT_DIR))\n\n# Same convention as bot.py: read secrets/tuning knobs from the repo-root .env.\nload_dotenv(ROOT_DIR / \".env\")\n```\n\n**The default model was simply too slow to use synchronously.** The initial\ndefault, `moonshotai/Kimi-K3` (about 2.8 trillion parameters), was the natural\nchoice since it's the model the original TribeMarkets script defaults to.\nVerified directly against NVIDIA's API, a real request timed out after 302\nseconds with a 504 from NVIDIA's own gateway. That's not a client-side\ntimeout, the gateway itself gave up. That's not viable for a script meant to\nrun synchronously every 15 minutes. `z-ai/glm-5.3-flash` responded in about 46\nseconds and became the new default.\n\nSwitching models surfaced a third, quieter issue: some NVIDIA-hosted reasoning\nmodels can return `\"content\": null` alongside populated `reasoning_content`\nwhen the response gets cut off before the final answer, most often with\n`finish_reason == \"length\"`. The original code treated that as \"non-JSON\noutput,\" technically true, but useless for debugging. Now it's a distinct,\ndiagnosable error:\n\n``` php\ndef chat_completion_output(payload: dict[str, Any]) -> tuple[str, str]:\n    \"\"\"Return ``(content_text, finish_reason)``.\"\"\"\n    choices = payload.get(\"choices\")\n    if not isinstance(choices, list) or not choices:\n        return \"\", \"\"\n    first = choices[0]\n    finish_reason = str(first.get(\"finish_reason\") or \"\")\n    message = first.get(\"message\")\n    if not isinstance(message, dict):\n        return \"\", finish_reason\n    content = message.get(\"content\")\n    if isinstance(content, str):\n        return content, finish_reason\n    ...\ntext, finish_reason = chat_completion_output(payload)\nif not text.strip():\n    detail = f\" (finish_reason={finish_reason!r})\" if finish_reason else \"\"\n    raise ModelOutputError(\n        f\"NVIDIA returned empty content{detail}; the model likely spent its token \"\n        \"budget on internal reasoning before answering\"\n    )\n```\n\nNeither of these would show up in a unit test written against a mocked API response, because the mock is exactly what wouldn't have exhibited the behavior. Both were found by insisting on a real triage-only run against real production before trusting the automation with write access.\n\n## What carried over, and what didn't\n\nThe shape of the guardrails is identical to TribeMarkets: filter candidates by occurrence count and classification, scrub before sending anything to a model, require a structured plan with an explicit confidence threshold and no risk flags, restrict patches to an allowlist that excludes the automation script itself, require a test in every patch, validate with the same lint/type/test suite a human would run, cap PRs per run and per day, and never merge or deploy automatically. That part is genuinely portable: it's a policy, not infrastructure.\n\nWhat isn't portable is the plumbing underneath it, and that's the part that\nactually took the iteration. No error tracker means building fingerprinting\nand log-rotation recovery from scratch, and no disposable CI runner means the\nvalidation sandbox has to be built by hand: a `git worktree`, a file lock,\nand a healthy amount of paranoia about what \"the same directory the bot is\nrunning from\" actually means while a patch is being tested next to it.\n\nThe full source for this automation lives in\n[`scripts/log_autofix.py`](https://github.com/MrDesjardins/python-discord-scheduler-bot/blob/main/scripts/log_autofix.py)\nin the bot's repository, alongside its tests in\n[`tests/log_autofix_unit_test.py`](https://github.com/MrDesjardins/python-discord-scheduler-bot/blob/main/tests/log_autofix_unit_test.py)\nand the systemd setup notes in\n[`systemd/README.md`](https://github.com/MrDesjardins/python-discord-scheduler-bot/blob/main/systemd/README.md).", "url": "https://wpnews.pro/news/porting-the-self-fixing-bug-workflow-to-a-bot-with-no-sentry-no-railway-no-ci", "canonical_source": "https://patrickdesjardins.com/blog/self-fixing-bug-discord-bot-app-log", "published_at": "2026-09-16 00:00:00+00:00", "updated_at": "2026-09-16 02:36:06.406279+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["TribeMarkets", "python-discord-scheduler-bot", "Bugsink", "Sentry", "Railway", "GitHub Actions", "Rainbow Six Siege"], "alternates": {"html": "https://wpnews.pro/news/porting-the-self-fixing-bug-workflow-to-a-bot-with-no-sentry-no-railway-no-ci", "markdown": "https://wpnews.pro/news/porting-the-self-fixing-bug-workflow-to-a-bot-with-no-sentry-no-railway-no-ci.md", "text": "https://wpnews.pro/news/porting-the-self-fixing-bug-workflow-to-a-bot-with-no-sentry-no-railway-no-ci.txt", "jsonld": "https://wpnews.pro/news/porting-the-self-fixing-bug-workflow-to-a-bot-with-no-sentry-no-railway-no-ci.jsonld"}}