{"slug": "upgrading-and-recovering-my-self-hosted-openclaw-agent-telegram-bot", "title": "upgrading and recovering my self-hosted openclaw agent + telegram bot", "summary": "A developer self-hosting an openclaw agent on an 8GiB, 4-core Linux VM had the agent upgrade itself, which stopped the gateway and left the Telegram bot dead when the post-upgrade doctor aborted on a missing codex plugin. Recovery required moving TMPDIR off a 3.8GiB RAM disk that hit ENOSPC, repairing a session-migration fingerprint in sqlite, and coordinating with the agent's own repair scripts; the developer also cut 51 plugins to 12, trimmed ~1,340 weekly unattended LLM runs to two scheduled jobs, and added daily backups and version control for the agent's memory files.", "body_md": "**TL;DR**\n\nI currently host an openclaw agent on a VM and use it for various coding tasks and project spec generation. it was feeling slow so I ssh-ed into the VM myself and asked a claude code agent to audit and upgrade it. this post is about what broke, what didn't, and how I'd do it next time.\n\nwhat happened:\n\n`/tmp` and ran out of space.\nwhat I upgraded/changed: openclaw to the most current release, primary model moved to GPT-6 Astra, container ports bound to localhost with firewall rules in `DOCKER-USER`, 4GB swap, a 1.2GB log database cut to 153MB, 51 plugins trimmed to 12, unattended LLM runs cut from ~1,340 a week to 2 small scheduled jobs, and daily backups plus version control for the agent's memory files, neither of which existed before.\n\nlessons learned: keep an out-of-band way in, bind containers to loopback because docker ignores ufw, treat a resumable agent with shell access as a second operator, measure cost as quota by querying task runs rather than billing, give every automation a timeout, and never automate deletion of something a service created until you have restarted the service and watched what it recreates.\n\nthe setup: openclaw gateway running as a root user-level systemd service on an 8GiB, 4-core linux VM, talks to a telegram bot, uses codex subscription auth. several project repos and docker containers live on the same host.\n\n| step | what happened | result | \n|---|---|---|\n| 1 | I asked the agent to audit itself, then to execute the remediation | agent started `openclaw update` and went silent | \n| 2 | upgrade installed 2026.9.5, stopped the gateway, post-upgrade doctor aborted on the codex plugin | telegram dead | \n| 3 | the plugin build failed with ENOSPC because /tmp was a 3.8GiB RAM disk | moved TMPDIR to disk | \n| 4 | session migration blocked by leftover JSONL transcripts; a file move changed inodes and broke the migration fingerprint | fixed fingerprint in sqlite, doctor passed, gateway back | \n| 5 | discovered the agent itself was concurrently running repair scripts and force-killing the gateway | coordinated instead of fighting | \n| 6 | firewall for docker ports, swap file, log pruning, plugin build cleanup | host stabilized | \n| 7 | switched primary model to GPT-6 Astra | verified with test turns | \n| 8 | efficiency audit: 1,300+ LLM turns by 3 watchdog cron jobs in a week | added rules to ensure heartbeats were bounded | \n| 9 | my own cleanup timer deleted live plugin build dirs; every turn failed for ~2 hours | rewrote the policy | \n\n**symptom.** mid-upgrade, my telegram bot stopped responding to me. no error, nothing loading, it just stopped responding.\n\n**how to look.**\n\n```\nopenclaw --version                              # what actually got installed\nsystemctl --user list-units 'openclaw*' --all   # is the gateway running?\njournalctl --user -u openclaw-gateway -n 200    # what happened before it died\n```\n\nthe gateway unit was `inactive (dead)`. the upgrade log — a transient `systemd-run` unit the agent had created for itself:\n\n```\n...\nUpdated post-plugin Doctor failed: Plugin \"codex\" state migration is pending:\nThe configured plugin package is missing or has not converged.\n```\n\n**what happened here** `openclaw update` stops the gateway, runs the doctor, and only restarts on success. so when the doctor failed, nothing restarted the gateway, and there was no automatic rollback. the upgrade just left the thing off.\n\n**what I learned.** this is probably obvious, and originally I wasn't going to let the agent upgrade itself, I just wanted to use it to help make the plans for the upgrade to hand off to another agent, but once it made the plans it seemed confident it could just do it, so I was like okay cool go for it. that was a mistake. so the learning is: do NOT let the agent upgrade itself over the same channel you use to talk to it (unless you have a second way in, I guess if I had had a second telegram bot set up I could have then used that to fix it, but I didn't so I had to wait until I was home to manually ssh in and fix it). additionally, this is risky because the second it stops the gateway, it has no way to tell you what went wrong.\n\n**symptom.** after I got the gateway running again, the codex plugin refused to load with the error: `ENOSPC: no space left on device, write`. but `df` showed plenty of room.\n\n**cause.** `/tmp` was a tmpfs (a RAM disk) of 3.8GiB. and it was already 73% full of old temp files. I didn't realize this, but openclaw builds a copy of each plugin package under `os.tmpdir()` every time it loads one, and the codex plugin's copy is 342MB. so a few concurrent builds overflowed it.\n\n**fix.** a systemd drop-in for the gateway service:\n\n```\n# ~/.config/systemd/user/openclaw-gateway.service.d/20-cache-paths.conf\n[Service]\nEnvironment=TMPDIR=/var/tmp/openclaw-tmp\nEnvironment=NODE_COMPILE_CACHE=/var/tmp/openclaw-compile-cache\n```\n\nand `export TMPDIR=/var/tmp/openclaw-tmp` before any `openclaw` CLI command, because the CLI builds plugins too. drop-ins survive `openclaw gateway install --force`, which rewrites the main unit file out from under you.\n\n**lesson.** check `findmnt /tmp` on any VM before you run large builds there. a RAM-backed `/tmp` also quietly eats memory you think you have: 2.8GiB of the \"used\" RAM on this box was temp files.\n\n**symptom.** `openclaw doctor --fix` kept stopping in the same place:\n\n```\nSQLite-backed session still has an unverified active JSONL transcript file ...\nDoctor stopped because a state migration refused to continue.\n```\n\nthe suggested `openclaw doctor --session-sqlite recover` did nothing at all (`restored=0`). meanwhile the codex plugin's \"retained state migration\" couldn't finish until this settled, so the gateway stayed in degraded mode.\n\n**how I verified the files were safe to move.** for each flagged JSONL, I compared its line count against the event count in sqlite:\n\n``` python\npython3 - <<'EOF'\nimport sqlite3, os\nbase = os.path.expanduser('~/.openclaw/agents/main')\nc = sqlite3.connect(f\"file:{base}/agent/openclaw-agent.sqlite?mode=ro\", uri=True)\ncounts = dict(c.execute('select session_id, count(*) from transcript_events group by session_id'))\nfor fn in os.listdir(f\"{base}/sessions\"):\n    if fn.endswith('.jsonl') and fn[:-6] in counts:\n        n = sum(1 for _ in open(f\"{base}/sessions/{fn}\", 'rb'))\n        print(fn, n, counts[fn[:-6]], 'MATCH' if n == counts[fn[:-6]] else 'MISMATCH')\nEOF\n```\n\nall 55 matched exactly. moving them to a backup folder let the codex migration complete on the next `doctor --fix`, which felt like the end of it.\n\n**what went wrong next.** the doctor then refused with `retained_plugin_source_conflict: source changed`. I copied the files back byte-for-byte and it still refused. the reason is that the migration fingerprint in `~/.openclaw/state/openclaw.sqlite` — tables `migration_runs` and `migration_sources` — records `dev`, `ino`, `mtimeNs`, `size`, and `sha256` for every source file, and a move-and-copy changes the inode. the bytes were identical and the fingerprint still didn't match. I backed up the state DB and updated the 55 `ino` values in the stored JSON to the current ones. the doctor completed, archived 129 legacy transcripts on its own, and the gateway came up clean.\n\n**lesson.** do NOT move openclaw session files, even temporarily. if you have to touch them, `cp -a` to a backup and leave the originals where they are. the doctor fingerprints inodes, not contents.\n\n**symptom.** I stopped the gateway to run a repair, but it would keep coming back up on its own, get force-killed, and then leave 6GB of half-built plugin copies behind.\n\n**cause.** every gateway start runs \"main-session-restart-recovery\", which resumes whatever the agent was doing when it got interrupted. which in my case was \"execute the remediation plan\". so every time I restarted the gateway, the agent would come back up, write another repair script under `~/.openclaw/recovery/`, scheduled it with `systemd-run`, and then that script sent `SIGKILL` to the gateway to get exclusive access to the sqlite files. so essentially I was fighting my own instructions.\n\n**how to detect it:**\n\n```\nls -la ~/.openclaw/recovery/\nsystemctl --user list-units 'openclaw-*' --all     # transient units the agent created\njournalctl --user | grep -E 'systemd-run|SIGKILL'\n```\n\n**lesson.** a resumable agent with shell access is a second operator. before doing manual editing, you should wait for its turn to end (the log says `restart recovery terminal`) or tell it in chat to pause. its scripts reached the same TMPDIR conclusion I did, so reading them saved me time.\n\n**finding.** the original audit flagged 4 ports as reachable on all interfaces: 2 postgres databases, a minio API, and a minio console. `ufw` was set to deny all incoming except tailscale, so I had assumed they were covered. I was wrong. \n\n**why they were exposed.** docker inserts its own iptables rules ahead of ufw when you publish a port, so `-p 55432:5432` in a `docker run`, or `'5433:5432'` in a compose file, is reachable from the internet no matter what ufw's policy says. the `DOCKER-USER` chain, which is where you're meant to put your own rules, was empty. one of those databases had been started by hand with an 8-character password.\n\n**fix that survives reboot.** append to `/etc/ufw/after.rules` and `after6.rules`:\n\n```\n*filter\n:DOCKER-USER - [0:0]\n-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN\n-A DOCKER-USER -i tailscale0 -j RETURN\n-A DOCKER-USER -i eth0 -m conntrack --ctstate NEW -j DROP\n-A DOCKER-USER -j RETURN\nCOMMIT\n```\n\nthen `ufw reload`. replace `eth0` with your public interface from `ip route show default`. that blocks new inbound connections to any published container port on the public interface while still allowing them over tailscale.\n\nthen fix the source instead of leaving the firewall: bind ports to loopback in compose files (`'127.0.0.1:5433:5432'`) and in `docker run` (`-p 127.0.0.1:55432:5432`), and use long random passwords even for test databases (` openssl rand -base64 30`). nothing running on the host noticed the change — not the app, not the tests, not tools over tailscale SSH — because all of them already connect via localhost.\n\n**metric.** ports listening on `0.0.0.0` other than SSH: 4 before, 0 after.\n\n`vm.swappiness=10`. under later pressure 3.3GiB got paged out instead of the OOM killer firing.`codex-home/logs_2.sqlite` had grown to 1.2GB of TRACE/DEBUG rows in 10 days. I deleted rows older than 2 days and ran `VACUUM` with `auto_vacuum=INCREMENTAL`: 1,229MB down to 153MB. then `Environment=RUST_LOG=info` in another drop-in to cut the volume at the source. these logs only feed codex's \"send feedback\" bug reports, so there's nothing to lose. an hourly job now prunes rows older than 7 days.`openclaw doctor` run, even a read-only one, left 3 build directories of 342MB each behind. over 1 afternoon that was 30 directories and 11GB. section 9 is how cleaning this up went wrong before it went right.\nthe agent was on GPT-5.6 Sol with Terra and Luna as fallbacks. `openclaw models list --provider openai` showed `openai/gpt-6-astra` (released 2026-09-03) already in the catalog and already authenticated, so the switch was 4 steps:\n\n`agents.defaults.models` and `agents.defaults.modelPolicy.allow`, because the allowlist blocks overrides otherwise.` openclaw agent --agent main --session-key agent:main:model-test --model openai/gpt-6-astra -m \"Reply with exactly: OK\"`.` agents.defaults.model.primary` and reorder the fallbacks. the gateway hot-reloads config so no restart.`openclaw sessions delete --agent main agent:main:model-test --yes`.\nI kept Luna as `utilityModel`, `heartbeat.model`, and `subagents.model` so my background work can remain cheap. test turns came back in about 6 seconds.\n\nquota on a codex subscription is measured in 5-hour and weekly windows, and `openclaw gateway usage-cost` reports $0 because nothing is metered per token. so the real signal is `task_runs` in the state database:\n\n``` python\npython3 - <<'EOF'\nimport sqlite3, os, time\nc = sqlite3.connect('file:' + os.path.expanduser('~/.openclaw/state/openclaw.sqlite') + '?mode=ro', uri=True)\nsince = int((time.time() - 7*86400) * 1000)\nfor r in c.execute(\"select label, status, count(*) from task_runs where created_at > ? group by label, status order by 3 desc limit 10\", (since,)):\n    print(r)\nEOF\n```\n\n1 week looked like this:\n\n| automation | LLM runs | \n|---|---|\n| my-menulet staged implementation watchdog | 1,209 | \n| guides-gallery-field-watchdog | 106 | \n| guides-testflight-build-16-monitor | 28 | \n| actual conversation with me | a few dozen | \n\n3 cron jobs the agent had created to \"monitor\" things had been polling with the full model every few minutes for days, reporting \"still blocked\" each time. roughly 95% of the week's turns went to that.\n\n**also found.** compaction of the main telegram session was timing out at 180 seconds and getting cancelled, so every turn was sending about 114k tokens of history. so I raised `agents.defaults.compaction.timeoutSeconds` to 600. one note if you're on a codex-backed agent: don't set `compaction.model`. codex owns compaction natively and the doctor strips that key back out.\n\n**rules now in the agent's AGENTS.md:**\n\n`/new` to the user when a project task wraps up. context past ~50k tokens per turn is waste.\n**heartbeats.** the reader agent's heartbeat had been running every 30 minutes for no purpose. I set `heartbeat.every: \"0m\"`, which disables the cadence, plus `activeHours` 08:00 to 22:00 and `lightContext: true` so that re-enabling it later stays bounded.\n\n**disk hygiene, same audit.** the workspace was 74GB: 31 worktrees plus 24 clones of one repo, and 33GB of duplicated `node_modules`. working out which were safe to remove:\n\n```\ncd projects/tro-net && git fetch origin\ngit branch -r --merged origin/main             # merged branches\ngit worktree list                              # all checkouts, including ones under projects/\ngit -C <checkout> status --porcelain           # must be empty (or only generated files)\ngit -C <checkout> log origin/main..HEAD        # must be empty (no unpushed commits)\n```\n\nI removed 7 merged worktrees and 2 merged checkouts with `git worktree remove --force`, then `git worktree prune`. that freed up 4GB, not the 9GB `du` had suggested, because pnpm had hardlinked files into a shared store and `du` was double-counting them. the other thing this exposed: the repo was actually an npm project — fresh `package-lock.json`, 4-month-old `pnpm-lock.yaml` — and the agent had been generating pnpm lockfiles inside it. don't infer the package manager from which lockfiles exist. check which one is committed and current.\n\n**symptom, 2 hours after everything was working.** every telegram message came back with \"Something went wrong while processing your request\", and `/new` didn't help.\n\n**cause.** my hourly cleanup timer removed plugin build directories older than 60 minutes. the running gateway keeps resolving its loaded plugins out of the build directories it created at startup, and it holds no open file handle on them, so `lsof` shows you nothing. when the timer removed them, every turn failed with `ENOENT ... openclaw-plugin-build-XXXX/.../@openclaw/codex/package.json`, and all 3 fallback models failed identically because they share the plugin.\n\n**fix.** `systemctl --user restart openclaw-gateway` rebuilt the directories immediately. the cleanup script now reads the gateway's start time (`ps -o etimes= -p <gateway pid>`) and only deletes build directories with an mtime older than that. anything created during the current gateway's lifetime is never touched, and after a restart the previous gateway's directories become deletable. I tested it by running the script with a 0-minute threshold — live dirs survived — and against a fake 3-hour-old directory, which it removed.\n\n**lesson.** \"no process has it open\" is not the same as \"nothing depends on it\". before you automate deletion of anything a long-running service created, restart the service and see whether it recreates it.\n\nthere were 0 backup runs recorded and the workspace git repo had 0 commits, which meant the agent's identity, instructions, and memory notes were all unversioned. I fixed this with:\n\n```\nopenclaw backup git init --repository ~/openclaw-backups-git\nopenclaw backup git create --repository ~/openclaw-backups-git --all --exclude-secrets\nopenclaw backup enable --repository ~/openclaw-backups-git --every 24h --exclude-secrets\ncd ~/.openclaw/workspace && printf 'projects/\\n.worktrees/\\n' > .gitignore && git add -A && git commit -m \"Initial commit of workspace continuity files\"\n```\n\nfirst backup: 301MB. it runs daily as an openclaw automation now.\n\n| measure | before | after | \n|---|---|---|\n| openclaw version | 2026.7.1-2 | 2026.9.5 | \n| primary model | GPT-5.6 Sol | GPT-6 Astra | \n| swap | none | 4GiB | \n| container ports on public interface | 4 | 0 | \n| codex log database | 1,229MB | 153MB | \n| enabled plugins | 51 | 12 (allowlist) | \n| unattended LLM runs per week | ~1,340 | a nightly memory job and a weekly skill review | \n| workspace continuity files under version control | no | yes, plus daily DB backups | \n| disk used | 111GB (peaked at 122GB during repair) | 107GB | \n\n`127.0.0.1` and put drop rules in `DOCKER-USER`.`/var/tmp`.` task_runs`, not billing. LLM-based watchdog crons are the most expensive thing you can accidentally create.", "url": "https://wpnews.pro/news/upgrading-and-recovering-my-self-hosted-openclaw-agent-telegram-bot", "canonical_source": "https://dev.to/emalia/upgrading-and-recovering-my-self-hosted-openclaw-agent-telegram-bot-oj", "published_at": "2026-09-26 23:20:53+00:00", "updated_at": "2026-09-27 00:00:59.832122+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "mlops", "ai-infrastructure"], "entities": ["openclaw", "Telegram", "Claude Code", "GPT-6 Astra", "Codex", "Docker", "systemd"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/upgrading-and-recovering-my-self-hosted-openclaw-agent-telegram-bot", "markdown": "https://wpnews.pro/news/upgrading-and-recovering-my-self-hosted-openclaw-agent-telegram-bot.md", "text": "https://wpnews.pro/news/upgrading-and-recovering-my-self-hosted-openclaw-agent-telegram-bot.txt", "jsonld": "https://wpnews.pro/news/upgrading-and-recovering-my-self-hosted-openclaw-agent-telegram-bot.jsonld"}}