{"slug": "workerdeck-run-one-isolated-background-process-per-user-python-zero-deps", "title": "WorkerDeck – Run one isolated background process per user (Python, zero deps)", "summary": "WorkerDeck, a new open-source Python library, enables developers to run one isolated background process per user with zero dependencies. The tool handles process lifecycle management, including start, stop, restart, and self-healing with circuit breakers, and includes a Flask dashboard for monitoring. It is designed for platforms where each user requires a long-lived process such as an AI agent or data pipeline.", "body_md": "**Run one isolated background process per user — and manage them all from one dashboard.**\n**Zero dependencies: the core is pure Python standard library.**\n\nIf you're building a platform where every user runs their own long-lived process — an AI agent, a data pipeline, a scheduled job, a bot, a scraper — you hit the same wall: starting one process per user, keeping them isolated, knowing which are alive, restarting the ones that died, and stopping them cleanly. It's fiddly, it's easy to get subtly wrong, and it has nothing to do with your actual product.\n\nWorkerDeck is that layer, done once. You bring the worker; it handles the lifecycle.\n\n```\n┌─────────────────────────────────────────────┐\n│  Dashboard                                   │\n│   ● Ada      running   pid 805  up 6s   [⏹]  │\n│   ● Linus    running   pid 807  up 6s   [⏹]  │\n│   ○ Grace    stopped                    [▶]  │\n└─────────────────────────────────────────────┘\n         │ start / stop / restart / inspect\n         ▼\n┌─────────────────────────────────────────────┐\n│  WorkerManager                               │\n│   user 1 → its own process, dir, config, log │\n│   user 2 → its own process, dir, config, log │\n│   user 3 → …                                 │\n└─────────────────────────────────────────────┘\n```\n\n**One process per user, isolated.** Each user's worker runs in its own directory, with its own config/secrets injected as environment variables, writing its own log. Users never see each other's anything.**Safe lifecycle.** Start, stop, restart. Graceful shutdown (SIGTERM) that escalates to a kill if a worker ignores it — and signals the whole process group, so children die too. No orphans.**Survives its own restart.** This is the one most process managers get wrong. When*WorkerDeck itself*restarts — a deploy, a code reload, a crash — your workers keep running, and the new manager re-adopts them instead of reporting them as dead. It matches processes by pid**and start time**, so a recycled pid can never be mistaken for your worker. (More below — this is worth understanding.)** Self-healing, safely (opt-in).**Turn on`auto_restart`\n\nand crashed workers come back on their own — but behind a circuit breaker: exponential backoff between attempts, and after too many crashes in a window the worker is parked in a`failed`\n\nstate for a human, instead of hot-looping and taking the box down with it.**Live status.** Running, stopped, crashed, or failed — since when, last log lines, per user, at a glance. Zombie (defunct) processes are reported as gone, not falsely \"running\".**Reconciliation.** One`reconcile()`\n\ncall compares what should be running against what actually is, cleans up stale bookkeeping, and (if enabled) recovers crashed workers. Point a cron job or timer at it and walk away.**A dashboard, included.** A small Flask control panel so you can see and drive everything without writing your own UI first.**Your worker is just a Python script.** WorkerDeck doesn't care what it does. Swap the example for yours and the lifecycle stays identical.\n\n```\npip install workerdeck\n```\n\nThat's the engine — zero dependencies, one class. See\n[Use it in your own code](#use-it-in-your-own-code) below.\n\nThe included Flask dashboard is a demo/dev tool, not part of the library. Run it from a clone of this repo:\n\n```\ngit clone https://github.com/gritfeld-design/workerdeck.git\ncd workerdeck\npip install -e \".[dashboard]\"\npython -m dashboard.app\n# open http://localhost:5000\n```\n\nYou'll see three demo users. Hit **Start** on one and watch its log stream\nin. The worker it runs is [ examples/scheduled_task_worker.py](/gritfeld-design/workerdeck/blob/main/examples/scheduled_task_worker.py)\n— a trivial task on a timer, standing in for whatever yours will do.\n\nThe dashboard is a demo; the engine is the point. It's one class:\n\n``` python\nfrom workerdeck.manager import WorkerManager\n\nmgr = WorkerManager(\n    worker_script=\"path/to/your_worker.py\",\n    work_root=\"./_work\",\n)\n\n# start a user's worker, injecting their own config as env vars\nmgr.start(user_id=42, config={\"API_KEY\": user_key, \"INTERVAL\": \"30\"})\n\n# check on it\nstatus = mgr.status(42)\nprint(status.state, status.uptime_seconds, status.last_log_lines)\n\n# stop it cleanly\nmgr.stop(42)\n```\n\nYour worker reads its per-user config from the environment and runs until WorkerDeck stops it:\n\n``` python\nimport os, signal, sys\n\napi_key = os.environ[\"API_KEY\"]           # this user's own key\nuser_id = os.environ[\"WORKER_USER_ID\"]    # always provided\n\nrunning = True\nsignal.signal(signal.SIGTERM, lambda *_: globals().__setitem__(\"running\", False))\n\nwhile running:\n    do_one_unit_of_work(api_key)          # ← your work here\n    ...\n\nsys.exit(0)\n```\n\nThat's the whole contract. See [ examples/scheduled_task_worker.py](/gritfeld-design/workerdeck/blob/main/examples/scheduled_task_worker.py)\nfor a complete, runnable version.\n\nHere's the failure mode WorkerDeck exists to prevent.\n\nA naive manager tracks its workers in memory — a dict of process handles.\nThat works right up until the manager process itself restarts (you deployed,\nthe code reloaded, it crashed and came back). Now that dict is empty. The\nworkers are all still running — they're separate processes — but the manager\nhas amnesia. It reports them as stopped. Click \"start\" and you get *a second\ncopy* fighting the first for the same port. Click \"stop\" and nothing happens,\nbecause the manager no longer holds their handles.\n\nWorkerDeck avoids this by persisting a small registry to disk — each worker's\npid and **start time** — and re-adopting live workers on startup:\n\n```\nmgr = WorkerManager(worker_script=\"worker.py\", work_root=\"./_work\")\nmgr.start(42)                 # worker running as some pid\n\n# ... later, the manager process is restarted entirely ...\n\nmgr = WorkerManager(worker_script=\"worker.py\", work_root=\"./_work\")\nmgr.status(42).state          # \"running\" — it re-adopted the live worker\nmgr.stop(42)                  # and yes, it can still stop it\n```\n\nWhy start time and not just pid? Because the OS recycles pid numbers. A pid\nthat was your worker yesterday might be an unrelated process today. A live pid\nalone is not proof; a live pid whose start time matches what we recorded *is*.\nThat check is the difference between \"usually works\" and \"safe to restart\".\n\nCrashes happen. Turn on `auto_restart`\n\nand let `reconcile()`\n\nbring workers\nback — with a brake so a genuinely-broken worker can't spin forever:\n\n```\nmgr = WorkerManager(\n    worker_script=\"worker.py\",\n    work_root=\"./_work\",\n    auto_restart=True,\n    max_crashes=5,        # give up after 5 crashes...\n    crash_window=300,     # ...within 5 minutes\n)\n\n# run this on a timer / cron. Each call:\n#   - reports current state per user\n#   - cleans up dead bookkeeping\n#   - restarts crashed workers (backoff between tries)\n#   - parks hopeless ones in state \"failed\" (no hot-loop)\nreport = mgr.reconcile()\n\n# after you've fixed a failed worker's root cause:\nmgr.reset_failed(42)\n```\n\nBackoff is exponential (1s, 2s, 4s, …, capped), so a worker that crashes on\nstartup doesn't hammer your machine. Once it trips the breaker it stays\n`failed`\n\n— visible, not silently retried into oblivion — until you clear it.\n\nPass an `on_event`\n\ncallback and WorkerDeck tells you about every meaningful\nstate change — wire it to your logging or your alerting:\n\n``` python\ndef on_event(ev):\n    # ev.user_id, ev.type, ev.timestamp, ev.detail\n    if ev.type == \"failed\":\n        alert(f\"Worker for user {ev.user_id} gave up after \"\n              f\"{ev.detail['crashes']} crashes\")\n    log.info(\"[%s] user %s %s\", ev.type, ev.user_id, ev.detail)\n\nmgr = WorkerManager(worker_script=\"w.py\", work_root=\"./_work\",\n                    on_event=on_event)\n```\n\nEvent types: `started`\n\n, `stopped`\n\n, `crashed`\n\n, `restarted`\n\n, `failed`\n\n, and\n`adopted`\n\n(a live worker re-attached after the manager restarted). Each\ncarries relevant `detail`\n\n— pid, crash count, backoff seconds. A callback\nthat raises is caught and logged, never allowed to take the manager down:\nyour notifications can be buggy without breaking your workers.\n\nIt's a **foundation**, not a finished platform. It deliberately leaves the\nparts that are specific to *your* product to you:\n\n**Auth & users**— the demo uses three hardcoded users. Bring your own users table and session auth; the`WorkerManager`\n\ncalls don't change.**Persistence of**— WorkerDeck tracks process lifecycle, not your domain data. Store config in your own database.*what*each user runs**Scaling across machines**— this manages workers on one host. Multi-host orchestration is out of scope (and usually premature).\n\nIt handles the annoying, easy-to-get-wrong part — per-user process lifecycle on a single host — so you can get to the part that's actually your product.\n\nBecause that's where most projects actually start, and because the failure modes of per-user subprocess management are real and annoying long before you need Kubernetes:\n\n- a worker that\n**ignores SIGTERM**→ escalated to SIGKILL - child processes\n**orphaned** when a worker dies → whole process group signalled - a\n**zombie**(defunct) process that looks alive but isn't → reported as gone - one user's worker\n**crashing**→ isolated, and optionally auto-restarted - the manager\n**restarting after a deploy** and losing track of everything → workers re-adopted by pid + start time\n\nEvery one of those is handled here, on one host, with no orchestration layer. Solve the real problem you have now.\n\n```\ngit clone https://github.com/gritfeld-design/workerdeck.git\ncd workerdeck\npip install -e \".[dev]\"\npytest -v\n```\n\nThe test suite exercises the library's core claims: the start/stop\nlifecycle, re-adoption of live workers after a manager restart (matched by\npid **and** start time), rejection of stale registry entries, the crash\ncircuit breaker, and the event callback. Tests treat zombie (defunct)\nprocesses as dead — using the library's own liveness check — because a\nnaive `os.kill(pid, 0)`\n\nreports zombies as alive. Ask me how I know.\n\nMIT — use it, change it, ship it.\n\n*WorkerDeck started life as the process-management core of a real multi-user\nplatform, extracted and generalized. If it saves you the week it took to get\nright the first time, that's the point.*\n\n\"Running per-user workers in production? I'm collecting failure modes — open an issue or mail me.\"", "url": "https://wpnews.pro/news/workerdeck-run-one-isolated-background-process-per-user-python-zero-deps", "canonical_source": "https://github.com/gritfeld-design/workerdeck", "published_at": "2026-07-14 12:06:57+00:00", "updated_at": "2026-07-14 12:19:29.212061+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["WorkerDeck", "Gritfeld Design"], "alternates": {"html": "https://wpnews.pro/news/workerdeck-run-one-isolated-background-process-per-user-python-zero-deps", "markdown": "https://wpnews.pro/news/workerdeck-run-one-isolated-background-process-per-user-python-zero-deps.md", "text": "https://wpnews.pro/news/workerdeck-run-one-isolated-background-process-per-user-python-zero-deps.txt", "jsonld": "https://wpnews.pro/news/workerdeck-run-one-isolated-background-process-per-user-python-zero-deps.jsonld"}}