WorkerDeck – Run one isolated background process per user (Python, zero deps)

WorkerDeck, a new open-source Python library, enables developers to run one isolated background process per user with zero dependencies. The tool handles process lifecycle management, including start, stop, restart, and self-healing with circuit breakers, and includes a Flask dashboard for monitoring. It is designed for platforms where each user requires a long-lived process such as an AI agent or data pipeline.

Run one isolated background process per user — and manage them all from one dashboard. Zero dependencies: the core is pure Python standard library. If you're building a platform where every user runs their own long-lived process — an AI agent, a data pipeline, a scheduled job, a bot, a scraper — you hit the same wall: starting one process per user, keeping them isolated, knowing which are alive, restarting the ones that died, and stopping them cleanly. It's fiddly, it's easy to get subtly wrong, and it has nothing to do with your actual product. WorkerDeck is that layer, done once. You bring the worker; it handles the lifecycle. ┌─────────────────────────────────────────────┐ │ Dashboard │ │ ● Ada running pid 805 up 6s ⏹ │ │ ● Linus running pid 807 up 6s ⏹ │ │ ○ Grace stopped ▶ │ └─────────────────────────────────────────────┘ │ start / stop / restart / inspect ▼ ┌─────────────────────────────────────────────┐ │ WorkerManager │ │ user 1 → its own process, dir, config, log │ │ user 2 → its own process, dir, config, log │ │ user 3 → … │ └─────────────────────────────────────────────┘ One process per user, isolated. Each user's worker runs in its own directory, with its own config/secrets injected as environment variables, writing its own log. Users never see each other's anything. Safe lifecycle. Start, stop, restart. Graceful shutdown SIGTERM that escalates to a kill if a worker ignores it — and signals the whole process group, so children die too. No orphans. Survives its own restart. This is the one most process managers get wrong. When WorkerDeck itself restarts — a deploy, a code reload, a crash — your workers keep running, and the new manager re-adopts them instead of reporting them as dead. It matches processes by pid and start time , so a recycled pid can never be mistaken for your worker. More below — this is worth understanding. Self-healing, safely opt-in . Turn on auto restart and crashed workers come back on their own — but behind a circuit breaker: exponential backoff between attempts, and after too many crashes in a window the worker is parked in a failed state for a human, instead of hot-looping and taking the box down with it. Live status. Running, stopped, crashed, or failed — since when, last log lines, per user, at a glance. Zombie defunct processes are reported as gone, not falsely "running". Reconciliation. One reconcile call compares what should be running against what actually is, cleans up stale bookkeeping, and if enabled recovers crashed workers. Point a cron job or timer at it and walk away. A dashboard, included. A small Flask control panel so you can see and drive everything without writing your own UI first. Your worker is just a Python script. WorkerDeck doesn't care what it does. Swap the example for yours and the lifecycle stays identical. pip install workerdeck That's the engine — zero dependencies, one class. See Use it in your own code use-it-in-your-own-code below. The included Flask dashboard is a demo/dev tool, not part of the library. Run it from a clone of this repo: git clone https://github.com/gritfeld-design/workerdeck.git cd workerdeck pip install -e ". dashboard " python -m dashboard.app open http://localhost:5000 You'll see three demo users. Hit Start on one and watch its log stream in. The worker it runs is examples/scheduled task worker.py /gritfeld-design/workerdeck/blob/main/examples/scheduled task worker.py — a trivial task on a timer, standing in for whatever yours will do. The dashboard is a demo; the engine is the point. It's one class: python from workerdeck.manager import WorkerManager mgr = WorkerManager worker script="path/to/your worker.py", work root="./ work", start a user's worker, injecting their own config as env vars mgr.start user id=42, config={"API KEY": user key, "INTERVAL": "30"} check on it status = mgr.status 42 print status.state, status.uptime seconds, status.last log lines stop it cleanly mgr.stop 42 Your worker reads its per-user config from the environment and runs until WorkerDeck stops it: python import os, signal, sys api key = os.environ "API KEY" this user's own key user id = os.environ "WORKER USER ID" always provided running = True signal.signal signal.SIGTERM, lambda : globals . setitem "running", False while running: do one unit of work api key ← your work here ... sys.exit 0 That's the whole contract. See examples/scheduled task worker.py /gritfeld-design/workerdeck/blob/main/examples/scheduled task worker.py for a complete, runnable version. Here's the failure mode WorkerDeck exists to prevent. A naive manager tracks its workers in memory — a dict of process handles. That works right up until the manager process itself restarts you deployed, the code reloaded, it crashed and came back . Now that dict is empty. The workers are all still running — they're separate processes — but the manager has amnesia. It reports them as stopped. Click "start" and you get a second copy fighting the first for the same port. Click "stop" and nothing happens, because the manager no longer holds their handles. WorkerDeck avoids this by persisting a small registry to disk — each worker's pid and start time — and re-adopting live workers on startup: mgr = WorkerManager worker script="worker.py", work root="./ work" mgr.start 42 worker running as some pid ... later, the manager process is restarted entirely ... mgr = WorkerManager worker script="worker.py", work root="./ work" mgr.status 42 .state "running" — it re-adopted the live worker mgr.stop 42 and yes, it can still stop it Why start time and not just pid? Because the OS recycles pid numbers. A pid that was your worker yesterday might be an unrelated process today. A live pid alone is not proof; a live pid whose start time matches what we recorded is . That check is the difference between "usually works" and "safe to restart". Crashes happen. Turn on auto restart and let reconcile bring workers back — with a brake so a genuinely-broken worker can't spin forever: mgr = WorkerManager worker script="worker.py", work root="./ work", auto restart=True, max crashes=5, give up after 5 crashes... crash window=300, ...within 5 minutes run this on a timer / cron. Each call: - reports current state per user - cleans up dead bookkeeping - restarts crashed workers backoff between tries - parks hopeless ones in state "failed" no hot-loop report = mgr.reconcile after you've fixed a failed worker's root cause: mgr.reset failed 42 Backoff is exponential 1s, 2s, 4s, …, capped , so a worker that crashes on startup doesn't hammer your machine. Once it trips the breaker it stays failed — visible, not silently retried into oblivion — until you clear it. Pass an on event callback and WorkerDeck tells you about every meaningful state change — wire it to your logging or your alerting: python def on event ev : ev.user id, ev.type, ev.timestamp, ev.detail if ev.type == "failed": alert f"Worker for user {ev.user id} gave up after " f"{ev.detail 'crashes' } crashes" log.info " %s user %s %s", ev.type, ev.user id, ev.detail mgr = WorkerManager worker script="w.py", work root="./ work", on event=on event Event types: started , stopped , crashed , restarted , failed , and adopted a live worker re-attached after the manager restarted . Each carries relevant detail — pid, crash count, backoff seconds. A callback that raises is caught and logged, never allowed to take the manager down: your notifications can be buggy without breaking your workers. It's a foundation , not a finished platform. It deliberately leaves the parts that are specific to your product to you: Auth & users — the demo uses three hardcoded users. Bring your own users table and session auth; the WorkerManager calls don't change. Persistence of — WorkerDeck tracks process lifecycle, not your domain data. Store config in your own database. what each user runs Scaling across machines — this manages workers on one host. Multi-host orchestration is out of scope and usually premature . It handles the annoying, easy-to-get-wrong part — per-user process lifecycle on a single host — so you can get to the part that's actually your product. Because that's where most projects actually start, and because the failure modes of per-user subprocess management are real and annoying long before you need Kubernetes: - a worker that ignores SIGTERM → escalated to SIGKILL - child processes orphaned when a worker dies → whole process group signalled - a zombie defunct process that looks alive but isn't → reported as gone - one user's worker crashing → isolated, and optionally auto-restarted - the manager restarting after a deploy and losing track of everything → workers re-adopted by pid + start time Every one of those is handled here, on one host, with no orchestration layer. Solve the real problem you have now. git clone https://github.com/gritfeld-design/workerdeck.git cd workerdeck pip install -e ". dev " pytest -v The test suite exercises the library's core claims: the start/stop lifecycle, re-adoption of live workers after a manager restart matched by pid and start time , rejection of stale registry entries, the crash circuit breaker, and the event callback. Tests treat zombie defunct processes as dead — using the library's own liveness check — because a naive os.kill pid, 0 reports zombies as alive. Ask me how I know. MIT — use it, change it, ship it. WorkerDeck started life as the process-management core of a real multi-user platform, extracted and generalized. If it saves you the week it took to get right the first time, that's the point. "Running per-user workers in production? I'm collecting failure modes — open an issue or mail me."