Gemma couldn't tell my robot was upside down, so I gave it a head camera. Now I sleep while it trains.

A robotics master's student built Night Watch, an open-source overnight monitoring system for reinforcement-learning robot training runs that combines a local Gemma 4 vision model, TabPFN metric anomaly detection, rule-based crash checks, and a Temporal workflow for durability. In a 600,000-step MuJoCo HalfCheetah run, the system caught a sign error in the leveling reward that left the robot training upside down while the reward curve climbed from 908 to 3,525, waking the developer at step 100,000 with a posture=upside_down alert. The MIT-licensed project runs on a Windows laptop with no GPU at $0 a night and caps wake-ups at three per night, deferring everything else to a morning report.

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady https://dev.to/challenges/hacktoberfest-week3-2026-10-19 At step 100,000, my robot was lying on its back with its legs in the air, kicking. The reward curve said it was having its best run yet. | step 0 | step 100,000 | |---|---| I'd written a reward term to keep its body level and got the sign wrong, so it was paid for tilting. Over 600,000 training steps the reward climbed from 908 to 3,525. Every alert I'd normally set reward dropped, KL divergence spiked, loss exploded stayed quiet, because nothing went down. TL;DR: Night Watch watches a robot-learning run so you can sleep through it. A local Gemma 4 looks through a camera bolted to the robot, TabPFN reads the training metrics, plain rules catch crashes, and a Temporal workflow keeps the watch alive even when the watcher itself dies. It wakes you at most 3 times a night; everything else waits for a one-page morning report. Open models only, on a laptop with no GPU card, $0 a night. I'm a robotics master's student. Robot learning in simulation is slow and steady by nature: a policy trains for hours, so you start it, go to bed, and wake up at 2 a.m. to check on it, because runs die quietly a hung loader, a NaN, a crash or go wrong while looking great, like the one above. Night Watch does the 2 a.m. check for me: It sends a notification ntfy or Telegram only when it's worth waking up for: at most 3 times a night, and never twice for the same problem within an hour. Everything else goes into a morning report with a time-lapse of the run, the reward curve with every alert marked, and a three-sentence summary. The video is a second live run 25 Oct, same flip bug , and it went less cleanly. TabPFN raised the first alert at step 98,304 p=0.98 , but it pointed at training speed and value loss while the laptop was also recording and running out of memory, so I don't count it as a clean catch. Gemma's first looks failed for lack of memory; it woke me at step 401,408. The time-lapse from the first run's morning report, one frame per test episode across 600,000 steps: What woke me: 00:38 step 100,000 vision/robot looks wrong The robot looks wrong in the last 2 test runs: posture=upside down. Gemma: "head camera, floor/sky: top/bottom, top/bottom, top/bottom, top/bottom" 00:38 step 106,496 tabpfn/metrics anomaly Training metrics look like past runs that broke p=0.86 . Sleep while your robot learns. Two open models watch an overnight reinforcement-learning run, and you only get woken when it is worth it. In the morning you get a one-page report: a time-lapse of the night, the reward curve, and what if anything went wrong. Built for the DEV x Hacktoberfest 2026 Open-Source AI Challenge, Week 3: Slow and Steady . The repository was started and completed during the challenge window 19 to 25 October 2026 . A real 600k-step run on a Windows laptop i7-1165G7, no NVIDIA GPU … MIT licensed, built during the challenge. 32 tests, including a full 8-hour night through Temporal's time-skipping test server in a few seconds. GitHub Actions runs them on Linux and Windows on every push, plus short real training runs with injected faults. trainer PPO + MuJoCo HalfCheetah watcher a Temporal workflow, a check every few minutes every PPO update - metrics.jsonl ------ rules NaN? log gone silent? process dead? every 20k steps - a short test run ---- TabPFN do these metrics look like past runs that broke? 4 head-camera ------ Gemma 4 where are the floor and the sky? code decides the posture snapshots heartbeat.json - alive? policy wake me max 3 a night or note it for the morning labels/ hidden - ground truth report time-lapse, reward chart, summary with checked numbers The trainer writes the files a watcher may read metrics, snapshots, a heartbeat and a labels/ folder it may not: the fault I injected and when, and the robot's true body angle from the physics engine. Only the evaluation reads labels/ , so "Gemma caught the flip" is checked against the simulator, not against Gemma. Then I broke things on purpose: a flip bug and a spin bug shaping terms with the sign flipped , a learning rate x200 mid-run, a physics blow-up timestep x4, motor gear x3 , a hung training loop and a silent crash. My first version showed Gemma the obvious picture: the side view, four snapshots in a 2x2 grid. Before trusting it with a night, I tested its eyesight with nightwatch eyetest : drop the robot into a known pose the right way up, on its back, or crouching , let the physics settle it, and ask the same judge that runs at night. The simulator knows the answer. It called every robot "upright", including the six lying on their backs. Bigger pictures didn't help Ollama gives Gemma 4 about one image token per 48x48 pixels, so my grid snapshots were a few dozen tokens each; at 1024x768, 336 tokens, still "upright" . Neither did other questions "what is touching the floor?" , or thinking mode: six minutes per look, still wrong. The 12B model doesn't fit my laptop's GPU. Then I treated it as a robotics problem instead of a prompt problem. A real robot doesn't work out which way up it is from a photographer standing beside it; it has its own sensors. So I bolted a camera to the robot's torso, in a render-only copy of the model built with MuJoCo's MjSpec , so the physics the policy learns on is untouched. The camera draws only the world, not the robot, like an artificial horizon. When the robot flips, its camera flips with it, and the question becomes one a 4B model can answer: where is the floor in this picture? 18 test episodes 6 upright, 6 upside down, 6 crouching or rearing, which count as upright , timed on my laptop's Intel Iris Xe: | what Gemma is shown | right calls | upside-down robots caught | seconds per test episode | |---|---|---|---| | side camera, 2x2 sheet first version, 12 episodes | 6 / 12 | 0 / 6 | 10 | | side camera, full-size snapshots | 12 / 18 | 0 / 6 | 22 | | head camera, E4B | 18 / 18 | 6 / 6 | 25 | | head camera, the smaller E2B | 12 / 18 | 0 / 6 | 18 | Gemma answers one question per snapshot in a fixed JSON schema floor and sky: top half, bottom half, whole picture, a side, not visible . Plain code turns the four answers into a posture: sky below floor means upside down, the picture flipping between snapshots means spinning. Gemma never decides to wake me, and the same problem has to show up in two test episodes in a row. Every window of 8 training-log rows becomes one table row of 22 numbers: how far the reward is below its best, KL divergence and value loss relative to their usual level, the learning-rate ratio and so on, labelled 1 if the run was already broken at that point. TabPFN-3.5 learns in context from that table, so there's no extra training run to babysit which would have been ironic . It runs locally on the laptop CPU. The first version was too slow for a live watcher: a 7-second fit, but about 96 seconds per prediction, because TabPFN re-processed the whole table every time. Reading TabPFN's source turned up fit mode="fit with cache" : one 92-second fit at the start of the night, then under a second per check. The watch is a Temporal workflow: check, decide, maybe alert, wait on a durable timer, repeat until morning. Checks are activities with retries, so Ollama loading a model is just a retry. Every 150 checks it continues as new, carrying who woke me and when, and snooze , stop and status work from the command line. The test that matters is killing the watcher mid-watch. In the first run, with a check every 60 seconds: The second run on video did the same: worker killed at 11:12:21, timer fired at 11:12:58 with no worker, the new worker picked it up at 11:16:06. A plain while True: sleep 300 loop would have died silently at 00:42, which is the exact failure Night Watch exists to catch. Gemma writes the summary, but only from facts the code computed. A check pulls every number out of its paragraph, and if any number isn't in the facts, the paragraph is replaced by a plain template. The report says which one you got. The first run's summary is Gemma's, and it passed: The killtest run in HalfCheetah-v5 finished with 598016 steps done. The reward increased from 908 to 3525, matching the best reward. A problem was noted at 00:38 where the robot looked upside down in the last two test runs. On 4 short runs with every watcher healthy, flip bug, LR blow-up, crash , Night Watch caught all 3 failures; the thresholds and Gemma caught 2 each, TabPFN 1. The rules caught the crash, the thresholds the LR blow-up, and only Gemma the robot on its back. Per test episode, Gemma matched the simulator on all 23 6 alarms, 17 fine . Then a fair test for TabPFN: 12 runs of 400,000 steps 4 healthy, 2 flip, 1 spin, 2 LR blow-ups, 1 physics, 1 stall, 1 crash , replayed leave-one-run-out through the thresholds, TabPFN, and a gradient-boosting baseline on the same 22 features docs/EVAL.md https://github.com/su0as/night-watch/blob/main/docs/EVAL.md , docs/EVAL gbm.md https://github.com/su0as/night-watch/blob/main/docs/EVAL gbm.md . Neither model was tuned: TabPFN-3.5 and scikit-learn's HistGradientBoostingClassifier both ran with default settings. | watcher | failures caught | false wakes in total | on the 4 healthy runs | AUROC 2,144 log windows | |---|---|---|---|---| | thresholds | 6 / 8 | 5 | 2 | n/a | | gradient boosting | 6 / 8 | 4 | 1 | 0.888 | | TabPFN | 6 / 8 | 1 | 0 | 0.93 | TabPFN and gradient boosting caught the same 6 runs; the difference is how often they woke me for nothing. TabPFN's median time to wake was 7.8 minutes, against 6.3 for the thresholds. The training step alone "how far into training are we" scores AUROC 0.793, so part of any score is knowing that failures come later in a run; without the training-progress feature, TabPFN still scores 0.932 and gradient boosting 0.89. The thresholds and TabPFN caught different runs, and the split by kind is the interesting part: Metrics are good at noticing broken training and slow at noticing a robot doing the wrong thing. That gap is what the head camera is for. nightwatch fit on your own runs . The pictures stay home. These are frames from unpublished research, hundreds a night, and none of them leave my laptop. It runs where the compute is. Lab and cluster machines often have no outbound internet; Ollama, TabPFN's local weights and the Temporal dev server all run offline. It costs nothing per night, so I never ration how often it looks. I could open the box when it failed. I found Gemma's blindness to the side view only because I could test it cheaply and repeatedly against simulator ground truth, and the fix came from owning the whole stack, renderer included. Reading TabPFN's source is how a check went from 96 seconds to under one. Where a closed model would be better: judging the side view; a frontier vision model would very likely see the flip my 4B model couldn't. It would also mean a bill per look and research frames leaving my machine every night. It's one week old, so the evidence is two watched runs, an eyesight test and 16 recorded runs, not a month of nights. My next long run starts before bed, with the watch on. fit with cache . Built solo.