By Igor Eduardo · Austin, TX · with Lexi Armstrong
Site: igoreduardo.com · Repo: github.com/nomad-link-id/sentinel-mlx · Demo: youtu.be/c2v5Mdg5fpw
This is a build note from the webAI YOLO26 MLX Build Challenge (May 2026), not a product pitch. We shipped a single-file, on-device posture attention map that runs with WiFi physically off. The useful part for other builders is what failed first.
Seven days. Two-person build: Lexi Armstrong owned the problem and operational constraints; I owned the engineering.
What shipped uses yolo26n with released weights as-is, plus a deliberately simple bounding-box geometry heuristic for posture — standing / sitting / lying. Single-file Python (~200 LOC), single-thread synchronous loop, ~16 FPS at 720p on M4 / 32 GB / macOS 26.3 / Python 3.14, ~45 ms inference per frame.
An earlier attempt to fine-tune yolo26n on labeled posture data did not converge and was dropped.
We're writing this up because the failure modes are probably more useful than the demo — specifically the macOS 26 AVFoundation crash, the lazy-eval gotcha in yolo-mlx's Boxes, and the training collapse. And because the part that turned a demo into something a responder might actually trust came from the field side, not the code.
SENTINEL is the meeting of two views of the same problem. Lexi saw it from the field — denied-comms / industrial security / operational-edge constraints, including what "zero network egress" has to mean when operational security depends on it. I saw it from healthtech — production-class triage and patient-flow systems in extended pilot with tier-1 hospitals in São Paulo. The core problem of who needs attention first when there are too many patients and not enough hands shows up in mass-casualty triage too, at different scale and stakes.
Concretely, the field side drove:
Honest division of labor: Lexi made sure it was worth building and aimed at the right target; I built it.
The initial architecture was a FastAPI backend exposing a WebSocket stream to a browser overlay, with camera capture in a ThreadPoolExecutor. Two hard blockers killed it.
With OpenCV 4.13's AVFoundation backend, capturing frames from a non-main thread crashed reliably with SIGTRAP. The trace pointed at:
cv2.abi3.so -> CaptureDelegate captureOutput -> CFRelease
This looks like a CFRelease reference-counting issue when AVCaptureSession is owned outside the main thread on macOS 26. We didn't patch around it — rewriting the capture stack wasn't worth the risk in a 7-day window. Flagging it because anyone building on this stack with a worker-thread capture pattern is going to hit it.
Boxes proxies
The harder one to isolate. Benchmark showed clean FPS. Warmup completed. Model loaded. But when the WebSocket handler called detector.predict() on a real camera frame, it hung silently. Every time.
Diagnostic path:
dets = []) — video appeared immediately. Camera, WebSocket, frontend all confirmed working.predictor.py: _predict start, mx.eval done, returning results all appeared.predict() in detector.py: BEFORE PREDICT appeared, AFTER PREDICT never did.
The hang was not in MLX inference itself — it was in the box-iteration loop after the model returned:
for box in results[0].boxes:
x1, y1, x2, y2 = [int(v) for v in box.xyxy[0].tolist()] # hangs here
Root cause hypothesis: yolo-mlx's Boxes returns lazy MLX proxies for .xyxy, .conf, .cls. They aren't evaluated during inference — they're deferred. Calling .tolist() or indexing them triggers a secondary mx.eval() that deadlocked in the multi-thread setup. Warmup used np.zeros (zero detections), so the box loop never ran during warmup. Real frames produced detections, the loop ran for the first time, the lazy eval fired, and the pipeline froze.
Workaround — force-materialize before iterating:
import mlx.core as mx
import numpy as np
boxes = results[0].boxes
mx.eval(boxes.xyxy); mx.eval(boxes.conf); mx.eval(boxes.cls)
xyxy = np.array(boxes.xyxy)
conf = np.array(boxes.conf)
cls = np.array(boxes.cls)
for i in range(len(xyxy)):
x1, y1, x2, y2 = [int(v) for v in xyxy[i]]
No .tolist() on MLX proxies inside loops. Convert to NumPy once, then iterate. Warmup with synthetic zero-detection frames does not exercise the box-iteration path, so this bug is invisible until a real subject enters the frame — worth a doc note or an explicit .materialize() helper on Boxes.
Around day four we made a call. The architecture being hardened solved for production complexity not needed for a 7-day demo. We forked the official Yolo26-mlx challenge starter, deleted everything around the inference call, and rewrote it as a single synchronous loop on the main thread:
while True:
ok, frame = cap.read()
detections = model(frame)
overlay = render_sentinel_ui(frame, detections)
cv2.imshow("SENTINEL", overlay)
if cv2.waitKey(1) & 0xFF == ord("q"):
break
That sidesteps the AVFoundation issue (capture on main thread) and the lazy-eval issue (no async / threading). ~16 FPS at 720p on M4 once settled. That's what shipped to the public repo.
In parallel, we tried fine-tuning yolo26n on labeled posture data — a Roboflow posture-classification dataset (person_lying / person_sitting / person_standing). Training loss collapsed to near-zero by epoch 2; mAP stuck at 0.0 for the remaining epochs. We didn't isolate the root cause in the time available — most likely a label-format mismatch or normalization gap — but those debugging cycles weren't available with AVFoundation and lazy-eval also live.
Noting this because "trained posture head" appears on the roadmap, and we want to be explicit that it's an honest open problem — not something we skipped by choice.
The shipped version does not use a trained posture classifier, and SENTINEL does not make a clinical triage decision. It uses released yolo26n.npz weights as-is. The layer on top is a deterministic posture heuristic on bounding-box geometry. It reports what the camera can actually see — body posture — not medical severity.
class == 0 (person), conf ≥ 0.40
aspect = bbox_height / bbox_width
aspect > 1.6 → standing1.0 ≤ aspect ≤ 1.6 → sitting / slumpedaspect < 1.0 → lying down
Why posture, not severity: a camera cannot see a pulse, internal bleeding, or a blocked airway. The honest output is "who is upright, who is down, and for how long" — a visual cue that helps a responder decide where to look first. The human triages.
Where the heuristic fails (honest):
"We can write the rule on a napkin" was the right call for a 7-day safety-relevant demo — but it's a starting point, not the answer.
Worked: yolo-mlx 0.3.1 API; MLX Metal backend on Apple Silicon; starter repo structure (pivot took an afternoon); macOS 26 + Python 3.14 + M4 once threading was off the table.
Friction:
??.
Form factor: laptop demo ≠ product. Realistic V1 path we sketched — Vision Pro for prototype, field-grade waveguide for first-responder pilot — both keep inference on-device. Platforms that move compute to a paired puck reintroduce a network leg that weakens the zero-egress story for this use case.
We didn't build SENTINEL to chase a market — we built it because the environments where attention allocation matters most are often the ones where the network isn't there. Before this is a product: trained posture classifier, a real labeled dataset, and validation with a real responder who either uses the map or ignores it. No projections — we haven't earned them yet.
If the AVFoundation crash, the lazy-eval workaround, or the training-collapse note is useful to YOLO26-MLX docs, happy to write more on any section.
— Igor Eduardo (igoreduardo.com) & Lexi Armstrong
Repo: nomad-link-id/sentinel-mlx · Demo: youtu.be/c2v5Mdg5fpw
Disclosure: drafted with AI assistance from build notes; technical claims and wording owned by the authors.