{"slug": "linux-embedded-log-errors-detector-on-imx8-s-npu", "title": "Linux embedded log/errors detector on Imx8's NPU", "summary": "A developer released sentinel-imxd, a lightweight log-anomaly detection daemon for the NXP i.MX 8M Plus that runs a TensorFlow Lite autoencoder on the board's Vivante NPU via the VX external delegate. The daemon listens to the systemd journal and D-Bus system-bus signals, sanitizes each line into a template, encodes it into a fixed INT8 [1, 64] vector, and triggers an external alert script when reconstruction loss (MSE) exceeds a calibrated threshold of roughly 3× the p90 normal loss. Training runs on a host in Docker using the same encoder as the daemon, with the quantized model_quant.tflite and threshold deployed back to the board over SSH.", "body_md": "A lightweight log-anomaly detection daemon for the **NXP i.MX 8M Plus** (Yocto /\nDebian), plus a self-contained Docker build + model-training environment.\n\nThe daemon (`sentinel-imxd`) listens to the **systemd journal** (kernel messages\nplus userspace service logs) and D-Bus system-bus signals, sanitizes each line\ninto a stable *template*, encodes it into\na fixed **INT8 `[1, 64]`** vector, optionally runs a TensorFlow Lite autoencoder\non the **Vivante NPU** via the VX external delegate, and — when the reconstruction\nloss (MSE) exceeds a threshold — runs an external alert script.\n\nThe whole story on one page — train on your computer (in Docker), deploy over SSH,\nand detect on the board's NPU. Regenerate it any time with\n[`scripts/make-diagram.sh`](https://github.com/leonardosalvatore/sentinel-imx-sys/blob/main/scripts/make-diagram.sh)\n([PDF](https://github.com/leonardosalvatore/sentinel-imx-sys/blob/main/docs/sentinel-imx-solution.pdf)).\n\nWant the *why*? The technical brief\n[**Why standard log parsing fails on embedded boards**](https://github.com/leonardosalvatore/sentinel-imx-sys/blob/main/docs/why-on-device-anomaly-detection.pdf)\nmakes the case for on-device anomaly detection over rule-based / cloud log\nparsing. Regenerate it with `python3 docs/make_whitepaper.py`.\n\n[https://www.youtube.com/watch?v=d_EssCpTC4E](https://www.youtube.com/watch?v=d_EssCpTC4E)\n\nEverything runs **on the device**. Journal (kernel + userspace) and D-Bus events\nflow through a single `sd-event` loop, get turned into a normalized INT8 feature\nvector, and are scored by a TensorFlow Lite autoencoder that is offloaded to the\nVivante NPU. A high reconstruction error triggers a user-supplied action script.\n\nKernel messages arrive via journald (which is the canonical `/dev/kmsg`\nconsumer) rather than a second raw reader; entries with `_TRANSPORT=kernel` are\nstill tagged `source=kmsg`, while other journal entries are tagged\n`source=journal`. One tradeoff: journald rate-limits (` RateLimitBurst`), so a\nsevere printk storm can be dropped in the journal where a raw kmsg reader would\nstill see it until the ring buffer wraps.\n\n```\nflowchart LR\n subgraph board[\"i.MX 8M Plus · Yocto / Debian\"]\n direction LR\n subgraph src[\"Event sources\"]\n direction TB\n journal[\"systemd-journald
kernel + userspace logs\"]\n dbus[\"D-Bus system bus
systemd + logind\"]\n end\n\n subgraph daemon[\"sentinel-imxd · systemd service\"]\n direction TB\n loop[\"sd-event loop\"]\n san[\"Sanitize → template
mask PID / addr / UUID / num\"]\n enc[\"Encode → INT8 [1,64]
signed hash + L2 normalize\"]\n inf[\"TFLite autoencoder
reconstruct\"]\n mse[\"Dequantize → MSE
compare to threshold\"]\n loop --> san --> enc --> inf --> mse\n end\n\n npu[\"Vivante NPU
libvx_delegate.so\"]\n cap[(\"capture.jsonl
training data\")]\n alert[\"on-alert.sh
LED · restart · notify\"]\n\n journal --> loop\n dbus --> loop\n enc -. append .-> cap\n inf <-->|offload| npu\n mse -->|\"loss > threshold\"| alert\n end\n```\n\nThe daemon captures normal traffic; you train a model from it on the host (in the\nDocker environment) using the **same encoder** the daemon runs, then deploy the\nmodel and its calibrated threshold back to the board. Detection is a per-event\nreconstruction-error check against that threshold.\n\n```\nflowchart TB\n subgraph collect[\"1 · Collect on board\"]\n direction TB\n runcap[\"Run daemon in capture / auto mode\"]\n jsonl[(\"capture.jsonl
sanitized templates\")]\n runcap --> jsonl\n end\n\n subgraph train[\"2 · Train on host · Docker\"]\n direction TB\n reenc[\"Re-encode templates
sentinel_features.py ≡ encoder.cpp\"]\n prep[\"Dedupe + drop demo / load-test rows\"]\n ae[\"Train autoencoder
normal traffic only\"]\n quant[\"Full-INT8 quantize
→ model_quant.tflite\"]\n calib[\"Calibrate threshold
≈ 3× p90 normal loss\"]\n reenc --> prep --> ae --> quant --> calib\n end\n\n subgraph detect[\"3 · Detect on board · per event\"]\n direction TB\n line[\"New log line\"]\n enc2[\"Sanitize + encode INT8 [1,64]\"]\n recon[\"NPU autoencoder reconstruct\"]\n loss[\"MSE input vs output\"]\n decide{\"loss > threshold?\"}\n fire[\"Alert → on-alert.sh\"]\n ignore[\"Ignore · keep capturing\"]\n line --> enc2 --> recon --> loss --> decide\n decide -->|yes| fire\n decide -->|no| ignore\n end\n\n jsonl -->|copy off board| reenc\n quant -->|deploy model| recon\n calib -->|set threshold| decide\n```\n\nEmbedded Linux boxes are noisy: the kernel and system services emit a constant stream of log lines. Most are routine; a few (a failing eMMC, an OOM kill, a USB device misbehaving, an unexpected service crash) are early warnings. Watching those by hand doesn't scale, and shipping every log to the cloud is often impossible or undesirable on an edge device.\n\n`sentinel-imx-sys` learns what *normal* logs look like for **your** device, then\nflags the abnormal ones **on the device**, in real time, at near-zero cost — and\nruns a script of your choosing when something looks wrong. No cloud, no log\nshipping, data stays local.\n\nIt's an **autoencoder for logs**: it's trained only on normal traffic, so when it\nsees something unfamiliar it reconstructs it poorly (high error = anomaly).\n\nEach event is reduced to a template (PIDs/addresses/UUIDs/numbers masked), hashed\ninto a 64-bin signed histogram, and **L2-normalized** before it hits the model.\nThat normalization matters: without it the reconstruction error just tracks how\n*many* tokens a line has, so the model ranks events by length and a short kernel\n`BUG:` line looks more \"normal\" than routine chatter. Projecting every event onto\nthe unit sphere makes the score reflect the token *pattern* — novelty, not size.\n\n- **Capture-first.** With no model present (`mode=auto` ), the daemon just records\nsanitized/encoded events to a JSONL file — use it to collect training data on\nthe real board, then train and drop in`model_quant.tflite` .\n- **Actions are yours.** Alerts don't do anything hard-coded; they exec a\nconfigurable bash script with rich context in the environment.\n- **Minimal dependencies.** Only`libsystemd` (sd-event + sd-bus + sd-journal)\nand TensorFlow Lite. No Boost, no JSON library (JSON is written/parsed by hand).\n- **NPU acceleration.** Uses`/usr/lib/libvx_delegate.so` as a TFLite*external\ndelegate* ;`USE_GPU_INFERENCE=0` steers it to the NPU rather than the 3D GPU.\n\n```\nCMakeLists.txt Cross/native build\ndocker-compose.yml One-command dev environment\ndocker/ SDK download + cross-build + training image\nconfig/sentinel-imx.conf Runtime configuration\nsystemd/ Service unit\nscripts/on-alert.sh Alert hook stub (customize this)\nscripts/sentinel-tui.py Neon '80s dashboard for demos (run on the board)\nscripts/demo.sh One-screen tmux live demo (run on the board)\nscripts/npu-bench.sh CPU-vs-NPU latency benchmark (run on the board)\nscripts/npu-load.py NPU load generator (drives the load meter for demos)\ntools/ Host-side training + capture export (+ diag_loss.py)\ntools/sentinel_features.py Python mirror of the C++ sanitize+encode pipeline\nsrc/ Daemon sources\nLICENSE / NOTICE Apache-2.0\n```\n\nNXP's public EVK downloads are flashable `.wic` images, not an application SDK.\nThe container instead installs the public **standard Scarthgap (Yocto 5.0 LTS)\naarch64 SDK** — the same Yocto series as NXP LF 6.6.52 — giving the cross\ntoolchain in the familiar board-SDK layout. (The *standard* toolchain is used\nrather than the extensible `-ext-` one, which refuses to install as root.)\nTensorFlow Lite is fetched from the NXP git fork at configure time;\n`libvx_delegate.so` lives on the target at runtime.\n\n```\n# 1. Build the image (downloads the SDK + libsystemd, installs TF for training)\ndocker compose build\n\n# 2. Cross-compile the daemon -> ./out/sentinel-imxd (aarch64)\ndocker compose run --rm dev ./docker/scripts/build.sh\n\n# 3. Train a model from captured data -> ./out/model_quant.tflite\ndocker compose run --rm dev ./docker/scripts/train.sh data/capture.jsonl\n\n# ...or, with no capture file, train on synthetic data (placeholder model):\ndocker compose run --rm dev ./docker/scripts/train.sh\n```\n\nBuild outputs land in `./out/` owned by root (the container runs as root). Run\n`sudo chown -R \"$USER:$USER\" out build-aarch64` if you need to edit them from\nthe host.\n\nIf you have a real NXP SDK installer, drop it in `docker/sdk/*.sh`; it takes\nprecedence over the public poky toolchain (and already contains `libtensorflow-lite`\nand the i.MX tuning).\n\nTo skip the heavy TensorFlow Lite build (capture-only daemon):\n\n```\ndocker compose run --rm dev ./docker/scripts/build.sh -DSENTINEL_ENABLE_TFLITE=OFF\n```\n\nOn an i.MX image that already ships `libsystemd` and `libtensorflow-lite`:\n\n```\ncmake -S . -B build -DSENTINEL_ENABLE_TFLITE=ON\ncmake --build build -j\nsudo cmake --install build\n# Copy the cross-built binary + assets to the board, then:\nsudo install -m0755 out/sentinel-imxd /usr/bin/sentinel-imxd\nsudo install -Dm0644 config/sentinel-imx.conf /etc/sentinel-imx/sentinel-imx.conf\nsudo install -Dm0644 systemd/sentinel-imx.service /usr/lib/systemd/system/sentinel-imx.service\nsudo install -Dm0755 scripts/on-alert.sh /usr/libexec/sentinel-imx/on-alert.sh\nsudo install -Dm0644 out/model_quant.tflite /usr/share/sentinel-imx/model_quant.tflite # optional\nsudo systemctl daemon-reload\nsudo systemctl enable --now sentinel-imx.service\n```\n\nSee [`config/sentinel-imx.conf`](https://github.com/leonardosalvatore/sentinel-imx-sys/blob/main/config/sentinel-imx.conf). Every key can be\noverridden by a `SENTINEL_` environment variable. Key options:\n\n| Key | Default | Meaning | \n|---|---|---|\n| `mode` | `auto` | `auto` /`capture` /`infer` | \n| `model_path` | `/usr/share/sentinel-imx/model_quant.tflite` | TFLite autoencoder | \n| `delegate_path` | `/usr/lib/libvx_delegate.so` | Vivante VX external delegate | \n| `threshold` | `45` | MSE anomaly threshold (model-specific; the trainer suggests one in `out/threshold.txt` ) | \n| `alert_script` | `/usr/libexec/sentinel-imx/on-alert.sh` | Program run on anomaly | \n| `alert_cooldown_sec` | `10` | Min seconds between alerts | \n| `capture_path` | `/var/lib/sentinel-imx/capture.jsonl` | JSONL training data | \n| `journal` /`dbus` | `true` /`true` | Enable event sources ( `journal` = systemd journal, kernel + userspace) | \n| `dbus_match` | (built-in defaults) | Repeatable D-Bus match rule | \n\n`on-alert.sh` receives: `SENTINEL_LOSS`, `SENTINEL_THRESHOLD`, `SENTINEL_SOURCE`\n(`kmsg`/` journal`/` dbus`), `SENTINEL_TEMPLATE`, `SENTINEL_RAW` (truncated).\n\n`scripts/sentinel-tui.py` is a dependency-free ('80s synthwave) full-screen\ndashboard: animated banner, per-core CPU bars + sparklines, Vivante GPU/NPU\nload, memory/temp/uptime, the daemon's live vitals, and a colorized live event\nstream. Keys on camera:\n\n- `a` /`space` — inject a synthetic anomaly (watch the alert fire in the stream)\n- `n` — toggle**NPU stress** : loops the large showcase model on the NPU so the\naccelerator meter (`core c1` ) climbs to ~40–80% while the GPU (`core c0` ) stays\nflat — a live, honest proof that the work lands on the NPU. Copy the showcase\nmodel + load generator to the board first (see below).\n- `q` — quit\n\n```\n# copy the dashboard, load generator, and showcase model once:\nscp scripts/sentinel-tui.py scripts/npu-load.py root@:/usr/local/bin/\nscp out/model_showcase.tflite root@:/usr/share/sentinel-imx/\n# then run over an SSH TTY so keys work:\nssh -t root@ 'python3 /usr/local/bin/sentinel-tui.py'\n```\n\n`scripts/demo.sh` splits your terminal into a single screen that tells the whole\nstory — run it on the board (needs `tmux`):\n\n```\n./demo.sh\n```\n\n- **top-left** — live detections (`journalctl -u sentinel-imx -f` )\n- **top-right** — NPU proof: the daemon is a registered client of the Vivante\nNPU driver and its model tensors are resident in NPU video memory\n- **bottom-left** —`top` filtered to the daemon (CPU stays tiny)\n- **bottom-right** — a pre-typed fault injector; press Enter to fire one:\n\n```\necho \"kernel BUG: unable to handle kernel paging request at 00000000\" > /dev/kmsg\n```\n\nThe narrative: inject a scary kernel message → an alert fires instantly → CPU barely moves → because the model is running through the NPU delegate.\n\nThe i.MX NPU load gauge (`/sys/kernel/debug/gc/load`) stays near 0% for the\n*production* workload — a tiny INT8 autoencoder finishes each inference almost\ninstantly (even ~6000 inferences/sec only registers ~3%), so there are no\nsustained \"busy cycles\" to show. That's the efficiency story, not a bug. Two\nways to prove the NPU is really doing the work:\n\n**Live (visual).** Press `n` in the dashboard, or run the load generator\ndirectly. It loops the large showcase model on the NPU and the meter climbs to\n~40–80% on `core c1` (the NPU) while `core c0` (the 3D GPU) stays at 0%:\n\n```\nUSE_GPU_INFERENCE=0 python3 scripts/npu-load.py \\\n /usr/share/sentinel-imx/model_showcase.tflite\nwatch -n1 cat /sys/kernel/debug/gc/load # core 1 (NPU) rises; core 0 stays 0\n```\n\n**Static (binding + residency).** Regardless of load:\n\n```\n# 1. The daemon is bound to the Vivante NPU/GPU driver:\ncat /sys/kernel/debug/gc/clients # -> lists sentinel-imxd\n\n# 2. Its model tensors are resident in NPU (VIP) video memory:\ngrep -A5 sentinel-imxd /sys/kernel/debug/gc/database\n\n# 3. The journal shows the graph was delegated:\njournalctl -u sentinel-imx | grep -i delegate\n# -> using external delegate /usr/lib/libvx_delegate.so (NPU)\n```\n\nMeasured with NXP's `benchmark_model` (INT8), and via the running daemon:\n\n| Metric | Value | \n|---|---|\n| Daemon idle CPU | ~0.3% | \n| End-to-end CPU per event (parse → encode → infer → capture) | ~0.25 ms | \n| Resident memory (RSS) | ~37 MB | \n| Production model (4.1K MACs) — **CPU** (1× A53) | **~2.3 µs / inference** | \n| Production model (4.1K MACs) — NPU (VX delegate) | ~110 µs / inference | \n| Showcase model (8.8M MACs) — CPU (4 threads) | ~1130 µs / inference | \n| Showcase model (8.8M MACs) — NPU (VX delegate) | ~1180 µs / inference (+3.35 s first-run compile) | \n\n**Honest takeaway:** for this workload the **CPU is the right choice**. The i.MX\nNPU (VIP8000) is built for large convolutional/vision tensors; small INT8\nMLP/autoencoder inferences are dominated by per-invoke dispatch and weight-DMA\noverhead, so the NPU is far slower for the tiny production model and only\nbreak-even for a deliberately heavy one. The daemon supports the NPU delegate\n(and it works — the graph is fully delegated) so you can drop in a heavier model\nlater, but out of the box the anomaly signal simply doesn't need it, which is\n*why the daemon is so cheap to run*.\n\nReproduce the comparison on your board:\n\n```\n# tiny production model (CPU wins big)\n./npu-bench.sh /usr/share/sentinel-imx/model_quant.tflite\n\n# heavy showcase model (train it first: train.sh --arch large)\n./npu-bench.sh /tmp/model_showcase.tflite\n```\n\n1. Run in `capture` (or`auto` without a model) to collect`capture.jsonl` .\n2. Copy it off the board into `./data/capture.jsonl` .\n3. `docker compose run --rm dev ./docker/scripts/train.sh data/capture.jsonl` .\n4. Copy `out/model_quant.tflite` to the board, set`threshold` from`out/threshold.txt` in the config, and restart the service.\n\nThe model is a small autoencoder (`64 -> 32 -> 64`, INT8 in/out). Training\n**re-encodes the captured templates** with the exact same normalized encoder the\ndaemon uses (`tools/sentinel_features.py` mirrors `src/encoder.cpp`), so training\nmatches inference byte-for-byte. Two details make the scores meaningful:\n\n- **Deduplication** (`--dedupe` , on by default in`train.sh` ): each distinct\nnormal template is learned once, so a handful of very frequent patterns (e.g.\nsession churn) can't dominate and leave rare-but-normal messages under-learned.\n- **Excludes synthetic/demo traffic** : templates containing`demo-inject` ,`npuload` , or`loadtest` are dropped, so the model never learns the very\nanomalies you inject (or the benchmark load generators) as \"normal\".\n\nAfter export the trainer runs the quantized model over the normal set, prints the\nloss distribution, and writes a suggested `threshold` (~3x the p90 steady-state\nfloor) to `out/threshold.txt`. `tools/diag_loss.py` prints normal-vs-anomaly\nlosses if you want to eyeball the separation before deploying.\n\nTraining runs under the Keras 2 API (`tf-keras`, selected via\n`TF_USE_LEGACY_KERAS=1`): TensorFlow 2.16 defaults to Keras 3, whose graph the\nfull-int8 TFLite quantizer cannot lower. The dev image installs `tf-keras` and\nthe training script sets the flag automatically — no action needed.\n\nPass `--arch large` for a deliberately heavy (~8.8M MAC) autoencoder with the\nsame INT8 `[1,64]` interface — useful only for the NPU-vs-CPU benchmark, not for\nproduction:\n\n```\ndocker compose run --rm dev ./docker/scripts/train.sh # small (default)\ndocker compose run --rm --entrypoint bash dev -lc \\\n 'python3 tools/train_autoencoder.py --synthetic --arch large --output out/model_showcase.tflite'\n```\n\nOne JSON object per line:\n\n```\n{\"ts\":1725270000,\"source\":\"kmsg\",\"template\":\"usb - new high-speed usb device number using \",\"vector\":[3,-1,0, ...]}\n```\n\n`tools/export_capture.py` converts this to a NumPy `(N, 64)` int8 array.\n\nThe daemon appends forever. Add a `logrotate` snippet if needed:\n\n```\n/var/lib/sentinel-imx/capture.jsonl {\n weekly\n rotate 4\n missingok\n notifempty\n copytruncate\n}\n```\n\nApache-2.0 — see [`LICENSE`](https://github.com/leonardosalvatore/sentinel-imx-sys/blob/main/LICENSE) and [`NOTICE`](https://github.com/leonardosalvatore/sentinel-imx-sys/blob/main/NOTICE). Contributions\nwelcome; see [`CONTRIBUTING.md`](https://github.com/leonardosalvatore/sentinel-imx-sys/blob/main/CONTRIBUTING.md).\n\nTensorFlow Lite (NXP fork) is fetched at build time and the Vivante VX delegate lives on the target device; neither is redistributed here.", "url": "https://wpnews.pro/news/linux-embedded-log-errors-detector-on-imx8-s-npu", "canonical_source": "https://github.com/leonardosalvatore/sentinel-imx-sys", "published_at": "2026-09-15 19:33:01+00:00", "updated_at": "2026-09-15 19:49:46.963834+00:00", "lang": "en", "topics": ["ai-tools", "mlops", "ai-infrastructure", "developer-tools"], "entities": ["sentinel-imxd", "NXP i.MX 8M Plus", "Vivante NPU", "TensorFlow Lite", "systemd journal", "D-Bus", "Yocto", "Debian"], "alternates": {"html": "https://wpnews.pro/news/linux-embedded-log-errors-detector-on-imx8-s-npu", "markdown": "https://wpnews.pro/news/linux-embedded-log-errors-detector-on-imx8-s-npu.md", "text": "https://wpnews.pro/news/linux-embedded-log-errors-detector-on-imx8-s-npu.txt", "jsonld": "https://wpnews.pro/news/linux-embedded-log-errors-detector-on-imx8-s-npu.jsonld"}}