# My Validation Layer Was Correctly Deleting 16% of My Good Data > Source: > Published: 2026-08-25 03:12:31+00:00 *Originally published at ai.bedvibe.studio.* I built a real-time tracker in Rust — about two thousand lines — that reads a live ADS-B feed, keeps a Kalman-filtered track per aircraft, and screens every pair for closest approach against separation minima. Roughly 150 aircraft, a full cycle in under a millisecond. It ran clean. Tests passed, the picture looked right, the numbers were plausible. It was refusing about **one measurement in nine**, and the only reason I ever found out is that the rejections went to a counter instead of a log line. The tracker runs an innovation gate: when a position arrives, the filter predicts where the aircraft should be, and if the measurement is too far from that prediction it is rejected as physically impossible rather than believed. Once a track converges the innovation standard deviation settles around 36 m, so a five-sigma gate sits at roughly 180 m. An airliner at 250 m/s covers 180 m in **0.7 seconds**. So the gate's entire tolerance for a wrong timestamp is under one second. Any pipeline that mis-times its measurements by more than that will have them rejected — correctly, and invisibly. Every ADS-B record carries a field saying how old that position already was when the response was generated. In the original build it was parsed into the contact struct and never read again — the only other place that field appeared in the entire codebase was as `0.0` in test fixtures. Every measurement was therefore stamped with the tracker's own cycle clock, as though it had been observed at the instant it landed. This is the common case, not an exotic one. **A field that is decoded and then unused looks identical to a field that is decoded and used**, right up until you go looking for its second reference. Here is what that field actually contains, sampled across two consecutive polls of the live feed: ``` reported age of position median 0.31 s p90 3.97 s max 48.53 s change per aircraft between consecutive polls -15.76 s to +3.00 s re-served identical positions 17 of 135 contacts (12.6%) ``` The intuitive diagnosis is that the lag itself is the problem. It is not, and the distinction turns out to be the whole thing. **A constant lag is invisible to a constant-velocity filter.** If every measurement is uniformly two seconds old, the filter simply tracks a target that is uniformly two seconds behind. The innovations stay small. Nothing is rejected. The picture is late, but it is self-consistent. That has a sharp consequence for anyone writing a regression test: a fixture built with a constant age will not reproduce the fault no matter how large the age. The test passes and the bug survives. The fixture has to carry the *variation*, or it is testing nothing. Look at that middle row again. Between two polls two seconds apart, one aircraft's reported age fell by nearly sixteen seconds — a genuinely new observation arriving after a long gap. Stamped on arrival, both the stale one and the fresh one are marked "now", so the filter sees an aircraft that has apparently teleported. A steadily flying aeroplane appears to lurch, and a correctly functioning gate refuses to believe it. This is the part worth reading, and the reason the change is architectural rather than a one-line patch. The obvious repair: stamp each measurement with `arrival − age` , feed that to the filter, and guard against a measurement arriving from before the filter's current state, because you cannot predict backwards. ``` if observed_time <= track.epoch { reject } ``` That guard is fatal. Every cycle, the display loop advanced *every* track to the current wall-clock time so the picture and the collision screen would agree. So by the time a two-second-old measurement arrived, the track's clock already read later than the measurement, and the guard would have discarded it as stale. **A patch aimed at an 11% rejection rate would have rejected considerably more — and from the outside it would have looked exactly like the fix working.** The general form: **a Kalman filter's state is valid for exactly one instant.** Once anything other than a measurement is allowed to advance that instant, every measurement is applied to a state from a different moment than the one it describes. A rendering loop is not usually thought of as mutating the estimator, which is precisely why this survives review. The pre-fix code was not wrong about this. It was *consistently* wrong — the display and the measurements were stamped with the same fictional clock, so they agreed with each other. That is why correcting one half in isolation breaks it. One sentence: **the filter's validity time advances only when a measurement arrives. Nothing else may move it.** The picture and the conjunction screen do not predict tracks forward — they take a view, which extrapolates a copy and leaves the filter parked at the moment it was last given evidence for. A test asserts that rendering the screen thirty times does not move a single filter. Two properties follow, and both are load-bearing. **The absolute observation time is never reconstructed at all.** The step between two observations is computed entirely from differences — elapsed time between two arrivals the process witnessed, plus the two ages the sensor reported. Nothing subtracts a duration from a monotonic clock, so there is no underflow path. That closes a real crash: `Instant - Duration` panics in Rust if the result would precede the clock's origin, and on Windows that origin is boot time. **Deduplication becomes free.** A re-served snapshot has both its arrival and its reported age advance by the same amount, so it computes a step of exactly zero and is rejected by the same comparison that catches out-of-order data. One comparison covers both cases, which is a good sign the invariant is the right one. Measuring this against a live third-party feed is harder than it looks, and two designs had to be discarded before one held. Running both binaries simultaneously fails: the corrected build recorded 32 polls and 32 HTTP errors — zero data — while the old one ran fine. That is not a regression, it is the upstream rate-limiting per IP. Running them sequentially fails differently: the results invert, because the API throttles progressively and whichever build runs first gets the fresher quota. Time-to-first-byte on three consecutive manual requests climbed 0.43 s, 0.91 s, 4.22 s. Both would have produced a clean-looking number and a wrong conclusion. What caught them was running the corrected build alone, which worked — proving the failure was the environment, not the code under test. The design that survives: alternating order, 45-second windows, 75-second cooldowns, comparing gated plots as a *fraction* of observations so traffic volume cancels. ``` window build observations gated rate 1 corrected 2,137 3 0.14% 2 pre-fix 2,752 308 11.19% 3 corrected 2,208 81 3.67% 4 pre-fix 2,755 578 20.98% pooled pre-fix 886 / 5,507 16.1% corrected 84 / 4,345 1.9% ``` Both corrected windows sit below both pre-fix windows with no overlap. The corrected build varies — 0.14% against 3.67% — and I would not claim a precise point estimate from four windows. **The separation is the defensible result, not the number.** **It was silently failing most of its polls.** 888 observations at roughly 135 per response is about seven successful polls in sixty seconds, not thirty. Two-thirds of its requests were failing and the system reported nothing, because there were no sensor-health counters at all — a failed poll produces no data, which looks exactly like quiet airspace. **It never aged out a single track.** Zero drops in both windows. Of course not: if every measurement is stamped with arrival time, every track's last-seen time is always now, so nothing is ever stale. A track whose aircraft stopped reporting forty seconds ago sat on the display looking as current as everything else. Those counters were added to report on something else entirely — a limit on response body size. They immediately found a different fault. That is the lesson I would take from this project ahead of the timestamp one: **observability added for one problem finds problems you were not looking for, and a system with no failure counters cannot tell you the difference between "nothing is happening" and "nothing is working".** If you ingest anything from a source that reports its own staleness — market data, IoT sensors, GPS, log shipping, any polled API with a timestamp in the payload — check whether your pipeline records when the record *arrived* instead of when it was *observed*. The failure mode is nasty specifically because it is quiet. Nothing crashes. No error is logged. Your validation layer does its job perfectly and deletes your good data, and if the rejections are counted anywhere at all you will read the number as evidence that the validation is working. And a constant lag hides it completely. It is the jitter that bites — which means the systems most likely to have this bug are the ones whose feeds are *usually* fast. MIT, 55 tests, clippy clean at `-D warnings` : [github.com/Mormolykos/aether](https://github.com/Mormolykos/aether). Surveillance and state estimation only — no targeting, engagement or weapon functionality of any kind. The README carries the full measured results and a limitations section considerably longer than this post.