{"slug": "esa-s-satellite-is-dead-its-ghost-2123-signals-tabpfn-beat-the-published-sota-of", "title": "ESA's satellite is dead. Its ghost: 2,123 signals. TabPFN beat the published SOTA; 83¢ of Tinker stopped an LLM speaking for it", "summary": "A developer built The Last Satellite, a Mission Memory Engine that pairs two open models with an evidence layer to investigate telemetry from ESA's OPS-SAT CubeSat, which burned up in May 2024. The system tags each finding as [measured] from raw signal or [cited] from a document, and returns \"Root cause of this signal behaviour is not documented in the archive\" rather than inventing explanations for the 303,493 archived readings. The developer reports that TabPFN beat the published FCNN benchmark (AUCPR 0.979) and that 83 cents of Tinker fine-tuning stopped an LLM from speaking for the dead satellite.", "body_md": "*This is a submission for the [Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine](https://dev.to/challenges)*\n\n```\n2022-06-02 03:17:03 UTC   I_PD3_THETA   0.0540137\n2022-06-02 03:17:08 UTC   I_PD3_THETA   0.0462915\n2022-06-02 03:17:13 UTC   I_PD3_THETA   0.0000000\n2022-06-02 03:17:18 UTC   I_PD3_THETA   0.0000000\n2022-06-02 03:17:23 UTC   I_PD3_THETA   0.0025708\n```\n\n**Ten seconds of nothing.** Recorded by a light sensor on a spacecraft that no longer exists.\n\nESA's OPS-SAT burned up in the atmosphere on **22 May 2024**. Its engineers have moved on. Nobody is left to ask what happened at 03:17:13.\n\nAsk a chatbot and it will tell you anyway. A sensor fault. A reboot. An eclipse. It will sound certain, and **it will be making it up**, because nothing in the archive says why.\n\nSo I built a séance with one house rule: **nobody speaks for the dead.**\n\n**The Last Satellite** hands the archive to two open models:\n\nAsk it why the light went out at 03:17:13, and you get the most honest sentence a medium can give:\n\n**\"Root cause of this signal behaviour is not documented in the archive.\"**\n\nOne confession before we start. My model looked at those ten seconds and called them **0.098: probably fine.** It was wrong. That miss is in the demo, on purpose.\n\nOPS-SAT was ESA's flying laboratory: a CubeSat that teams outside ESA could upload experiments to. Over four and a half years it sent home thousands of telemetry fragments. Then, in the words of its mission-end coverage, it was *\"impacted by the recent unexpectedly strong increase in solar activity that began in early 2024\"*, and it came down early.\n\nSome of what it sent is now **OPSSAT-AD**:\n\nThe labels say **that** something was wrong. Nothing in the archive says **why**.\n\nThat's the trap. Point a modern AI at this archive and ask \"what happened to segment 204?\" and it will give you a confident, plausible, completely made-up story. A sensor fault. A reboot. Cosmic rays. The data can't contradict it, because the people who could are gone.\n\n**The failure you have to fear isn't \"the model is wrong.\" It's \"the model invents a memory the archive doesn't hold, and you believe it.\"**\n\n| Option | What it gives you | What it can't tell you | \n|---|---|---|\n| Read the paper | A benchmark: FCNN at AUCPR 0.979 | Which of *these* segments to look at first, or what the documents say about them | \n| Train your own deep model | Maybe the paper's number, after a training loop | Whether its probabilities mean anything (calibration) | \n| Ask a general chatbot | A fluent explanation | Whether a single word of it is in the archive | \n| **The Last Satellite** | A calibrated anomaly queue, cited documents, a card per segment | It tells you what it **doesn't** know:*\"Root cause not documented in the archive.\"* | \n\n**The Last Satellite** is a *Mission Memory Engine*: an archival investigation console with two open models, one evidence layer and one rule.\n\n`[measured]` from the raw signal or `[cited]` from a document, **Why it fits Ghost in the Machine:** this week runs from Halloween night into Día de los Muertos, when the line between past and present is supposed to be thinnest. OPS-SAT is a ghost in the most literal sense: a machine that no longer exists, still talking through 303,493 archived readings.\n\nDía de los Muertos isn't about being afraid of the dead. It's about **remembering them accurately**: the ofrenda holds their real photos, their real favourite foods, their real stories. So the console is built like one. Marigold and candlelight, not horror. The satellite's own words are the offerings, every one sourced. And there's one rule a séance usually breaks: **nobody gets to put words in the ghost's mouth.** Not me, not the language model. Measured, cited, or **unknown**.\n\nThe video above is the whole tour. The console talks to the live FastAPI backend when it's running. Without it, it falls back to a **labelled archived snapshot of real API responses**, so it works anywhere.\n\n**The signal that should not be there.** Pick one of six candles: six real segments from the official test split, labels hidden. Step through the signal, the 18 measured features, TabPFN's probability, the documents, the most similar past segments, and the investigation card. Then **reveal the truth**.\n\n**And the one it missed.** The demo set deliberately includes the false alarm and a missed anomaly. Notice the banner on the summary: the fine-tuned interpreter's draft was **rejected by the validator**, so the card fell back to plain, deterministic text. More on that below.\n\n**Ask the archive. It tells you what it doesn't know.**\n\n**The queue.** 529 segments ranked by TabPFN, each with a card and three buttons: Confirm, Reject, Investigate.\n\n*Its mission ended. Its memories did not.*\n\nOPS-SAT was ESA's flying laboratory, a CubeSat that re-entered the atmosphere on **22 May 2024**. This project is an **archival** analysis system built on its real, public telemetry and documents. It has two halves:\n\nAn evidence layer joins the two into **investigation cards**. Every statement on a card is either *measured* or *cited*, and an analyst can confirm, reject or investigate it.\n\nNothing here is live. All numbers below are produced by scripts…\n\n**The rule behind all of it: the models read, code measures, and nothing is claimed that the archive cannot show.**\n\nLanguage models are brilliant readers and terrible witnesses. TabPFN is a superb estimator, but a probability is not an explanation. So each job goes to whatever can be held accountable for it, and **five independent checks** stand between the models and the screen:\n\n`unknown`.\n**The data: real, public, and licence-aware.**\n\n**The published state of the art on this dataset is a fully connected neural network, trained, tuned and reported in *Scientific Data* in 2025. TabPFN-v2 never ran a training loop on this data. I handed it the training rows and the paper's own 18 features, and it beat the paper on the official test split. Then I engineered better features, and it beat the paper by more.**\n\nThat's the point. The anomaly detector at the heart of a mission-memory console shouldn't need a GPU, a hyperparameter sweep and a week. It should need a table.\n\n| Official test split (529) | AUCPR | F1 | MCC | \n|---|---|---|---|\n| FCNN, best published (Sci Data 2025) | 0.979 | 0.946 | 0.932 | \n| **TabPFN-v2, paper's 18 features** | **0.989** | **0.950** | **0.937** | \n| **TabPFN-v2, selected features** | **0.996** [0.989–1.000] | **0.982** [0.963–0.996] | **0.977** | \n\nBrackets are 95% bootstrap CIs over test segments.\n\nMy goal wasn't a big number. It was **a big number I could defend line by line.** Here is exactly how.\n\nI downloaded OPSSAT-AD, verified both md5 checksums, and counted everything myself:\n\n*The paper's text says 1,494 training segments; its own Table 1 and the released data both say 1,594.*\n\nThen I ported the published feature code. It reproduces **16 of the 18 features exactly** (99.97% of all values). On the way it caught a nasty bug of mine: the raw file has **22,115 duplicate timestamps**, and a non-stable sort silently changed every derivative feature.\n\nI didn't copy the benchmark from a PDF by eye. The script pulls the paper's full text (JATS XML) from Europe PMC and checks every number in Table 3. The bar to beat is **FCNN: AUCPR 0.979, F1 0.946, MCC 0.932** (best unsupervised: MO-GAAL at 0.779). The 2025 correction notice only re-ordered figures; no table changed.\n\nI ran 5-fold CV on the **training split**, grouped so neighbouring segments of a channel never straddle folds. Each candidate feature group had to earn its place:\n\nEvery threshold comes from **training out-of-fold** predictions, before the test split was opened:\n\nA guard logs every test-set run and **refuses to run the same configuration twice**. One shot:\n\n**415 nominal segments correctly left in the archive, 110 anomalies correctly flagged, 1 false alarm, 3 misses.**\n\nSame split, same features, same folds. Five baselines and a heuristic:\n\n| Official test split | AUCPR | F1 | MCC | \n|---|---|---|---|\n| **TabPFN-v2** | **0.996** | **0.982** | **0.977** | \n| Random Forest | 0.994 | 0.973 | 0.966 | \n| XGBoost | 0.993 | 0.968 | 0.960 | \n| LightGBM | 0.990 | 0.969 | 0.960 | \n| Logistic regression | 0.987 | 0.956 | 0.944 | \n| Isolation Forest (unsupervised) | 0.401 | 0.499 | 0.362 | \n| \"Lots of zeros or a gap\" rule | 0.293 | 0.527 | 0.414 | \n\n**A win, but on my features not a thrashing.** Good features lift every model. Against Random Forest, TabPFN is a **statistical tie** (McNemar p = 0.63; AUPRC difference +0.002, 95% CI −0.0005 to +0.006). Against LightGBM and XGBoost, the paired-bootstrap AUPRC differences sit just above zero, and against logistic regression TabPFN wins with p = 0.03.\n\nThat's why the like-for-like table at the top matters most. **On the paper's own features, TabPFN beats the tuned trees by ~0.02 AUCPR, and the paper's neural network, out of the box.**\n\nThe probability *is* the investigation queue, so it has to be honest. Brier score on test: **0.0073**.\n\n**Qwen3-8B out of the box makes up channel IDs and paraphrases quotes, and with a confident tone that is exactly how false memories get made. It scored 65.3%. Same model, same prompt, same validator: after 83 cents of Tinker, all-in, it scored 93.0%.**\n\nI didn't need a model that knows everything about spacecraft. I needed one that **copies instead of inventing**, and fine-tuning is how a small model learns that.\n\n|  | base Qwen3-8B | **Tinker LoRA** | \n|---|---|---|\n| **Field-level exact match, validation** | 65.3% | **93.0%** [90.7–94.8] | \n| Field-level exact match, held-out test documents | 72.2% | **86.3%** [83.2–88.9] | \n| Channel IDs correct | 70.8% | **100%** | \n| Evidence copied verbatim from the source | 81.1% | **100%** | \n| \"Unknown\" used correctly | 82.1% | **99.8%** | \n| Whole record exactly right | 0% | **44.4%** | \n| Investigation notes passing every check (task D) | 56% | **100%** | \n| Errors caught by the independent validator | 40 | **0** | \n| Unsupported root causes | 0 | **0** | \n\nMy goal wasn't the biggest number. It was **a real number at the lowest price I could defend.** Here is exactly how.\n\nBefore any model saw any data, I froze the headline: **field-level exact match on held-out documents, scored on raw output before any repair.** Companion metrics:\n\nTinker bills per token, so the output is 8 compact lines that code turns into Pydantic JSON:\n\n```\nchannel_id: CADC0873\nsensor_type: magnetometer\nunits: ?\nevent_date: 2024-05-22\nobservation: On 22 May 2024 the magnetometer I_B_FB_MM_1 reported a run of zero values.\noperational_response: ?\nroot_cause: ?\nevidence: On 22 May 2024 the magnetometer I_B_FB_MM_1 reported a run of zero values.\n```\n\nA full training example, prompt and passage included, is **about 380 tokens**.\n\nTraining per million tokens:\n\n```\nrenderer = renderers.get_renderer(\"qwen3_disable_thinking\", tok)\nmi, w = renderer.build_supervised_example(messages, train_on_what=TrainOnWhat.LAST_ASSISTANT_MESSAGE)\n\nfb = tc.forward_backward(batch, loss_fn=\"cross_entropy\")\nop = tc.optim_step(tinker.AdamParams(learning_rate=lr_t, beta1=0.9, beta2=0.95, eps=1e-8))\n```\n\nI didn't train once and hope. I trained in stages, scored every run on the same validation set, and priced each one:\n\n| Run | Est. cost | Validation field exact match | \n|---|---|---|\n| base Qwen3-8B, untrained | n/a | 65.3% | \n| run 1 | $0.025 | 90.8% | \n| run 2 | $0.049 | 91.8% | \n| run 3 | $0.097 | 92.2% | \n| **final model** | **$0.224** | **93.0%** | \n\n**The first 2.5 cents of training took the model from 65.3% to 90.8%.** Every run after that added another point or so, and the final model landed at **93.0%**, all inside an 83-cent budget.\n\nMidway through, most remaining \"errors\" were the model **correctly refusing** to quote code blocks, footnotes and markdown bullet lists that my labelling rules had picked as \"observations\".\n\nPaired, cell by cell, on the same 576 field cells: **95 cells only the fine-tune got right, 14 only the base got right**. Exact McNemar **p = 5.8×10⁻¹⁶**.\n\nThe base model's **40 validator catches** are what the validator is for: channel IDs and quotes that don't exist in the source. **The fine-tune made zero.** Even after the validator repairs the base model's output, it reaches 78.5%, still below the fine-tune's *unrepaired* 86.3%.\n\nI exported the adapter as PEFT (one tensor rename: `unembed_tokens` → `lm_head`), converted it to GGUF with llama.cpp, and ran it over a Q4_K_M Qwen3-8B base:\n\n**92.5% field exact match on held-out passages, 100% field-for-field agreement with the same checkpoint on Tinker, 4.7 s per passage.** M3 Pro, 18 GB, Hugging Face offline, no Tinker key in the environment.\n\n| Item | Est. cost | \n|---|---|\n| Learning-curve runs (runs 1–3) | $0.17 | \n| A run I stopped at 65/84 steps after finding label bugs | $0.17 | \n| **Final model (3 epochs)** | **$0.22** | \n| Downloadable sibling adapter (see Known edges) | $0.10 | \n| All evaluation on Tinker | $0.17 | \n| **Everything on Tinker** | **$0.83** | \n\n*Token counts at Tinker's list prices, logged before every job. Base-model validation ran locally for free.*\n\nI'd rather you hear these from me, with numbers.\n\n**What went wrong on the way (and made it better):** my base-model comparison was unfair at first. The prompt never named the eight fields and the parser rejected markdown, so I fixed both *before* training anything. A root-cause rule briefly treated \"*due to* the underlying nature of the mission\" as a cause, and a Q&A ranking tweak made one answer prettier but quietly removed the \"cause unknown\" warning from others. Both were caught and reverted.\n\nA closed API gives you an answer. Open weights and open data give you **a way to check it.** For a project whose whole job is refusing to invent memories, that difference is everything.\n\n**Where open beat closed, concretely:**\n\n| What I needed | Closed API | What open let me do | \n|---|---|---|\n| Stop the model inventing channel IDs | Write a longer prompt and hope | Fine-tune the weights: invented IDs went to **zero** , for $0.22 | \n| Trust the anomaly probabilities | Take the vendor's word | Measure calibration myself on the public test split (Brier **0.0073** ) | \n| Beat a published benchmark | Not possible to verify | Reproduce the paper's features to the decimal, then beat it **like-for-like** | \n| Keep it running after the hype | Depends on the vendor staying up | Adapter + TabPFN run **offline on a laptop** , no key, no meter | \n\nThe reasons behind that table:\n\nThe honest caveat: training ran on Tinker's servers, on public documents only. At run time, nothing leaves the laptop.\n\nI built The Last Satellite with **Claude Code** as a pair, from a written 12-hour plan with a cost estimate up front, a budget guard on every Tinker job and a guard on every test-set run.\n\n**Best Use of TabPFN.** TabPFN is the engine of the console, not a bolt-on: every probability, every queue position and every \"top measured characteristic\" comes from it.\n\n**Best Use of Tinker.** The LoRA fine-tune (loss on answers only, per-epoch checkpoints), the learning curve, a budget guard, held-out evaluation with paired statistics, and the adapter download behind the offline story all ran on Tinker. **$0.22 for the final model, $0.83 for everything.**\n\n*Built solo, with Claude Code, over one long Halloween night, finished as Día de los Muertos began.*\n\n**Two open models. Eighty-three cents. One rule: measure it, cite it, or call it unknown.** If that sounds like the right way to remember a satellite, a ❤️ or 🦄 helps more people find it.\n\nThis Día de los Muertos, light a candle for a machine you loved that's gone: an old phone, a first laptop, a server that ran for a decade. What would you want its archive to remember? Leave it in the comments. 🕯️", "url": "https://wpnews.pro/news/esa-s-satellite-is-dead-its-ghost-2123-signals-tabpfn-beat-the-published-sota-of", "canonical_source": "https://dev.to/advikdivekar/esas-satellite-is-dead-its-ghost-2123-signals-tabpfn-beat-the-published-sota-83c-of-tinker-3ke1", "published_at": "2026-10-31 22:18:15+00:00", "updated_at": "2026-10-31 22:22:20.580330+00:00", "lang": "en", "topics": ["machine-learning", "ai-tools", "ai-agents", "ai-safety"], "entities": ["ESA", "OPS-SAT", "TabPFN", "Tinker", "OPSSAT-AD", "The Last Satellite", "FastAPI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/esa-s-satellite-is-dead-its-ghost-2123-signals-tabpfn-beat-the-published-sota-of", "markdown": "https://wpnews.pro/news/esa-s-satellite-is-dead-its-ghost-2123-signals-tabpfn-beat-the-published-sota-of.md", "text": "https://wpnews.pro/news/esa-s-satellite-is-dead-its-ghost-2123-signals-tabpfn-beat-the-published-sota-of.txt", "jsonld": "https://wpnews.pro/news/esa-s-satellite-is-dead-its-ghost-2123-signals-tabpfn-beat-the-published-sota-of.jsonld"}}