AeroLoop — The Air-Quality Agent That Keeps Its Own Ghosts

A developer built AeroLoop, an open-source self-improving agent that predicts PM2.5 air pollution across London and versions every estimation strategy, preserving failed experiments as 'rejected' ghosts rather than overwriting them. In a simulated 120-pair London day, an interpretable four-parameter linear correction over CAMS model output, tuned by gemma-3-27b-it, cut holdout MAE from 3.94 to 2.99, a roughly 24.1% improvement, while a TabPFN challenger reached 3.08. The results come from a deterministic synthetic dataset, not real pollution readings, and the demo runs locally via Ollama.

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine https://dev.to/challenges/hacktoberfest-week4-2026-10-26 What if an AI agent remembered every mistake it ever made? Not just its successful predictions. Not just its latest model weights. Every failed experiment, every rejected idea, every strategy that looked promising but couldn't survive a proper test. That's AeroLoop. AeroLoop is a self-improving AI agent that predicts PM2.5 air pollution across London. It saves every prediction before ground truth exists. When the actual reading arrives, the agent scores itself, examines its prediction errors, and attempts to improve its estimation strategy. But there's a twist: it doesn't just learn from its mistakes. It preserves them. Every strategy is versioned. Successful experiments become the new incumbent; unsuccessful ones are saved as vN-rejected and displayed on the dashboard as literal ghosts 👻. Nothing is silently overwritten. The inspiration came from the idea that an intelligent machine shouldn't just remember what worked. It should remember what failed, why it failed, and what it learned from the experience. Air pollution is already a ghost story: PM2.5 is the invisible residue of traffic, heating, and weather that lingers in a city long after its source is gone. AeroLoop adds another ghost to the story: the history of an AI learning from its own mistakes. The strategy is an intentionally small, interpretable linear correction over the CAMS model output: pred = pm25 model + humidity coef humidity - 50 + wind coef wind - 5 + precip coef precipitation mm + bias The LLM provides the imagination; the holdout provides the judgment. Any proposed parameters are clamped to sensible bounds. If the model is unavailable or returns invalid output, a deterministic heuristic can propose a candidate instead. The report records exactly which model and parameter source were used. No silent fallbacks. I ran a simulated London day with 4 stations × 30 hourly ticks, producing 120 matched prediction/reading pairs. Here are the results: | Version | Proposed by | What changed | Holdout MAE | Verdict | |---|---|---|---|---| | v3 | gemma-3-27b-it | Bias 2.0 → 3.0 | 3.94 → 3.37 | Kept | | v4-tabpfn | TabPFN Prior Labs | Foundation model trained on tune split | 3.08 | Recorded as challenger | | v5 | gemma-3-27b-it | Bias 3.0 → 4.0, humidity → -0.02 | 3.37 → 3.06 | Kept | | v6 | gemma-3-27b-it | Bias → 4.2, wind → -0.2, precipitation → -0.55 | 3.06 → 2.99 | Kept | | v7-rejected | gemma-3-27b-it | Bias → 4.5 | 3.01 | Rejected 👻 | The final result: Holdout MAE: 3.94 → 2.99 — approximately 24.1% improvement. One particularly interesting subplot: TabPFN initially beat the hand-tuned formula outright. Gemma's incremental tuning subsequently edged ahead. A foundation model and a four-parameter formula competed on the same unseen data, with exactly the same evaluation gate. And when Gemma proposed increasing the bias to 4.5, the holdout said no. That strategy became a ghost instead of silently replacing the previous version. Important: These results come from a deterministic synthetic demo dataset, not real-world measured pollution readings. The holdout is chronological and was kept separate from strategy tuning. To reproduce the demo: The demo can run entirely locally with Ollama. GitHub repository: https://github.com/udirobert/aeroloop https://github.com/udirobert/aeroloop Clone and run: git clone https://github.com/udirobert/aeroloop.git cd aeroloop/backend python -m venv .venv source .venv/bin/activate pip install -r requirements.txt cp ../.env.example .env uvicorn main:app --port 8000 Then open the dashboard and run the simulation. Two important endpoints: POST /demo-day — generates a complete day of prediction and ground-truth memories using deterministic synthetic data. POST /self-improve — runs the complete analyse → propose → holdout-gate → keep/reject loop. AeroLoop combines open data, open-weight AI models, local inference, and a persistent agent memory. The reasoning engine: The model configuration is switchable, allowing different reasoning engines to compete without changing the core agent architecture. The data layer: The self-improvement mechanism: The LLM doesn't directly rewrite the application or modify arbitrary code. It proposes parameters for a small, interpretable estimation formula. Candidates are validated, bounded, evaluated against the incumbent, and either promoted or rejected. This makes the system easier to inspect, reproduce, and trust. AeroLoop would be a very different project if its intelligence, memory, and evaluation process were locked inside a proprietary API. Open innovation made several things possible: 1. Running with the lights and internet off. The reasoning engine can run locally through Ollama using Nemotron Mini. No hosted inference service is required for the local demo. 2. Freedom to change the brain. The architecture supports different OpenAI-compatible endpoints. Gemma, Nemotron, and other compatible models can be tested without rebuilding the agent. 3. Memory that belongs to the project. Predictions, readings, strategy versions, and rejected experiments live in a local SQLite database. The agent's history isn't trapped in a vendor's infrastructure. 4. Independent challengers. TabPFN can compete against the LLM-guided formula under exactly the same evaluation rules. The system doesn't care which model wins; it cares whether the candidate performs better on unseen data. 5. Reproducibility and transparency. The data sources, strategy parameters, model choices, fallbacks, and evaluation results are inspectable. If a model fails, the system records what happened rather than pretending everything worked. Open innovation made it possible to build an agent whose intelligence can be exchanged, whose memory can be inspected, and whose improvements can be independently challenged. The agent's experiment history is preserved in its SQLite database, including the proposals, parameter changes, holdout scores, and rejected versions. I haven't included a DevRelay session recording for this submission. The version history itself demonstrates the agent's decision-making process: every improvement has evidence, and every rejected experiment remains part of the record. The most interesting moment wasn't when the agent improved. It was when the agent failed. Gemma proposed increasing the bias from 4.2 to 4.5. The candidate looked like another reasonable incremental adjustment. But the holdout evaluation showed that it performed worse. So AeroLoop rejected it. The strategy was saved as v7-rejected and appeared on the dashboard as a ghost. It wasn't deleted. It wasn't quietly forgotten. It remained visible as part of the machine's history. That is what I wanted the Ghost in the Machine theme to mean. We often build AI systems that celebrate their latest answer and hide everything that came before it. AeroLoop takes the opposite approach. Its ghosts are reminders that intelligence isn't just about getting things right. It's also about remembering when you got things wrong. The LLM is the imagination. The holdout is the judge. The graveyard is the audit trail. 👻 Add the applicable partner prize categories here once confirmed against the challenge rules. Built for Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine. Repository: https://github.com/udirobert/aeroloop https://github.com/udirobert/aeroloop