{"slug": "plan-26-36-ground-control-to-major-modis", "title": ".plan-26-36: Ground control to major MODIS", "summary": "Tessera v2.0 beta2, published by Anil Madhavapeddy and Mark Elvers to source.coop, reduces boundary artefacts by 35% using MODIS corrections but regresses temporal alignment, so beta1 may remain the best v2 embedding without a retrain. James G. C. Ball found that training on 2018 and testing on 2019 dropped macro F1 from 0.586 to 0.511 for beta2, while beta1 improved from 0.576 to 0.626, prompting the team to seek GPU resources for a v2.5 model trained on corrected data.", "body_md": "Got back to Cambridge mid-week and back on the keyboard to start preparing for Michaelmas term, which creeps up fast! [Tessera v2 beta2](https://anil.recoil.org/#tessera-v2-beta2-and-the-temporal-axis) turns out to smooth space at the cost of time, so beta1 may be as far as the v2 series goes without a retrain. Evidence TAP gets [a splendid explainer from Sam](https://anil.recoil.org/#more-polite-crawling-for-the-evidence-tap) and some new Associate Scholars at Pembroke, along with [a TDM reservations library](https://anil.recoil.org/#text-and-data-mining-reservation) and some thoughts on [working across OCaml and OxCaml](https://anil.recoil.org/#working-on-both-ocaml-and-oxcaml). Elsewhere there's [Scrutineer's git workflow](https://anil.recoil.org/#scrutineers-git-workflow-for-security-scanning) and an invitation to a bug sprint, rather a lot of [the Windows tax](https://anil.recoil.org/#the-windows-tax), why [AI disclosure is a social problem](https://anil.recoil.org/#ai-disclosure-is-more-of-a-social-issue-than-technical) rather than a technical one, [a fourth Recoil host](https://anil.recoil.org/#a-fourth-recoil-host) running OpenBSD, and the usual [fun links](https://anil.recoil.org/#fun-links).\n\n[Mark Elvers](https://www.tunbury.org/) and I published more [Tessera](https://anil.recoil.org/projects/tessera) v2.0 beta2 to [source.coop](https://source.coop/tessera/tessera/zarr) this week to test out some fixes to the improved embeddings.\n\nThe v1 embeddings sometimes show a \"checkerboard\" where Sentinel-1 and Sentinel-2 coverage is thin. This happens in areas where there are few satellite observations for the model to infer embeddings from. v2-beta2 tried to smooth them over by applying [MODIS](https://modis.gsfc.nasa.gov/about/) corrections during inference to augment the data, since MODIS is coarser spatially but has more consistent coverage. This actually works pretty well, and [Mark Elvers](https://www.tunbury.org/) measured [a 35% reduction in the boundary artefact](https://www.tunbury.org/2026/08/31/week-35-2025/#the-tessera-temporal-axis).\n\nUnfortunately, this does regress another aspect of the embedding temporally.. The v2 model wasn't trained on MODIS, and so inferring with it shifts the embeddings year-on-year in ways the model does not anticipate. Because MODIS observations themselves vary year to year, this breaks the cross-year alignment [v1.1 added](https://anil.recoil.org/notes/tessera-v11-out). [James G. C. Ball](https://patball1.github.io) caught this by testing them on the v2 [Trentino](https://anil.recoil.org/papers/2026-tessera-trentino) embedding as while intra-year cross-validation is neutral, training on 2018 and testing on 2019 dropped the macro F1 (from 0.586 -> 0.511) wherase the v2-beta1 embeddings instead *improve* (from 0.576 to 0.626).\n\nSo it looks like, somewhat reluctantly, that the v2-beta1 embeddings may be the best we can do in the v2 series without retraining a model with more MODIS observations or (maybe) doing some extensive fine tuning. Opinions from other testers are still incoming, so do get in touch if you have a view. We're still looking for GPU resource to train the v2.5 model on corrected data rather, so get in touch if you can help with those too!\n\nMeanwhile, the wall-to-wall inference v1.1 run for 2017 to 2025 is underway after last month's GPU shortage and should finish in a few weeks. I released [GeoTessera 0.10.2](https://github.com/ucam-eo/geotessera/releases/tag/v0.10.2) as a point release to fix v1.1 Zarr URL resolution that broke in last week's [Source Cooperative migration](https://anil.recoil.org/notes/2026w35), and am working on a [robustness pass](https://github.com/ucam-eo/geotessera/pull/403) for recoverable Zarr writes and [improved Windows support](https://anil.recoil.org/#the-windows-tax) for Zarr.\n\n[Sam Reynolds](https://samreynolds.org) has written [a splendid piece for the Pembroke blog](https://www.pem.cam.ac.uk/college/news/august-2026-dr-sam-reynolds) on what the Evidence Traceable AI Pipeline is for (as part of his introductory post for College!).\n\nFor the last three years, we have been working with colleagues in Computer\nScience on an AI pipeline that finds, classifies, extracts data and\nsummarises findings from the academic literature using self-hosted\nopen-access Large Language Models, benchmarking performance against our hard\nwon, human created database. Underpinning this are pioneering agreements with\nthe largest scientific publishers, brokered with the help from the University\nLibrary, to download millions of academic papers.  We are also developing\ntools that let decision makers interact with this extracted information. This\nis the foundation for the Evidence Traceable Accountable Pipeline (E-TAP)\nproject here at Pembroke, generously funded by the MacArthur Foundation,\nthrough which we hope to generalise this approach for other fields.\n-- [Sam Reynolds, Aug 2026](https://www.pem.cam.ac.uk/college/news/august-2026-dr-sam-reynolds)\n\nI'm delighted that [Sam Reynolds](https://samreynolds.org) and Mélanie Gréaux have now joined Pembroke as\nAssociate Scholars alongside [Sadiq Jaffer](https://toao.com). Having the [conservation evidence](https://anil.recoil.org/projects/ce), education experts and the computer scientists all in College\nwill make for some fun socials, especially as many other colleagues from other\ndepartments have expressed interest in the project as well!\n\nOn the coding side, I released a [TDMRep 1.0](https://github.com/ocaml/opam-repository/pull/30616) library to opam. Our download engine is crawling papers at some scale now, and publishers have adopted a standard to express their responses to AI crawlers via the [W3C TDM Reservation Protocol](https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240202/). My [tdmrep library](https://tangled.org/anil.recoil.org/ocaml-tdmrep) gives us the ability to read this metadata from OCaml code, and the Taposaur crawler declares its intentions (non-commercial, research use only) when requesting papers as well.\n\nI also pulled out [json-pointer 1.0](https://github.com/ocaml/opam-repository/pull/30656) into my [tangled](https://anil.recoil.org/notes/tangled-and-ci) repos. This is a query syntax for JSON that's seeing some adoption (e.g. `/users/0/name`), and I'm using this to parse Semantic Scholar fulltexts in the Evidence TAP corpus.\n\nSince we're using these libraries in my [OxCaml httpz stack](https://anil.recoil.org/#working-on-both-ocaml-and-oxcaml) now, I've also started extracting that code out of our internal monorepo. E-TAP is a \"live\" service now, with the stack fetching papers and categorising them, so I'm getting more handson experience with OxCaml in production.\n\nOne challenge is that a lot of our high-performance infra (both for ETAP and Tessera) is built in OxCaml, which is a fast moving target as Jane Street release compilers quickly and with breaking changes. There's also some very ugly (but performant) edges to the language as it evolves, such as the use of ppx to get around the lack of layout polymorphism. While this makes code hard to edit sometimes, I've found that coding in OCaml and then agentically translating to OxCaml works very well, since the type system catches layout issues very reliably.\n\nI got my [oxcaml/opam-repository#59](https://github.com/oxcaml/opam-repository/pull/59) merged, which makes it much easier to mix OCaml and OxCaml packages through the [guard packages](https://anil.recoil.org/notes/oxcaml-opam-guards).\nHowever, this approach does neccessitate having full control over dependencies, since writing a\nparser in OxCaml is very different from normal OCaml and many dependencies need annotations.  My\n[httpz](https://anil.recoil.org/notes/oxcaml-httpz) library has matured in recent months from a parser into\na full HTTP stack with a zero-allocation `fetch` for HTTP clients, and\n`proffer`, a portable HTTP server layer. I've been experimenting with various\nstrategies to packaging this up which I'll share in the coming weeks.\n\nI've also started [httnope](https://tangled.org/anil.recoil.org/httnope), to\nbuild a conformance corpus that supplies adversarial peers to httpz clients or\nservers and checks the observable effects on fresh connections. This is more\nuseful than testing against a well-behaved HTTP peer since all the\n\"interesting\" failures all found in behaviours no reasonable HTTP server would\nsend (but bad attackers probably will). I'm bootstrapping this using LLMs\nparsing RFCs, but [Thomas Gazagnaire](https://github.com/samoht) has also pointed me to some of his work on an FSM\nlibrary that he's having good success with, so I'll work more on this...\n\nThe [Scrutineer security scanner deployment](https://anil.recoil.org/notes/scrutineer-local-llm) from last week now has a private Git remediation workflow I added.  A gated patch attempt is exported as a cherry-pickable commit on a private Git remote, so I can review with:\n\n``` bash\n$ git fetch scrutineer '+refs/findings/*:refs/remotes/scrutineer/findings/*'\n$ git cherry-pick scrutineer/findings/F-63\n```\n\n...over an ssh tunnel. Triage decisions go back as Git notes under `refs/notes/scrutineer/triage/<principal>`. This is a local-only prototype for now, but I'll continue to gain experience over the next few weeks.\n\nThere are quite a few bugs the scanners are finding across a variety of repos, but I released the ones found in [cstruct](https://github.com/mirage/ocaml-cstruct) via [#324](https://github.com/mirage/ocaml-cstruct/pull/324) which corrects indexing and subview offset handling. [Thomas Gazagnaire](https://github.com/samoht) reviewed it and I cut [6.3.0](https://github.com/mirage/ocaml-cstruct/releases/tag/v6.3.0). That only leaves about 150 verified bugs to triage elsewhere...\n\nI also got a couple of fixes merged upstream into Scrutineer ([#949](https://github.com/alpha-omega-security/scrutineer/pull/949) and [#950](https://github.com/alpha-omega-security/scrutineer/pull/950)). Separately, we've been invited to a week-long bug sprint as part of [Patch the Planet](https://trailofbits.com/patch-the-planet/), the Trail of Bits initiative that pairs security engineers with maintainers. Their projects already include Python, PyPI, cURL and Go, so OCaml would be in decent company! I'll email the OCaml security team and a few others next week to arrange something for October.\n\nUnfortunately there comes a time in every open source library's life when it has to pay a Windows tax, and productivity comes to a grinding halt. Since [GeoTessera](https://github.com/ucam-eo/geotessera) users are typically ecologists rather than computery people, I have to figure out new bugs whenever I add a new feature (in this case, Zarr).\n\nEvery time I SSH'd into my Windows box this week I got random errors, and it turned out to be [RedirectionGuard](https://www.microsoft.com/en-us/msrc/blog/2025/06/redirectionguard-mitigating-unsafe-junction-traversal-in-windows/), a newish mitigation that the builtin OpenSSH server now activates by default. It stops privileged processes following symlinks created by unprivileged users. Unfortunately for some reason my opam install has Flexlink installed via a symlink, which maddeningly lead to obscure toolchain errors *only* when I was remotely connected and didn't manifest when I was at my keyboard. Argh!\n\nThe work in [geotessera#403](https://github.com/ucam-eo/geotessera/pull/403) now makes the Zarr stores handle Windows paths consistently. On Linux or macOS I can reproduce a path bug in seconds, but for Windows I have to go through a painful CI loop and also understand a [very complex path mechanism](https://github.com/ocaml-multicore/eio/blob/main/lib_eio/utils/nt_path.ml). I kind of assumed that Python would have good support for abstracting all this, but it feels very similar to OCaml in its difficulty writing portable code...\n\nI also dove into Eio while I was in front of my Windows box, and opened [#929](https://github.com/ocaml-multicore/eio/pull/929) to fix anonymous-pipes and then also worked on improving [Forester support in #931](https://github.com/ocaml-multicore/eio/issues/931). Fixing that was a bit of a [rabbithole#932](https://github.com/ocaml-multicore/eio/pull/932) but has improved the state of Eio on Windows quite a bit. Just need process and pty support next I think.\n\nI removed the AI disclosure attributes and opam fields across my monorepo this week, and deleted the `ai-disclosure` skill from [my OCaml Claude marketplace](https://github.com/avsm/ocaml-claude-marketplace). That reverses what I [proposed in April](https://anil.recoil.org/notes/opam-ai-disclosure) and [May](https://anil.recoil.org/notes/opam-ai-disclosure-update).\n\nDisclosure of how people code is a technical answer to a social question. When I wrote the proposal you could still imagine agent-written code as a distinguishable subset of a codebase. But today agents are absolutely everywhere, and the number of models have exploded. Tracking this in detail doesn't seem like a winning strategy.\n\nThe [opam-repository policy](https://github.com/ocaml/opam-repository/tree/master/governance/policies#14-package-contributors-accounts-should-have-a-human-behind-them) that [Romain Calascibetta](https://github.com/dinosaure) started is a much better social solution. I wrote my thoughts on [OCaml Discuss](https://discuss.ocaml.org/t/opam-repository-package-contributors-should-have-a-human-behind-them/18466); we ask only that a human is around to answer our reviewers when they have a question. The opam-repository is where we need to aspire to build community within OCaml, which is only getting harder as automated code generation removes the necessity of collaboration we used to have.\n\nTo put this in context, some of the PRs we get to opam-repository are a bit surreal these days. [#30634](https://github.com/ocaml/opam-repository/pull/30634) proposed a `realdentalcosts` library with \"stdlib-only OCaml URL helpers\", from an organisation that only appeared on GitHub on a few days ago and was spraying packages across several ecosystems at once. [Sacha Ayoun](https://bsky.app/profile/giltho.bsky.social/post/3muoop5x6as2m) dubbed this LLM-oriented SEO.\nMeanwhile, [#30643](https://github.com/ocaml/opam-repository/pull/30643) went the other way with a very pleasant interaction with a `vscoqbot` bot account for [rocq-prover/vsrocq](https://github.com/rocq-prover/vsrocq). [Enrico Tassi](https://github.com/gares) replied within the hour with \"sure, tell me what needs to be fixed\" and we got the minor issues sorted quickly.\n\nI've been provisioning a fourth Recoil host at [Mythic Beasts](https://mythic-beasts.com) and putting OpenBSD back on it, so [our routable IPv4 allocation](https://anil.recoil.org/notes/recoil-self-hosting-2026) can finally host throwaway services I've been using [exe.dev](https://anil.recoil.org/notes/2026w30) for. OpenBSD's `vmm` suits this well since it brings some sorely needed [software diversity](https://anil.recoil.org/notes/rewilding-the-web-report), and its single-vCPU hypervisor is just fine for this.\n\nMythic Beasts remain a joy to host with since there's always a real human at the other end who replies (thanks Pete!) quickly. They're plugging a USB stick into the machine so I can netinstall OpenBSD myself, and quickly sorted out routing the custom `/28` from our block as well.", "url": "https://wpnews.pro/news/plan-26-36-ground-control-to-major-modis", "canonical_source": "https://anil.recoil.org/notes/2026w36", "published_at": "2026-09-06 00:00:00+00:00", "updated_at": "2026-09-07 21:32:33.360716+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "computer-vision"], "entities": ["Tessera", "Anil Madhavapeddy", "Mark Elvers", "source.coop", "MODIS", "James G. C. Ball", "GeoTessera", "Sam Reynolds"], "alternates": {"html": "https://wpnews.pro/news/plan-26-36-ground-control-to-major-modis", "markdown": "https://wpnews.pro/news/plan-26-36-ground-control-to-major-modis.md", "text": "https://wpnews.pro/news/plan-26-36-ground-control-to-major-modis.txt", "jsonld": "https://wpnews.pro/news/plan-26-36-ground-control-to-major-modis.jsonld"}}