cd /news/developer-tools/an-oracle-for-coding-agents-testing-… · home › topics › developer-tools › article
[ARTICLE · art-155197] src=minikin.me ↗ pub= topic=developer-tools verified=true sentiment=↑ positive

An Oracle for Coding Agents: Testing a Flutter Popover Against UIKit

The popover 0.5 Flutter package now tests its popover rendering against outlines and frames recorded from Apple's UIKit and AppKit, using a Swift probe app run in an iOS 26 simulator as a differential-testing oracle so that neither the developer nor a coding agent judges the shape by eye. The recorded fixtures in test/fixtures/ hold 31 paths covering every side and arrows in the middle, at the corners and squashed against edges, and every test using them runs without a simulator; the fixtures describe continuous corners with a 34 point radius and an arrow 26 points wide by 13 high whose sides are filleted into the body. The package previously drew a sharp triangle on a white card, and Flutter's own request for a popover with an arrow, flutter/flutter#53103, has been open since March 2020.

by read10 min views35 publishedOct 14, 2026
An Oracle for Coding Agents: Testing a Flutter Popover Against UIKit
Image: Minikin (auto-discovered)

popover 0.5 tests its Flutter code against outlines and frames recorded from UIKit and AppKit, so neither I nor a coding agent judges the shape by eye.

For popover 0.5 I wanted a Flutter popover you cannot tell from the native one on iOS, iPadOS and macOS, and I wanted coding agents to write a large part of it. A popover that looks close is not hard to build: a rounded rectangle, a softened triangle, a blur. Whether it is the same as the native one is not something I can judge by looking, and a coding agent cannot either. An agent asked to make the arrow look more like iOS can change the curve, write a test for the new curve and report that the test passes. We needed a reference that neither of us could edit, and the only one that exists is Apple’s own code.

The left popover is UIKit, the right one is the package. The Flutter code draws the outline UIKit draws, and a test fails when the two differ.

Until 0.5 the package drew a sharp triangle on a white card. The iOS arrow has a rounded tip, sides that flow into the body, and near a corner it merges with the corner into one curve. I tried to draw that shape by hand more than once and never got one I trusted. Flutter has no popover with an arrow of its own: the request for one, flutter/flutter#53103, has been open since March 2020.

A closed reference implementation# #

In backend and distributed systems work this is differential testing: run your implementation next to a reference known to be right, whether a specification, a model or an older system, and treat every difference in output as a bug in yours.

For a popover the reference is UIKit on iOS and AppKit on macOS. Neither is open source, and Apple’s Human Interface Guidelines describe what a popover is for, not its radius, margins or timing. Screenshots were the first thing I tried. They were good enough to see that the arrow is about 26 points wide and 13 high, and not good enough for anything else. At the 3x scale of an iPhone screen a corner is a few rows of half-transparent pixels, and I could not tell Apple’s continuous corner from a circular one, let alone read its radius. A screenshot also shows one popover in one place, and says nothing about what happens next to a screen edge. But UIKit and AppKit can be run, which makes them a test oracle: a black box you feed inputs and whose answers you keep.

The query is a probe app, one Swift file built with swiftc and launched in an iOS 26 simulator. The anchor position comes from environment variables, so the probe can be launched once per position across the screen:

On an iPhone, UIKit turns a popover into a sheet unless the delegate returns .none from adaptivePresentationStyle, so that method is the first thing the probe needs. Each question after that got its own variant: one prints the frame UIKit chose, one prints the outline, one dumps the animations. The probes live outside the repository and are not published. What they produced is in the repository: the fixtures in test/fixtures/ are plain text, and every test that uses them runs without a simulator.

The arrow as recorded output# #

UIKit masks the popover background with a path, and that path is the whole shape. The probe walks it with CGPath.applyWithBlock and prints one line per element. Here is the start of the case with the anchor at the left edge of the screen:

Read as geometry, it has continuous corners with a 34 point radius, an arrow 26 by 13 whose sides are filleted into the body, a curve for a tip, and near a screen edge one curve where the arrow merges into the corner.

The fixture holds 31 of these paths, covering every side and arrows in the middle, at the corners and squashed against an edge. test/apple_popover_path_test.dart builds the Flutter path for each case and compares it with UIKit’s element for element. The largest difference across all 31 is 0.0005 points.

On macOS, NSPopover has 20 point corners and an arc for an arrow tip, and test/mac_popover_path_test.dart checks it against 171 paths dumped from AppKit. AppKit starts its path at a different point than the Flutter code does, so that test samples both outlines and compares the shapes instead of the elements.

Placement works the same way. The frames from the probe are expected values in test/apple_placement_test.dart, and they encode rules Apple does not document: a 10 point margin on iPhone and 13 on macOS, 19 at the sides, 30 at the top and 10 at the bottom on iPad, a flip to the opposite side when the body does not fit, and on macOS a second try on the perpendicular sides before the body shrinks.

Once the fixtures existed, I could hand the rest of the work to agents. The repository’s AGENTS.md, which coding agents load at the start of every session, says:

Placement, outlines, glass, shadows and motion on Apple platforms are measured from UIKit and AppKit. Do not change a measured constant or an expected frame in test/apple_placement_test.dart without measuring the native popover again.

An agent can refactor the path builder, split a function or change how the shape is painted, but not the fixture.

.mcp.json registers the Dart MCP server, so an agent runs the analyzer and flutter test in its own session. When a change moves an iOS outline by more than 0.002 points, test/apple_popover_path_test.dart fails there, before the change reaches my review.

Where the reference cannot be queried# #

Three parts of the popover could not be dumped as numbers: the shadow, the glass and the motion on macOS. They had to be measured from the outside and fitted, which turns an exact comparison into a model with an error bound. The iOS shadow, a single BoxShadow with a 210 point blur, matches the simulator within one level of brightness. The macOS shadow is fitted to a screenshot of NSPopover.

Flutter has no Liquid Glass, Apple’s translucent material in iOS 26 and macOS 26, only BackdropFilter, which can blur the content behind a widget and pass it through a color filter. I captured native popovers over 22 backgrounds in light and dark mode and fitted a blur sigma, a 4 by 5 color matrix and a rim color to them:

Each row keeps a fraction of the color behind and adds a large constant, so the glass passes some of the background through and fills the rest with near white. On flat backgrounds the fit lands within a few levels of brightness of the simulator. It does not bend light near the outline the way Liquid Glass does, and it does not adapt to the wider area behind the popover. The comparison at the top of this post shows the difference: the Flutter glass keeps a little more of the green.

The first build of the glass showed a flat gray tint for the whole fade-in and turned into glass only when the fade ended. In my testing, Impeller, Flutter’s renderer, gives a BackdropFilter inside an Opacity layer an empty backdrop, so with a FadeTransition around the popover the blur had nothing to blur. The glass now fades inside the render object that paints it, with the blur scaled by the opacity. That rule is in AGENTS.md too, since nothing in the API would stop someone from wrapping the popover in an Opacity again.

The iOS springs needed no fitting. UIKit attaches CASpringAnimation objects to the layers of _UIPopoverView, a private class the probe reads and the package never touches. Each animation reports its mass, stiffness, damping and duration. Flutter’s SpringDescription takes the first three, and the duration drives the curve that wraps it.

NSPopover took three attempts. A dump of the probe’s layers found no animations. Polling the popover window’s alpha from the app saw no fade. Capturing the window in a loop with CGWindowListCreateImage missed it too. NSPopover animates in the window server, outside the app process, where none of those can see it.

What can see it is ScreenCaptureKit. I work on that Mac, so the recording was limited to the probe’s own window. At 120 frames per second it showed the popover scaling up and fading in from the arrow tip over 440 ms, and on close AppKit removing the window 200 ms in, at about 5% of its size. The macOS springs are fitted to those frames.

Where the Flutter version disagrees on purpose# #

UIKit moves an anchor that is partly off screen back onto the screen edge before it places the popover, at the bottom and the sides. An anchor above the top edge pushes the popover off screen instead. That looks like a bug, so the Flutter version clamps on every edge, and the changelog names the difference.

Dark mode is the other one. A native popover follows the system setting, and that was the plan for the Flutter one. But a Flutter app can force a light theme, and on a phone in dark mode it would get dark glass under its dark text. So the popover picks the light or the dark background from the text color the body inherits. In 0.5.0 the switch sat at a text luminance of 0.34. The same evening I derived it from the WCAG contrast formula instead. The switch is the gray where text has the same contrast against the light and the dark background, worked out for the triangle and for the glass over its least contrasting backdrops: a luminance of 0.23. That became 0.5.1. #999999 text on the triangle went from 2.9:1 on the white card to 4.9:1 on the dark one.

Guardrails for the rest of the code# #

Coding agents did a large part of the work on 0.5, under my review, and the fixtures cover only what can be measured on an Apple platform. The rest is covered the way I argued for in the cargo-crap post, with checks an agent cannot talk its way past:

  • Golden tests rendered on Linux. CI renders them on its runner, and tool/update_goldens.sh renders them in a Docker container locally so the pixels match.
  • Mutation testing over lib/ . CI fails below a 75% score, and on a pull request it mutates only the chunks of about 10 lines around changed lines, so a change whose mutants survive pulls the score under 75% and fails the check.

Neither knows what a popover should look like. They catch the regressions the fixtures do not cover.

What it does not do# #

  • While closing, iOS squashes the glass vertically for the first few frames. The Flutter popover does not.
  • The fixtures come from iOS 26 and macOS 26, and the probes that made them are not published. When Apple changes the popover, the fixtures are wrong until I measure again.
  • Android, Windows and Linux have no native popover to measure, so they keep a plain triangle with the iPhone margin.
  • Flutter web follows the host OS on purpose: Chrome on a Mac gets the macOS shape, Chrome on Linux the triangle.
## Try it[#](#try-it)

If a coding agent writes your app, `dart run skills@ get --package popover` installs the skill the package ships for it. It tells the agent which API to pick and which mistakes to avoid, such as an opaque `Material` around the body, which hides the glass.

0.5 has breaking changes, and the [README](https://github.com/minikin/popover#migrating-from-04) has a migration guide from 0.4.
── more in #developer-tools 4 stories · sorted by recency
── more on @popover 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/an-oracle-for-coding…] indexed:0 read:10min 2026-10-14 · —