We build Zugo, which turns one written sentence into a running game, site or app. For a while, the thing that decided whether a build had succeeded was this:
function htmlComplete(text) {
return /<\/html>/i.test(text);
}
A closing tag. That was the gate. If the model emitted </html>
, the turn was a success, the green tick appeared, and the credits were charged. Whether the page actually opened was not part of the decision, because at the moment we charged, nothing had mounted the document anywhere.
If you are building anything that generates code, this is the trap. Writing the file is easy to observe. Whether the file works is not, so it quietly stops being what "done" means.
Our second-line check counted opening and closing tags. Its own header admitted what it was: a balance check, not "it renders." A document can be perfectly balanced, end in </html>
, and throw on the first line of its first script. That build billed as a complete success.
Worse, the failure was invisible in exactly the way that costs the most trust: the tick went green the instant the stream finished, and the real answer arrived seconds later. Measured on our own paths, that gap is 2.6s for a site, 4.2s for a game and 6.2s for a React app, plus one silent re-mount if the first verdict is bad. So a broken build showed a green tick over a blank white frame for up to about twelve seconds. Long enough to read it as the product lying to you and close the tab.
The obvious fix is to wrap the preview in an error boundary and listen. That does not work here, and the reason is worth knowing if you sandbox generated code.
The preview frame is sandbox="allow-scripts"
with no allow-same-origin
. That gives it an opaque origin, which is the whole point: a generated document must not be able to reach the host app, its storage or its session. But the same wall means the host cannot see inside. A React error boundary around the preview catches nothing from the document. If a user ever sees our error boundary, the editor crashed, not their build.
So the listener has to live inside the frame, which means we inject it.
We prepend a small script to the generated document that reports error
, unhandledrejection
and console.error
back to the host via postMessage
. Two details decide whether it is worth anything.
It has to be in <head>, ahead of everything the build wrote. Our first version appended the hook before
</body>
. A script that threw during load ran before any listener existed, so the loudest failures were exactly the ones nothing reported. The ordering is now pinned by a test that asserts the hook's index is lower than both </head>
and any script the build placed in the body.A console.error is not a failure. React logs ordinary development warnings through
console.error
, and "each child in a list should have a unique key" is not a broken app. So the harness reports it as a separate field:
var ce = console.error;
console.error = function () {
rep({ logErr: [].map.call(arguments, s).join(" ").slice(0, 400) });
return ce.apply(console, arguments);
};
Everything downstream that decides pass or fail checks if (d.logErr) return;
before it counts anything. The warning is still recorded, it just never fails a healthy build. Collapsing these two into one channel is how you get a verifier that cries wolf until people stop reading it.
The last rule is the one I would keep if I had to throw the rest away. Verdicts only travel in one direction:
if (tn.files !== vFiles || (tn.rendered && (!tn.rendered.ok || v.ok))) return tn;
Once a build has been marked bad, no later signal is allowed to mark it good. A second render, a retry, a re-mount, an error that happens not to repeat: none of them can promote a failure back to a success. Combined with a 9 second window after mount during which errors still count, this closes the obvious escape hatch where a flaky build eventually reports clean and the user is told everything is fine.
And the green tick now waits. Between "the stream finished" and "the frame answered" the UI says it is checking whether the build opens. Neutral, not a claim. We would rather show uncertainty for three seconds than certainty that turns out to be wrong.
The bug was not the regex. The regex was fine at what it did. The bug was letting the cheap observable stand in for the expensive one, because the expensive one arrives late and the cheap one is available right now.
Three questions worth asking about your own pipeline:
If you want to watch the version with the harness in it, Zugo is free to try, and the templates are all real builds you can open and edit.