AI pair-programming became useful when every idea met a benchmark

A developer's pair-programming notes conclude that AI coding assistants are useful for generating optimization hypotheses and tracing failures but unreliable as judges of whether their own proposals work, after several confident suggestions — including an 8x8 register block and packed f16 storage — benchmarked slower or broke the real training loop. The workflow that proved productive was to define a baseline, implement the smallest test, measure it, and retain negative results, with end-to-end parity tests catching errors standalone kernel tests missed.

The most confident optimization suggestions were not always the useful ones. An 8 by 8 register block sounded better than 4 by 4 because it offered more reuse. It ran slower. Packed f16 storage sounded additive with tiling. It was not. A vectorized kernel won its standalone benchmark and initially broke the real training loop. The productive workflow was simple: take the suggestion, define the right baseline, implement the smallest test, measure it, and keep the negative result. The benchmark had authority over the explanation. The negative results became more reusable than many wins. They recorded which limit an idea actually attacked and why the next attempt should differ. End-to-end parity tests caught errors that standalone kernel tests could not see. AI was useful for generating hypotheses and tracing failures. It was unreliable as the judge of whether its own proposal worked. That role stayed with frozen inputs, reproducible commands, and measured output. The complete pair-programming notes are at https://posttrainllm.com/devlog https://posttrainllm.com/devlog .