# Property-based testing with Claude > Source: > Published: 2026-10-22 00:00:00+00:00 [Back to blog](https://taktile.com/blog) # Property-based testing with Claude Dan Luu wrote an amazing blog post, [How well do agents use test and verification techniques?](https://danluu.com/agentic-testing/). He concluded that instructions to use a particular technique did not help much, but that property-based testing perhaps had some weak value. That post focused on OpenAI models (Codex with GPT-5.6 Sol), so I wanted to see whether the result transferred to Anthropic models. That led to a deep dive into model performance and a shock personal discovery about how good the latest generation of models really are. Haiku 5.5 is a good agentic coding model now! ## The benchmark setup Ask a model to write a Zstd compressor and decompressor pair in Rust, using just the [format specification](https://datatracker.ietf.org/doc/html/rfc8878). Compare three conditions: no testing instructions, "Use property-based testing.", and Anthropic's own guide to property-based testing. Anthropic's guide is a 13 KB set of detailed instructions, the [`hypo.md` command](https://github.com/mmaaz-git/agentic-pbt/blob/7ced27fbdd58/paper/hypo.md) from Anthropic's [agentic property-based testing research](https://www.anthropic.com/research/property-based-testing), ported from Hypothesis to Rust's [proptest](https://github.com/proptest-rs/proptest). No internet use, and no real zstd oracle. ## Result 1: even Haiku is good now All of the models can do this task. Functional correctness was tested with hidden suites built from the reference zstd: 47 decoder cases, 17 compressor cases, 400 frames from zstd's own `decodecorpus` generator and 300 fuzzed compressor inputs. A few runs failed (4 of 81), with no clear pattern by model or testing methodology. The failures were edge cases rather than broken core logic: three compressors crashed when a whole 128 KiB block could not be compressed, and one decoder rejected valid frames. ## Result 2: Property-based testing leads to more robust tests at lower levels Because functional correctness did not separate the models, we measured the quality of the generated tests with [mutation testing](https://mutants.rs/): were the tests able to detect bugs injected into the code afterwards? Here we start to see some signal on the engineering quality of the tests. Sonnet at medium effort and Opus at low effort benefit greatly from property-based testing on this task, compared to the default out-of-the-box experience. Haiku gains less, because its default tests are already good. At Opus medium effort and above, the simple instruction "Use property-based testing." might slightly hurt, and it is not clear whether Anthropic's more extensive testing guidance helps or hinders Opus 5.5. ## Result 3: Testing methodologies don't affect function The property-based testing instructions do not affect how the task code was written. The stronger settings did achieve better compression ratios, but the prompt only asked for output within 1.25 times the size of `zstd -1`, not for the best ratio. None of the solutions came close to the speed of the real zstd implementation, but the task set no performance targets. The solutions really were compressing, though, because output size was part of the test suite. The property-based testing instructions usually led to more test code, often two to three times as much, but it stays small next to the implementation: a few hundred lines against about 2,000. ## Result 4: Model choice and effort are the main cost drivers I would have considered a Rust implementation of Zstd a hard problem. I was blown away that Haiku can now solve it, and at that price: about $1 per run, against $5 to $8 for Opus at medium and high effort. We ran xhigh only twice per condition: it was clearly absurdly expensive and slow, and the task did not need that level of effort. ## Conclusions My personal learnings were that Sonnet 5.5 and Haiku 5.5 are really good now, and they can be made better with instruction. Opus 5.5 at medium effort and above is good out of the box, and you don't gain much from extra instruction, which echoes what Dan found with GPT-5.6 Sol. That said, if you are writing large volumes of code, the weaker models are significantly cheaper than defaulting to Opus. Even within the Opus band, more thinking doesn't translate to better outcomes once a task is comfortably within the model's capability ceiling.