Small test (600 items, irony detection) comparing Laya against OpenAI’s Decisions API. Laya’s calibration held up better when I skewed the class mix, but equivalent rewordings flipped far more of its decisions at a fixed threshold. If you’re using it for routing or gating, version your question wording and regression-test it. Details and the commercial-model comparison . article here:The Decision Contract
One split worth making before treating the 40% as instability: separate the flips that cross your fixed threshold from flips in the underlying ranking. Rewording mostly nudges scores a little, so if the flipped items sit within a narrow band around the threshold, the model is ranking almost the same way and the threshold is doing the flipping — report the flip rate inside versus outside that band. If flips are just as common far from the threshold, the wording really is changing the decision and versioning the wording is the right defense; if they cluster at the boundary, a margin zone that sends near-threshold items to a second look fixes most of it without retraining.