Laya (open decision model) stays calibrated under class shift but flips ~40% of decisions when you reword the question

A 600-item irony-detection test comparing Laya, an open decision model, against OpenAI's Decisions API found Laya stayed better calibrated when the class mix was skewed but flipped roughly 40% of its decisions under equivalent question rewordings at a fixed threshold. The tester advises anyone using Laya for routing or gating to version question wording and regression-test it, and suggests separating threshold-crossing flips from ranking changes: if flips cluster in a narrow band near the threshold, a margin zone sending near-threshold items to a second look fixes most of it without retraining. If flips are as common far from the threshold, the wording itself is changing the decision and versioning the wording is the right defense.

Small test 600 items, irony detection comparing Laya against OpenAI’s Decisions API. Laya’s calibration held up better when I skewed the class mix, but equivalent rewordings flipped far more of its decisions at a fixed threshold. If you’re using it for routing or gating, version your question wording and regression-test it. Details and the commercial-model comparison . article here: The Decision Contract https://www.linkedin.com/pulse/decision-contract-sonal-pardeshi-ro2jc/ One split worth making before treating the 40% as instability: separate the flips that cross your fixed threshold from flips in the underlying ranking. Rewording mostly nudges scores a little, so if the flipped items sit within a narrow band around the threshold, the model is ranking almost the same way and the threshold is doing the flipping — report the flip rate inside versus outside that band. If flips are just as common far from the threshold, the wording really is changing the decision and versioning the wording is the right defense; if they cluster at the boundary, a margin zone that sends near-threshold items to a second look fixes most of it without retraining.