{"slug": "ask-hn-how-do-you-use-and-interpret-openai-decisions-api", "title": "Ask HN: How do you use and interpret OpenAI Decisions API", "summary": "A Hacker News poster reported that OpenAI's Decisions API produced biased probability estimates when asked to \"choose an outcome,\" returning a red marble 86% of the time in a jar experiment where the true probability was 50%, while predicate questions matched expected probabilities. In the poster's thousand-trial tests, reordering choices shifted the red marble's estimated probability from 86% to 73%, and the poster asked how developers trust and implement such a \"System One\" style classifier.", "body_md": "I've had some strange results with Jev, and I question the \"probabilistic\" aspects of it. So I put OpenAI Decisions API against classic statistics experiments to see how it holds up:\n\n(1) loaded coin with probability of heads biased towards 70%\n\n(2) marble selection from a jar, with replacement; 5 red, 3 blue, 2 white\n\nI tested both predicate and choice questions. I ran thousand trials against each experiment, and I also did an experiment where I change the order of choices, to see if it matters.\n\nSummary:\n\n* Using predicate questions gave nearly perfect/expected probability outcomes. E.g. for the coin toss, it sampled heads 70% of the time, and for the marble experiment, it sampled the red marble 50% of the time\n\n* Asking it to \"choose an outcome\" behaved differently from drawing randomly - if the true probability of a red marble draw was 50%, using Decisions API produced 86%, i.e. it picked the right marble but gave a significantly more biased weight on its choice\n\n* Changing the choice order changes the probabilities! Moving the red marble from first to last choice changed its probability estimate from 86% to 73%\n\nI have full summary of results here: https://gist.github.com/acatovic/6b31f0061603b3de97731a8a29576dbd\n\nMy question to you is: how do you trust and implement a \"System One\" style classifier like Decisions API, in your work?\n\nComments URL: [https://news.ycombinator.com/item?id=49992182](https://news.ycombinator.com/item?id=49992182)\n\nPoints: 1\n\n# Comments: 0", "url": "https://wpnews.pro/news/ask-hn-how-do-you-use-and-interpret-openai-decisions-api", "canonical_source": "https://news.ycombinator.com/item?id=49992182", "published_at": "2026-10-07 13:01:53+00:00", "updated_at": "2026-10-07 13:20:41.404485+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-products"], "entities": ["OpenAI", "OpenAI Decisions API", "Hacker News"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ask-hn-how-do-you-use-and-interpret-openai-decisions-api", "markdown": "https://wpnews.pro/news/ask-hn-how-do-you-use-and-interpret-openai-decisions-api.md", "text": "https://wpnews.pro/news/ask-hn-how-do-you-use-and-interpret-openai-decisions-api.txt", "jsonld": "https://wpnews.pro/news/ask-hn-how-do-you-use-and-interpret-openai-decisions-api.jsonld"}}