{"slug": "python-returned-zero-the-coin-model-still-assigned-a-positive-probability", "title": "Python returned zero. The coin model still assigned a positive probability.", "summary": "A developer demonstrated that binary64 floating-point arithmetic silently collapses the likelihoods of two competing coin models to zero at long sequence lengths, erasing the difference between them. Using high-precision Decimal arithmetic, the sweep found that for 1,200 heads in a row, models with p=0.5 and p=0.5001 yield normalized likelihood weights of roughly 44% and 56%, yet both ordinary Python powers return 0.0, and repeated multiplication instead produces 5e-324 and a spurious 0%/100% split. Direct powers first became subnormal at n=1,023 and rounded to zero at n=1,075 and 1,076 for the two models.", "body_md": "AI assistance: An AI agent ran the experiments and helped draft this article.\n\nTry this in Python:\n\n```\n0.5 ** 1200\n# 0.0\n```\n\nThat is a strange result if we read zero as “this cannot happen.” Under an independent fair-coin model, 1,200 heads in a row has a positive probability. A very small one, yes. But positive.\n\nIn the [cookie-jar example](https://dev.to/panditshivam/probabilities-can-sum-to-one-and-still-describe-the-wrong-experiment-1dl), some outcomes really were impossible: we could not draw more plain cookies than the jar contained. Here the formula permits the outcome. The number system runs out of room.\n\nThe interesting part is what happens when we compare two models. The calculation can lose the difference between them, or even turn a rounding artifact into apparent certainty.\n\nImagine two fixed models of independent tosses:\n\nFor a specified sequence of n heads, each model gives a likelihood of p raised to n. No combination factor is needed: the sequence is all heads.\n\nThese are synthetic arithmetic inputs. Nobody observed 1,200 heads, trained a model, or collected evidence of a real biased coin. The models were chosen to make the numerical comparison easy to inspect.\n\nAt n = 1,200, high-precision Decimal arithmetic gives approximately:\n\n| Quantity | Model A: p = 0.5 | Model B: p = 0.5001 | \n|---|---|---|\n| Likelihood | 5.807714 × 10⁻³⁶² | 7.382874 × 10⁻³⁶² | \n| Natural log likelihood | −831.776616672 | −831.536640669 | \n| Normalized likelihood weight | 44.02923% | 55.97077% | \n\nBoth ordinary Python powers return 0.0. Dividing either by their sum would mean 0/0, so the direct calculation cannot produce those weights.\n\nHere, “normalized likelihood weight” means L_A / (L_A + L_B), and similarly for B. It compares these two specified models on this sequence. It is not the chance of heads on the next toss. With a separate assumption of equal prior probabilities over exactly these two models, the same numbers could be interpreted as posterior model probabilities. That extra assumption matters.\n\nOn the tested machine, Python floats use binary64 arithmetic. The smallest positive value prints as `5e-324`. Values near zero are spaced apart; a sufficiently small positive result rounds to zero. A value just below the smallest positive float can still round up to it. The halfway boundary and rounding rule matter.\n\nThere is also a region of *subnormal* values below the smallest positive normal value. They extend the available range, with less relative precision. NumPy's [floating-point limits documentation](https://numpy.org/doc/stable/reference/generated/numpy.finfo.html) explains this distinction.\n\nThe local sweep checked n = 1 through 1,300. Direct powers first became subnormal at n = 1,023 for both models. Model A first rounded to zero at n = 1,075; Model B at n = 1,076. These thresholds describe the tested binary64 routes and inputs, rather than every language, data type or device.\n\n*Agent-run arithmetic sweep, 1,300 sequence lengths. The upper chart uses log10 likelihood so the scale remains readable. The Decimal reference continues below binary64 range. The lower chart zooms into the failure region. A missing direct-power curve means normalization is undefined after both values become zero.*\n\nOkay, so perhaps we should multiply one toss at a time instead?\n\nThe sweep also computed each likelihood by repeatedly multiplying its previous value by p.\n\nAt n = 1,200, that route produced:\n\n`5e-324`.\nNormalizing those two values gives 0% and 100%.\n\nThe second model has become stuck. Multiplying the smallest positive float by 0.5001 gives an exact intermediate value just above half of that smallest float. Rounding brings it back to the same smallest value. Repeating the operation repeats the rounding. With p = 0.5, half of the smallest float is an exact tie between zero and that smallest value; binary64 round-to-nearest, ties-to-even selects zero.\n\nThe direct power and repeated product represent the same mathematical expression, but they round at different stages. The surviving nonzero value does not establish that Model B is certain.\n\n*Comparison on the same saved n = 1,200 inputs. The apparent 100% weight comes from rounding in repeated multiplication. Both direct powers are zero, so there is no valid direct-power bar to display.*\n\nThis is a useful reason to inspect a calculation even when its final weights add up to one.\n\nProducts become sums in log space:\n\n```\nlog L = log(p × p × … × p) = n × log(p)\n```\n\nThe log likelihoods around −832 fit comfortably in binary64, even though the corresponding probabilities do not.\n\nFor comparison, we also need the logarithm of the sum of the likelihoods. Computing the tiny likelihoods first would lose them again. The standard log-sum-exp identity handles this:\n\n```\nm = max(a, b)\nlog(exp(a) + exp(b)) = m + log(exp(a − m) + exp(b − m))\n```\n\nFor finite a and b, the shifted exponents are nonpositive and at least one is zero. The shared scale is removed before exponentiation. [SciPy's logsumexp](https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.logsumexp.html) implements a stable calculation, including cases beyond this two-number example.\n\nHere is the tested calculation:\n\n``` python\nimport math\nimport numpy as np\nfrom scipy.special import logsumexp\n\nn = 1200\nlog_likelihoods = np.array([\n    n * math.log(0.5),\n    n * math.log(0.5001),\n])\nweights = np.exp(log_likelihoods - logsumexp(log_likelihoods))\nprint(weights)\n# [0.44029226 0.55970774]\n```\n\nThe log-likelihood difference is approximately 0.2399760032. For just two models, we can check Model B's weight directly:\n\n```\nweight_B = 1 / (1 + exp(−0.2399760032)) ≈ 0.5597077356\n```\n\nThis is the same comparison after the common scale cancels. The calculation retains useful relative information with small rounding error; the logs are not exact.\n\nThe weights are ordinary-sized numbers because the subtraction removes the shared tiny scale. Exponentiating the original log likelihoods still returns zero on this machine. Taking a logarithm *after* a value has already underflowed cannot recover what was lost.\n\nThe checks also included inputs outside the coin example.\n\nFor log values −1,000 and −1,001, the naive exponential sum underflowed and its logarithm became minus infinity. SciPy returned approximately −999.686738312.\n\nFor 1,000 and 1,001, the naive exponentials overflowed. SciPy returned approximately 1,001.313261688.\n\nAdding a common shift of −1,000 or +1,000 to a separate two-score example preserved the normalized weights within about 3.45 × 10⁻¹⁵ in this run. That is the shared-scale cancellation in action.\n\nA genuinely impossible event still has probability zero and log probability minus infinity. Two minus-infinity inputs give a log sum of minus infinity. They do not define useful normalized model weights. A stable function cannot supply evidence that the model does not contain.\n\nThere is another boundary before multiplication: rounding the input itself. With p = 10⁻²⁰, ordinary `1 - p` rounded to 1.0, and `math.log(1 - p)` returned 0.0. `math.log1p(-p)` retained approximately −10⁻²⁰. Python documents [log1p](https://docs.python.org/3/library/math.html#math.log1p) for accurate logarithms of one plus a small input.\n\nThe local run used Python 3.13.9, NumPy 2.5.4 and SciPy 1.18.1 on macOS arm64. The inputs and tolerances were fixed before execution.\n\nFor each of the 1,300 sequence lengths, Decimal references used exact decimal inputs at both 100 and 160 digits of precision. The two reference calculations agreed to better than 10⁻⁹⁰ under the recorded relative-probability and absolute-weight checks. Small cases at n = 1, 2 and 10 also matched exact fractions.\n\nAcross the grid, the largest absolute log-likelihood error was about 1.27 × 10⁻¹³. The largest absolute normalized-weight error was about 7.01 × 10⁻¹⁴. Both were below the preset tolerances of 10⁻¹⁰ and 10⁻¹². Errors were compared against high-precision Decimal values before reducing the reported discrepancy to a float.\n\nThe comparison includes binary64 input rounding: 0.5001 has an exact decimal reference but is not exactly representable as a binary float. This small input difference is part of the measured discrepancy.\n\nThe installed SciPy source matched its [1.18.1 tagged source](https://github.com/scipy/scipy/blob/v1.18.1/scipy/special/_logsumexp.py). Reading the 1.14.1 and 1.16.2 source also showed maximum-shift calculations; those older versions were not executed. Numerical stability here is an established technique, not a new library improvement. For a deeper treatment, see Blanchard, D. J. Higham and N. J. Higham's [analysis of log-sum-exp and softmax](https://arxiv.org/abs/1909.03469).\n\nThere is also a model-choice catch. The weights compare only the candidates supplied. A tidy 56/44 split does not establish that either model describes the data well. Tiny likelihoods of exact long sequences alone do not establish that a model is wrong, because many possible sequences each have tiny probability. An all-heads count of 1,200 would be striking under either nearly fair model; checking that statistic against the model is a separate question from normalizing two likelihoods.\n\nThe experiment covers two small synthetic models and a few numerical controls. It does not establish accuracy for every distribution, backend, precision, cancellation pattern or trained model. No SciPy defect was identified.\n\nA displayed zero can therefore need two very different explanations: the model rules out the event, or the chosen arithmetic has rounded away a positive value. Keeping likelihoods in log space preserved the comparison in this example. Before interpreting a zero, ask whether it comes from what the model permits or from how the computer stores the result.", "url": "https://wpnews.pro/news/python-returned-zero-the-coin-model-still-assigned-a-positive-probability", "canonical_source": "https://dev.to/panditshivam/python-returned-zero-the-coin-model-still-assigned-a-positive-probability-4i9o", "published_at": "2026-10-19 16:40:04+00:00", "updated_at": "2026-10-19 16:44:27.406750+00:00", "lang": "en", "topics": ["machine-learning", "ai-research"], "entities": ["Python", "NumPy"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/python-returned-zero-the-coin-model-still-assigned-a-positive-probability", "markdown": "https://wpnews.pro/news/python-returned-zero-the-coin-model-still-assigned-a-positive-probability.md", "text": "https://wpnews.pro/news/python-returned-zero-the-coin-model-still-assigned-a-positive-probability.txt", "jsonld": "https://wpnews.pro/news/python-returned-zero-the-coin-model-still-assigned-a-positive-probability.jsonld"}}