{"slug": "operator-aware-mixed-precision-tolerance-calibration-for-tensor-kernels", "title": "Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels", "summary": "A new calibration method for mixed-precision tolerance in tensor kernels, tested on the 26-entry gpuemu corpus with 8,076 result rows, achieves up to 2,184× tighter absolute tolerances than hand-picked values. For seven LLM-style buggy variants, the calibrated tolerances raise bug-detection recall from 73.2% to 82.4% (an absolute gain of 9.3 percentage points, +229 new detections) while increasing false positives from 0 to 20 out of 1,882 correct-control cases (+1.1 percentage points). The study, published on arXiv, mines element-wise error distributions from cloud GPU runs to determine the tolerance each kernel justifies under its correct implementation.", "body_md": "arXiv:2607.16228v1 Announce Type: new\nAbstract: Most tensor-kernel correctness tests go through a fixed-shape all close-style check with hand-picked absolute and relative tolerances. The thresholds are copied across the corpus and rarely revisited. We mine the element-wise error distribution of every test case from accumulated cloud GPU runs across the 26-entry gpuemu corpus and 2 dtypes (8,076 result rows). We then ask one empirical question: what absolute tolerance would the kernel itself, observed under its correct implementation, justify?\nThe answer is much tighter than the current hand-picked atol. The largest tightening is attention_triton fp16 at $2{,}184\\times$. Restricted to the seven LLM-style buggy variants for which the corpus ships a paired correct counterpart, calibrated per-(op, dtype) tolerances raise bug-detection recall from 73.2% (1,805 of 2,467) to 82.4% (2,034 of 2,467), an absolute gain of 9.3 percentage points (+229 new detections). The control false-positive count rises from 0 to 20 out of 1,882 correct-control cases (+1.1 percentage points).", "url": "https://wpnews.pro/news/operator-aware-mixed-precision-tolerance-calibration-for-tensor-kernels", "canonical_source": "https://arxiv.org/abs/2607.16228", "published_at": "2026-07-21 04:00:00+00:00", "updated_at": "2026-07-21 04:12:02.667672+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence"], "entities": ["arXiv", "gpuemu corpus"], "alternates": {"html": "https://wpnews.pro/news/operator-aware-mixed-precision-tolerance-calibration-for-tensor-kernels", "markdown": "https://wpnews.pro/news/operator-aware-mixed-precision-tolerance-calibration-for-tensor-kernels.md", "text": "https://wpnews.pro/news/operator-aware-mixed-precision-tolerance-calibration-for-tensor-kernels.txt", "jsonld": "https://wpnews.pro/news/operator-aware-mixed-precision-tolerance-calibration-for-tensor-kernels.jsonld"}}