Same same, but faster At the 2026 Triton and PyTorch conferences in San Jose, ByteDance researchers presented OpGuard, a tool that compares two training runs operation-by-operation and reports the first op where bits differ, uncovering a race condition in PyTorch's embedding backwards that had persisted for 7 years while barely affecting loss (about 1e-5). LinkedIn presented a people-search system that scores member matches via GPU embeddings in FP8, where roughly a third of top-10 results shifted versus the FP16 reference even though LLM-scored relevance did not move, leading LinkedIn to keep the FP16 copy in DRAM and rerank FP8 candidates against it. AMD's effort to run DeepSeek V4 Flash reinforcement learning on MI355X with SGLang rollouts and Megatron training hit disagreements — SGLang scored experts with a sigmoid where Megatron used sqrt softplus, plus scale and clamp config differences — and the team used an independent HuggingFace version as the reference to break ties. This week I’ve been at the Triton https://tritonconference2026.splashthat.com/ and PyTorch https://events.linuxfoundation.org/pytorch-conference-north-america/?utm source=google&utm medium=paid-search&utm campaign=41307495-PyTorch%20Conference%202026%20-%20North%20America&utm term=Events PyTorch-Conference-North-America-2026 NA Conversions Intent Search PyTorch BoFU&campaignid=24057000471&adgroupid=197348859286&creative=818279663933&matchtype=e&network=g&device=c&keyword=pytorch%20conference%202026&utm term=pytorch%20conference%202026&utm campaign=Events+%7C+PyTorch+Conference+North+America+2026+%7C+NA+%7C+Conversions+%7C+Intent+%7C+Search+%7C+PyTorch+%7C+BoFU&utm source=google&utm medium=ppc&utm id=24057000471&hsa acc=8666746580&hsa cam=24057000471&hsa grp=197348859286&hsa ad=818279663933&hsa src=g&hsa tgt=kwd-2463335132350&hsa kw=pytorch%20conference%202026&hsa mt=e&hsa net=adwords&hsa ver=3&gad source=1&gad campaignid=24057000471&gbraid=0AAAAAD2R-lq4GbiHTgX4i2MZII6fY40NW&gclid=CjwKCAjwxeHWBhBPEiwAiqtQ-dAt4xCSyvXOWEaukrLcL2ob5dlWTpTqEPVd0bUeBVj0EnDYwASsURoCTv0QAvD BwE conferences in San Jose, and yesterday I had a chance to browse around the PyTorch poster presentations. There’s a lot of talk about agentic development, of course, and about how to scale it. Much of that comes down to verification. But what are we verifying against ? The theme from many of the posters was: we are verifying the code against… some other code. Which is odd We could just check whether our models are working properly, but models are surprisingly forgiving: they will happily train straight through a bug. Folks from ByteDance presented OpGuard https://www.usenix.org/conference/osdi26/presentation/zhou-ziming which compares two training runs op-by-op and reports the first op where bits differ. They found race in PyTorch’s embedding backwards that had been there for 7 years. The bug hardly affected the loss about 1e-5 LinkedIn presented https://arxiv.org/abs/2608.28968 their people search, which scores member matched via embedding on GPU in FP8. Compared to their FP16 reference about a third of the top 10 results shifted, but the LLM scored relevance didn’t move a bit. This forgiveness is a nice feature most of the time, but makes validation difficult. LinkedIN ended up keeping the FP16 copy in DRAM and reranks the FP8 candidates with it. Metric wise it isn’t it was just… there first. These reference implementations are becoming a kind of an extant spec, that will be then processed by agents under some set of constraints around exactly how close an optimized version has to be. Guilherme Leobas at OpenTeams building on an idea from Richard Zou has been rebuilding Dynamo’s the PyTorch compile front end object model to mirror CPython’s directly, using Python’s own test suite as the reference. The HuggingFace folks have been enabling on-device models for WebGPU https://huggingface.co/blog/webgpu-kernels , where they match against a CPU reference for correctness. Nvidia presented a “Day-0 RL” poster that was basically a checklist for using something as a reference: use identical weights and inputs, check trainer and rollout log-probs agree, and monitor more than just the reward score. As if to demonstrate why, AMD presented on getting DeepSeek V4 Flash RL on MI355X https://www.lmsys.org/blog/2026-07-10-rocm-miles-dsv4 . They had SGLang doing rollouts and Megatron doing training, and ran into disagreements. It wasn’t the usual suspects either, SGLang scored experts with a sigmoid where Meatron used sqrt softplus