Semgrep posted an argument that model weights defeat inspection, leaving AI supply chains beyond audit. They are right to ask for provenance and independent verification, but the approach was such a mess that I was surprised how they decided to land.
| Open Weights Claim | Reality |
|---|---|
| The problem is lost reverse engineering | Reverse engineering is when you don’t get the source. That’s a confidentiality problem. Model risk instead is an integrity problem. The difference is in attested lineage, reproducible training, and tamper evidence on the artifact. The essay’s own conclusion talks about integrity controls, which is the right landing, but all the stuff above it is stuck on confidentiality concerns that don’t fit. |
| The problem is poisoning is undetectable, as if open weights add risk | Every detection method that exists requires some kind of access: weight diffing, trigger reconstruction, activation probing. Anthropic’s defection probes reached AUROC above 99 percent against its own |
Shadow Alignmentscenario, produces a new artifact with a new hash. That’s apples and oranges. Consumers pinned to the canonical release stay untouched. Signing and namespace controls solved this class of problem years ago. Silent model substitution is instead the default condition of API consumption, where the provider swaps weights at will. And THAT is a maddening integrity breach severely under reported. Anthropic Opus 4.7 did this in April 2026: a launch-week model that degraded within days, apostmortem admitting three silent behavior-altering changeswhile denying any inference-layer change, which demanded the need for a changelog that nobody ever saw.Sleeper Agentsfound backdoor persistence greatest in the largest models, and it places the adversary at the trainer, a threat model that actually indicts a closed distribution.Open weights mean any auditor can get an artifact.
Closed weights still mean there’s no independent auditing.