# Hugging Face Transformers MIC: Stop Silent ML Regressions > Source: > Published: 2026-11-06 16:09:03+00:00 A masking bug in Hugging Face Transformers gave UMT5’s decoder the ability to peek at future training tokens. It sat in `main` for 59 days — from July 30 to September 28, 2026 — producing no crash, no error, and no test failure. Just silently corrupted gradients, quietly undermining every fine-tune that pulled from that branch during the window. That bug is why Hugging Face shipped **Model Integration Contracts (MICs)** on November 3, 2026: a lightweight testing framework purpose-built to catch exactly this class of invisible numerical regression before it hits main. ## What Went Wrong With UMT5 The bug traces back to a routine optimization. On July 30, the Transformers team enabled PyTorch’s scaled dot-product attention (SDPA) for T5 models. The change was performance-motivated and landed cleanly — except that as a side effect, it disabled the causal mask in UMT5’s decoder. During training, the decoder could now attend to tokens it was supposed to predict, effectively leaking the answer into the input. Standard testing did not catch it. Forward passes still produced outputs. Shapes were correct. The tokenizer worked fine. The only signal was numerical: training loss behaved differently, label log-probabilities shifted. These are exactly the signals that normal unit tests do not measure. When the team later replayed the bug period against a frozen MIC fixture, the detection was immediate. A small UMT5 fixture — 66 MB, running on CPU — flagged every affected commit with a label-log-probability difference of 1.22 and a training-loss difference of around 0.20. That signal was there the whole time. There was just no test looking for it. ## The Three-Part Structure of a MIC A [Model Integration Contract](https://huggingface.co/blog/transformers-mic) packages three things into a single reproducible artifact pinned to a frozen Hub revision: - **Fixture** — A shrunken model derived from the real checkpoint’s config. Same architecture, same layer types, but reduced hidden dimensions and depth. The UMT5 fixture is 66 MB; the source checkpoint is 1,227 MB. It runs on a cheap CPU runner. - **Baselines** — Outputs recorded from running the fixture, reviewed by a human maintainer. These are the reference values that future CI runs compare against. - **Example** — Runnable usage code showing something real people do with the model: text generation, a training step, a classification forward pass. Crucially, MICs test the architecture and framework code, not a specific checkpoint. The fixture is built from the config, not downloaded weights. This keeps it fast, cheap, and architecture-portable. ## Adding a MIC With the tmic CLI The `tmic` CLI handles the full contract lifecycle. Install it with: ``` pip install "transformers-mic[runtime]" ``` To create a contract for a model: ``` tmic run your-org/your-model ``` The CLI builds the fixture, records baselines, and walks you through a review before publishing anything to the Hub. Run it with `--mode human` to review each step interactively, or `--mode agent` for a fully automated run that pauses only before upload. Add `--local-only` to stop after validation without uploading. Under the hood, the workflow covers workspace creation, fixture building, Auto-model checks, reproducible baseline recording, baseline review, validation, and fleet dataset update. For a first-time contributor, human mode works well: an LLM drafts the initial usage code and manifest, and you review and adjust before anything is committed. ## How MICs Run in CI Today The MIC job in Transformers CI triggers on changed model files. As of November 1, 2026, the job **reports failures but does not block pull requests**. That is intentional — the team wants to accumulate coverage data and calibrate signal quality before turning MIC failures into a hard merge gate. This is the right call. Shipping a gate before you understand its false-positive rate is how you create alert fatigue. The current phase is surfacing: identifying which architectures produce stable signals, which regressions MICs catch, and whether any fixtures need adjustment. Blocking comes after that data exists. ## What MIC Is Not MICs occupy a specific, narrow niche. They are not an output quality tool — if you need to evaluate whether your model’s answers are getting worse across a prompt distribution, look at [DeepEval](https://github.com/confident-ai/deepeval) or Confident AI. They are not a prompt regression framework, and they do not replace end-to-end evaluation against held-out datasets. What they do — and what nothing else in the ecosystem was doing at the framework level — is catch numerical regressions in model architecture code. The kind of bug where attention patterns change, training signal leaks, or gradient flow breaks, but nothing crashes and no metric visibly alarms until someone happens to notice a downstream model behaving oddly. ## When to Add One If you maintain a model architecture in Transformers, adding a MIC today is worthwhile. The process is straightforward, the fixture is tiny, and the [MIC repo](https://github.com/huggingface/transformers-mic) is designed for community contributions. You are not doing it because CI will block you if you do not — you are doing it because the UMT5 scenario is not a one-off. Any change that touches attention masks, SDPA dispatch, or training mode behavior is a candidate for exactly this class of silent regression. The fixture will be there to catch it. The original UMT5 bug that started all of this is tracked at [issue #49134](https://github.com/huggingface/transformers/issues/49134).