{"slug": "transformers-model-integration-contracts", "title": "Transformers Model Integration Contracts", "summary": "Hugging Face proposed Model Integration Contracts (MICs), a standard for packaging small \"fixture\" models with runnable usage paths and human-reviewed baselines, published in the huggingface-mic repository, after a July 30, 2026 change enabling PyTorch scaled dot-product attention (SDPA) for T5 let UMT5's decoder attend to future tokens and the bug stayed in main for almost two months. The experimental UMT5 fixture uses 66 MB of weights versus 1,227 MB for the source checkpoint, letting fast integration tests run on cheap CPU runners against a suite that otherwise spans 88,000+ CPU tests on main plus 200,000+ GPU tests in a daily run. The SmolLM contract generates from prompts such as \"The capital of France is\" and compares token IDs with a recorded answer, and also compares training loss, checks gradients are finite and verifies an optimizer step changes the weights.", "body_md": "[3.22M • Updated   •  702](https://huggingface.co/huggingface-mic/smollm-mic)    \n\n# \n\t\tTransformers Model Integration Contracts\n\t\n\n [Update on GitHub](https://github.com/huggingface/blog/blob/main/transformers-mic.md)\n\nOn July 30, 2026, Transformers enabled PyTorch's scaled dot-product attention (SDPA) for T5. As a side effect, UMT5's decoder could attend to future tokens. During training, it could peek at the answer it was supposed to predict. Nothing crashed, and the bug stayed in main for almost two months.\n\nWhen we later replayed those commits, a small UMT5 model running on CPU flagged it on every one of them. This post is about that model and the tests built around it.\n\n## \n\t\tWhy bugs reach main\n\t\n\n[Transformers](https://github.com/huggingface/transformers) supports over 750\nmodel types, and its test suite is big: **88,000+ CPU tests on main, plus\n200,000+ GPU tests in a daily run.** Model tests are only part of it: we also\ntest pipelines, training, distributed execution, quantization and attention\nkernels.\n\nThe integration tests load real checkpoints and exercise the model's API on real hardware. Running all of that on every PR would mean keeping a lot of GPUs busy all day long, which isn't practical. By the time a scheduled run finds a regression, it may already be in main.\n\nOur approach is to shrink the model instead. We build a **fixture**: a small\nmodel made from the real checkpoint's configuration, with fewer and narrower\nlayers but every kind of layer it has, and its tokenizer and processor files\nunchanged. It is published once, then reused. We run \"fast integration tests\"\nagainst it on a cheap CPU runner. In our\n[CPU study](https://github.com/huggingface/transformers-mic/blob/main/docs/evidence.md#on-cpu),\nthe experimental UMT5 fixture used 66 MB of weights; the source checkpoint\nis 1,227 MB.\n\nThis also covers very large models: some don't fit on our standard GPU runners, but their fixtures run on CPU.\n\nModel integrators already build tiny test models (Transformers' `tiny-random-*`\ncheckpoints). We wanted a common recipe, and\nchecks that go beyond loading the model and running a forward pass. The UMT5\nbug would have passed a forward pass just fine.\n\nFixtures won't replace full integration tests, but on PRs they give us a chance to catch bugs before they reach main.\n\n## \n\t\tIntroducing Model Integration Contracts\n\t\n\n[transformers-mic](https://github.com/huggingface/transformers-mic) is a proposed\nstandard for packaging these tests. A Model Integration Contract (MIC) has\nthree parts:\n\n- a **fixture** : the shrunk model described above, published once and pinned\n- a **usage path** : a runnable Markdown example of something people do with the\nmodel, such as generating text or running a training step\n- **baselines** : the outputs recorded when running that example, reviewed by a\nhuman\n\nThe contract holds if running the usage path on the fixture still produces the baselines.\n\nA contract covers an architecture and a usage path, not a single checkpoint:\ncompatible checkpoints can share a fixture, while each keeps its own expected\nresults. We publish them in [huggingface-mic](https://huggingface.co/huggingface-mic).\n\nFor example, the SmolLM contract generates from prompts such as \"The capital of France is\" and compares its token IDs with a recorded answer. After loading the model and tokenizer, the generation code is, simplified to one prompt:\n\n```\ninputs = tokenizer(\"The capital of France is\", return_tensors=\"pt\")\noutputs = model.generate(**inputs, max_new_tokens=4, do_sample=False)\nnew_tokens = outputs[:, inputs[\"input_ids\"].shape[1]:]\nprint(tokenizer.batch_decode(new_tokens, skip_special_tokens=True))\n```\n\nIt also compares training loss, checks that gradients are finite and verifies\nthat an optimizer step changes the weights. See the full\n[usage.md](https://huggingface.co/huggingface-mic/smollm-mic/blob/main/integrations/transformers/usage.md).\n\nA [fleet dataset](https://huggingface.co/datasets/huggingface-mic/transformers-fleet)\non the Hub pins each MIC to a frozen revision, along with the package versions\nused to record its baselines. Each CI run records the fleet revision it\nused, so we can reproduce it later.\n\nOn a PR, we pick MICs based on the files the patch changes.\n\n## \n\t\tThe UMT5 regression\n\t\n\nWe replayed frozen MICs over five months of Transformers commits. The small\nUMT5 fixture flagged the regression from its introduction on [July 30, 2026](https://github.com/huggingface/transformers/commit/560f36cc561fa154d224b7a6dfcba1d57a621d7b)\nuntil its fix on [September 28](https://github.com/huggingface/transformers/commit/d7ba937671edb174fd80d33f9ecbd38f6559dfca).\nLabel log-probabilities differed from the reviewed baseline by 1.22 and training\nloss by about 0.20.\n\nOne detail mattered: the contract used SDPA, the default attention path. Running only the eager attention variant missed the bug. A fixture only helps if it goes through the same code as the real checkpoint.\n\nWe then reverted the fix on a proof branch in Transformers' CI. The check failed with the same differences, and the selected T5 contract's runner step took 13.4 seconds on CPU, excluding setup.\n\nThe [replay results](https://github.com/huggingface/transformers-mic/blob/main/docs/evidence.md#shadow-backfill)\nand the [CI demo](https://github.com/huggingface/transformers-mic/blob/main/docs/evidence.md#transformers-ci-proof)\nhave the measurements and reproduction details.\n\n## \n\t\tHow models are shrunk\n\t\n\nThe first recipe was simple: make the dimensions smaller and keep fewer layers. That wasn't enough for mixed architectures. Keeping two layers is no use if both use sliding attention and the full-attention layers disappear.\n\nThe recipe now has four steps:\n\n1. **Shrink the configuration.** Start from a real checkpoint and reduce\nhidden sizes, attention heads and depth. Keep the different layer kinds,\nincluding mixtures of sliding and full attention or dense and MoE layers.\nSome architectures need extra rules to keep their components compatible.\nWe also look at the model's own Transformers tests: the`ModelTester` in`test_modeling_<model>.py` already picks a small config by hand, so we\ntake its sizes for any width our generic rule doesn't know by name.\n2. **Keep preprocessing unchanged.** Preserve vocabulary, special tokens and\ninput dimensions, and copy tokenizer and processor files byte for byte.\nLoading and preprocessing are part of what we want to test.\n3. **Build and validate the weights.** Try seeded random weights and change\nthe inputs to check that the model responds. If it doesn't, try training\nit on the contract's small batch of examples. Reload the saved model and\ncheck that outputs, loss and gradients are finite. Reject failed builds.\n4. **Freeze and record.** Save the accepted weights and measure repeatable\noutputs on a known-good Transformers build. Review the baselines for each\nsupported platform and precision, then publish a pinned snapshot. CI reuses\nit; it does not rebuild the fixture for every PR.\n\nWe also keep the original weights file format. Otherwise we would skip the\nconversion code that runs when a user loads the real checkpoint.\n[The full recipe](https://github.com/huggingface/transformers-mic/blob/main/docs/fixtures.md#recipe)\nhas more detail.\n\nThe [Llama 4 Scout shrink report](https://huggingface-mic-mic-coverage.static.hf.space/shrink/Llama-4-Scout-17B-16E-Instruct-mic.html)\nshows what this looks like on a big model. 108.6 billion parameters become 44.7\nmillion. The vision stack goes from 34 layers to 2 and the language stack from\n48 to 4, keeping the checked layer kinds and MoE routing style.\n\n### \n\t\tBuilding MICs with agents\n\t\n\nOur CLI, `tmic`, creates the contract and builds the fixture. For a new\nmodel, an LLM drafts the usage code and manifest, and CPU and GPU runs measure\nthe candidate baselines. The LLM drafts; it doesn't approve. Changed\nexpectations require human review.\n\nTo try it on your machine, use your source checkpoint in place of `org/model`:\n\n```\npip install \"transformers-mic[runtime]\" \"hf-doc-builder @ git+https://github.com/huggingface/doc-builder.git\"\nhf auth login\ntmic run org/model --local-only --fixtures-only --no-history --bill-to ''\n```\n\nIt drafts the contract, builds the fixture and records its baselines on your\nmachine, asking you to review each step, and stops before uploading anything.\nFor a new model, the draft comes from an LLM through Inference Providers, billed\nto your own account (`--bill-to ''`). The\n[CLI docs](https://github.com/huggingface/transformers-mic/blob/main/docs/cli.md#automated-mode)\ncover the individual steps.\n\nWe used agents to build MICs across model families. When they hit an edge\ncase, we fixed the recipe itself rather than the one fixture, with help from\nmodel experts. The published models and remaining gaps are in the\n[coverage space](https://huggingface.co/spaces/huggingface-mic/mic-coverage).\n\n## \n\t\tRandom vs Overfit\n\t\n\nWith random weights, our Whisper fixture barely responded to changes in the audio. It ran, but that made it a poor test of the audio path.\n\nSo we trained it on a few tones paired with words such as \"red\" and \"green\".\nWe weren't teaching it speech recognition, just making audio changes visible\nin its outputs. In our [weight-initialization study](https://github.com/huggingface/transformers-mic/blob/main/docs/fixtures.md#choosing-the-weights),\nthe random Whisper fixtures caught 18 of 24 injected faults. The overfit\nfixtures caught all 24.\n\nThat gives us two ways to build the weights:\n\n- Start with random weights and a fixed seed.\n- Overfit on a few examples, like \"The capital of France is\" => \"Paris\".\n\nThe Whisper result doesn't settle which is better. Those were synthetic faults, and we haven't found a real regression caught only by the overfit version. Both versions missed an assisted-decoding bug that the real Whisper checkpoint exposed.\n\nSo we keep random weights when they pass the checks, and overfit when we need\nbetter sensitivity or readable answers. Training takes seconds, but it is one\nmore step that can fail to converge, and one more thing to maintain. The [comparison](https://github.com/huggingface/transformers-mic/blob/main/docs/fixtures.md#overfit-versus-random-what-can-we-conclude)\nhas the results for both.\n\n## \n\t\tOpen questions\n\t\n\nAs of November 1, 2026, the [MIC job in Transformers CI](https://transformers-ci.huggingface.cool/d/mic-failures/mic-failures?from=now-4w&to=now&timezone=browser)\nreports failures without blocking PRs. Whether it should block depends on\nthings we haven't measured yet:\n\n- How many real bugs does it catch on PRs, not just in replays?\n- How often does it send us chasing false alarms?\n- How much work do baseline reviews take?\n\nWe also don't know how many bugs, like the Whisper assisted-decoding one, only show up with real weights, or whether overfitting catches real bugs that random weights miss.\n\nCoverage is still partial, across model families and the APIs people use. When a regression slips through, we can add the missing usage path and replay it on the broken and fixed versions. We'll keep existing tests until their coverage is accounted for.\n\nFinally, we'd like to link original Hub model pages to their MICs, so a developer or an agent using a checkpoint could start from an example tested on that checkpoint, its environment and its expected outputs.\n\nWant to help? Pick an uncovered model from the\n[coverage space](https://huggingface.co/spaces/huggingface-mic/mic-coverage), try\nthe [tmic workflow](https://github.com/huggingface/transformers-mic/blob/main/docs/cli.md#automated-mode),\nand contribute a usage path or review a generated contract.", "url": "https://wpnews.pro/news/transformers-model-integration-contracts", "canonical_source": "https://huggingface.co/blog/transformers-mic", "published_at": "2026-11-03 00:00:00+00:00", "updated_at": "2026-11-03 07:20:15.200284+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "developer-tools"], "entities": ["Hugging Face", "Transformers", "UMT5", "T5", "SmolLM", "transformers-mic", "huggingface-mic", "PyTorch"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/transformers-model-integration-contracts", "markdown": "https://wpnews.pro/news/transformers-model-integration-contracts.md", "text": "https://wpnews.pro/news/transformers-model-integration-contracts.txt", "jsonld": "https://wpnews.pro/news/transformers-model-integration-contracts.jsonld"}}