Tarek Ziadé · Nov 3, 2026
Tarek Ziadé proposes Model Integration Contracts for testing Transformers models
Markdown · JSON · Original source
Tarek Ziadé describes a bug in which enabling PyTorch's scaled dot-product attention (SDPA) for T5 let UMT5's decoder attend to future tokens during training. He says the bug passed a forward pass and stayed in the main code branch for almost two months, but a small model flagged it when the team replayed earlier commits. His proposed Model Integration Contracts (MICs) pair a smaller fixture model with a runnable usage example and human-reviewed expected outputs. The team runs these fast integration tests on CPU during pull requests, while retaining full integration tests. Ziadé argues that shrinking models can make tests practical for large models and catch errors that a simple loading check or forward pass would miss. The UMT5 replay found the regression, but the post does not establish how often MICs catch real bugs in pull requests or how often they produce false alarms. Ziadé says coverage is partial, and some bugs may appear only with real . As of November 1, 2026, the MIC job reported failures without blocking pull requests. How often will it catch real bugs on pull requests, and how much review work will it require?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Nov 3, 2026 (2026-11-03T00:00:00.763Z).
Date basis: Date stated by the publisher.
Added to Escape Meridian: 2026-11-10T19:02:28.475Z. First observed: 2026-11-03T07:12:47.428Z. Retrieved: 2026-11-03T14:46:28.174Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The post does not establish how often MICs catch real bugs on pull requests or how often they produce false alarms; the author says these questions have not been measured. Coverage is partial across model families and , and the post says some bugs may appear only with real weights.
Open question
How often will MICs catch real bugs on pull requests, and how much work will baseline reviews require?
Source-stated
Tarek Ziadé says enabling SDPA for T5 allowed UMT5's decoder to attend to future tokens during training.
As a side effect, UMT5's decoder could attend to future tokens. During training, it could peek at the answer it was supposed to predict.
Source-stated
Ziadé says a small UMT5 model running on CPU flagged the regression when the team replayed earlier commits.
When we later replayed those commits, a small UMT5 model running on CPU flagged it on every one of them.
Source-stated
The proposed MIC standard combines a fixture, a runnable usage path, and human-reviewed baselines.
A Model Integration Contract (MIC) has three parts: a fixture: the shrunk model described above, published once and pinned a usage path: a runnable Markdown example of something people do with the model, such as generating text or running a training step baselines: the outputs recorded when running
Source-stated
Ziadé says the MIC job was not blocking pull requests, and its real-world effectiveness and review burden remained unmeasured.
Whether it should block depends on things we haven't measured yet: How many real bugs does it catch on PRs, not just in replays? How often does it send us chasing false alarms? How much work do baseline reviews take?
Source-stated
The post says coverage is partial and some bugs may appear only with real weights.
We also don't know how many bugs, like the Whisper assisted-decoding one, only show up with real weights
Source and contributor context
Tarek Ziadé · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
CPU
The central processing unit: a general-purpose processor that carries out a computer's instructions.
Model weights
The learned numbers that make up a trained model; publishing them lets others run it.
API
An application programming interface: a defined way for software to request data or actions from another system.
Hugging Face
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Editorial provenance
Version 1. Publication ID: 949a5ab9-586d-48d8-8b70-fd457cbc636b.
Prepared by server-excerpt-repair/stream-excerpt-repair-v1 (original draft: gpt-6-luna); grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T19:02:28.475Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 8d5293f5e9d929ea252fc54ceaaaea8fa6c61ac1b465010b634bc15682ce7f71. Source text hash: 05ebd728f98a5633aea02e4146d3a4c2743312e8ac79a3663913f29cbb4abe8e.
Revision and source-check history
Version 1 · 2026-11-10T19:02:28.475Z
Tarek Ziadé proposes Model Integration Contracts for testing Transformers models
Tarek Ziadé describes a bug in which enabling PyTorch's scaled dot-product attention (SDPA) for T5 let UMT5's decoder attend to future tokens during training. He says the bug passed a forward pass and stayed in the main code branch for almost two months, but a small model flagged it when the team replayed earlier commits. His proposed Model Integration Contracts (MICs) pair a smaller fixture model with a runnable usage example and human-reviewed expected outputs. The team runs these fast integration tests on CPU during pull requests, while retaining full integration tests. Ziadé argues that shrinking models can make tests practical for large models and catch errors that a simple loading check or forward pass would miss. The UMT5 replay found the regression, but the post does not establish how often MICs catch real bugs in pull requests or how often they produce false alarms. Ziadé says coverage is partial, and some bugs may appear only with real . As of November 1, 2026, the MIC job reported failures without blocking pull requests. How often will it catch real bugs on pull requests, and how much review work will it require?
Original source
· Article
Source details
Original source date: Nov 3, 2026 (2026-11-03T00:00:00.763Z).
Date basis: Date stated by the publisher.
Added to Escape Meridian: 2026-11-10T19:02:28.475Z. First observed: 2026-11-03T07:12:47.428Z. Retrieved: 2026-11-03T14:46:28.174Z.
Limits
The post does not establish how often MICs catch real bugs on pull requests or how often they produce false alarms; the author says these questions have not been measured. Coverage is partial across model families and , and the post says some bugs may appear only with real weights.
Open question
How often will MICs catch real bugs on pull requests, and how much work will baseline reviews require?
Source-stated
Tarek Ziadé says enabling SDPA for T5 allowed UMT5's decoder to attend to future tokens during training.
As a side effect, UMT5's decoder could attend to future tokens. During training, it could peek at the answer it was supposed to predict.
Source-stated
Ziadé says a small UMT5 model running on CPU flagged the regression when the team replayed earlier commits.
When we later replayed those commits, a small UMT5 model running on CPU flagged it on every one of them.
Source-stated
The proposed MIC standard combines a fixture, a runnable usage path, and human-reviewed baselines.
A Model Integration Contract (MIC) has three parts: a fixture: the shrunk model described above, published once and pinned a usage path: a runnable Markdown example of something people do with the model, such as generating text or running a training step baselines: the outputs recorded when running
Source-stated
Ziadé says the MIC job was not blocking pull requests, and its real-world effectiveness and review burden remained unmeasured.
Whether it should block depends on things we haven't measured yet: How many real bugs does it catch on PRs, not just in replays? How often does it send us chasing false alarms? How much work do baseline reviews take?
Source-stated
The post says coverage is partial and some bugs may appear only with real weights.
We also don't know how many bugs, like the Whisper assisted-decoding one, only show up with real weights
Source and contributor context
Tarek Ziadé · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
CPU
The central processing unit: a general-purpose processor that carries out a computer's instructions.
Model weights
The learned numbers that make up a trained model; publishing them lets others run it.
API
An application programming interface: a defined way for software to request data or actions from another system.
Hugging Face
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Editorial provenance
Version 1. Publication ID: 949a5ab9-586d-48d8-8b70-fd457cbc636b.
Prepared by server-excerpt-repair/stream-excerpt-repair-v1 (original draft: gpt-6-luna); grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T19:02:28.475Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 8d5293f5e9d929ea252fc54ceaaaea8fa6c61ac1b465010b634bc15682ce7f71. Source text hash: 05ebd728f98a5633aea02e4146d3a4c2743312e8ac79a3663913f29cbb4abe8e.
uncertain · 2026-11-10T19:02:28.475Z · anthropic/claude-opus-5.5
Publication: 949a5ab9-586d-48d8-8b70-fd457cbc636b
attribution — pass: The reading correctly attributes the post and its claims to Tarek Ziadé on the Hugging Face blog. It also preserves the post's team voice ('we', 'the team') for the replay and CI work.
Tarek Ziadé tarekziade
summary — uncertain: The reading is faithful, but a non-specialist may not follow the central story. 'Attend to future tokens' is left unexplained, and the reading drops the source's plain gloss that the model 'could peek at the answer it was supposed to predict.' It also never says that Transformers is Hugging Face's widely used model library, and 'forward pass' is unexplained. Smallest repair: add the source's 'peek at the answer' phrase and a brief identification of Transformers. Separately, the closing question in the summary repeats the question field; this is stylistic, not a fidelity issue.
During training, it could peek at the answer it was supposed to predict.
date — pass: The originalPublished date of 2026-11-03 matches the page's stated publication date of November 3, 2026. The reading's 'As of November 1, 2026' matches the source.
Published November 3, 2026
support — pass: Each claim is supported by the source. These include the bug, the replay detection, the three MIC parts, the non-blocking CI status with unmeasured effectiveness, and the partial coverage. The limitations and the question reflect the author's own open questions rather than invented stakes. 'Passed a forward pass' slightly hardens the source's counterfactual 'would have passed', but this is not material.
As of November 1, 2026, the MIC job in Transformers CI reports failures without blocking PRs.
media — pass: The selected media is the verified direct link to the original Hugging Face blog post. It is the most relevant access point.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
The central processing unit: a general-purpose processor that carries out a computer's instructions.
Definition referenceThe learned numbers that make up a trained model; publishing them lets others run it.
Definition referenceHugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceAn application programming interface: a defined way for software to request data or actions from another system.
Definition referenceThe central processing unit: a general-purpose processor that carries out a computer's instructions.
Definition referenceThe learned numbers that make up a trained model; publishing them lets others run it.
Definition referenceHugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceAn application programming interface: a defined way for software to request data or actions from another system.
Definition reference