Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Ali Hatamizadeh, Yejin Choi and Jan Kautz introduce Gated DeltaNet-2, a linear attention model that separates two operations in its compressed recurrent memory: erasing old content and writing new content. They argue that earlier delta-rule models tied these operations to one scalar gate, limiting how memory could be edited. Their method uses separate channel-wise erase and write gates, while retaining adaptive forgetting and channel-wise decay. The paper also describes a chunkwise WY algorithm and a gate-aware backward pass for parallel training. In experiments with a 1.3B-parameter model trained on 100B FineWeb-Edu tokens, the authors report the strongest overall results among the compared Mamba-2, Gated DeltaNet, KDA and Mamba-3 variants across language modeling, commonsense reasoning and retrieval. They say the advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, particularly multi-key retrieval, and report strong results in recurrent and hybrid settings. These findings concern the reported model size, training data and evaluations; they do not establish performance in other settings.

Publication: 51a904f3-8e05-40e5-82d3-fd9d2e47be54

attribution — pass: The draft correctly names authors Ali Hatamizadeh, Yejin Choi and Jan Kautz. It frames the method and results as the authors' claims and reported results rather than established fact.

Authors:Ali Hatamizadeh, Yejin Choi, Jan Kautz

summary — uncertain: The summary is faithful but hard for a non-specialist to follow. Central terms are not explained: linear attention, scalar gate, channel-wise, chunkwise WY algorithm, backward pass, RULER needle-in-a-haystack, and the Mamba and KDA baselines. It also omits why linear attention matters: the source says a fixed-size state replaces softmax attention's unbounded cache, giving linear-time mixing and constant-memory decoding. Smallest repair: add one plain-language sentence on that efficiency motive and why editing compressed memory is hard. Keep the technical terms exact, with tooltips. Title and question are acceptable.

Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.

date — pass: evidence.originalPublished is 2026-05-21, which matches the v1 submission date. The 6 Nov 2026 revision and the November feed date are correctly not used as the original publication date.

[Submitted on 21 May 2026 (v1), last revised 6 Nov 2026 (this version, v2)]

support — pass: The claims match the abstract: separate erase and write gates, strongest overall results among the named baselines at 1.3B parameters and 100B tokens, and the most pronounced advantage on RULER, including multi-key retrieval. The limitations properly confine the results to the evaluated setup and note that the abstract gives no numerical scores. The question follows from the evidence without leading.

Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t

media — pass: The selected media is the verified arXiv abstract page for this paper. It is the direct original source and the most relevant access point.