arXiv cs.AI: Breakdown of Autoregressive Multi-Step Reasoning Under Concentrated vs. Diffuse Attention Head Dropout

arXiv cs.AI: Breakdown of Autoregressive Multi-Step Reasoning Under Concentrated vs. Diffuse Attention Head Dropout

Karol Struniawski reports that concentrated bursts of attention head masking disrupted autoregressive reasoning more than budget-matched diffuse dropout in tests on GSM8K. The study evaluated 250 paired derivations across Qwen3-8B and LLaMA-3.1-8B-Instruct. At the stated onset parameter, Qwen3-8B accuracy was 78.8% under burst dropout versus 91.2% under diffuse noise; LLaMA-3.1-8B-Instruct scored 76.8% versus 84.0%. When the same cumulative budget was dispersed uniformly across generation steps, the paper reports substantially milder degradation, with both models retaining their accuracy under diffuse noise. The contrast suggests that where disruption falls in a reasoning sequence, not only the total dropout budget, may matter in these tests. An exploratory analysis associated single-step dropout density with an acute drop in one schedule draw, but found the multi-token continuity compounding trend inconclusive. Error patterns also differed: Qwen failures reflected derivation errors within formatted responses, while some LLaMA losses lacked a final answer format line. Struniawski presents the findings as empirical benchmarks for fault tolerance in transformer architectures. The evaluation covers one benchmark and two models, so whether the results extend beyond these tested conditions remains unresolved.

Publication: 0df16650-c366-4d29-844c-8be202b3adb3

attribution — pass: The reading correctly attributes the paper to Karol Struniawski on arXiv cs.AI. It keeps the results as the paper's reported findings rather than established facts.

Authors:Karol Struniawski

summary — uncertain: The numbers are faithful, but a non-specialist cannot follow the central idea. Key terms are never explained in plain language: attention head masking/dropout, autoregressive, GSM8K, onset parameter, budget-matched, schedule draw, multi-token continuity. Readers cannot tell what was disrupted or what GSM8K measures. The reading also drops the paper's motivating analogy to brief neuronal silences in tired brains (off-states). Smallest repair: one plain-language line saying the test temporarily switches off parts of the model's attention while it generates text, GSM8K is a set of grade-school math word problems, and the paper frames the work as an analogue of brain off-states.

During extended wakefulness, mammalian neocortical networks experience localized periods of transient neuronal silence termed off-states.

date — pass: The originalPublished date of 2026-11-10 matches the v1 submission date on arXiv.

[Submitted on 10 Nov 2026]

support — pass: The figures for 250 derivations, two 8B models and both accuracy pairs match the source. So do the milder degradation under dispersed budget, the exploratory acute and inconclusive results, and the error patterns. The interpretive sentence that where disruption falls may matter is hedged and limited to these tests. Its sharper source framing is concentrated versus dispersed rather than position, but it does not contradict the source. The limitations and question follow from the stated scope.

dispersing the identical cumulative budget uniformly across generation steps via stratified quintile matching yields substantially milder degradation

media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.