Agarwal, Shradha, Rajbhar, Deepak, J, Tariq · May 15, 2026
LinAlg-Bench: A Forensic Benchmark Revealing Structural Failure Modes in LLM Mathematical Reasoning
Markdown · JSON · Original source
Shradha Agarwal, Deepak Rajbhar and Tariq J present LinAlg-Bench, a diagnostic of 10 frontier solving structured linear algebra problems on 3x3, 4x4 and 5x5 matrices. The authors argue that the models’ errors follow a pattern rather than occurring at random: at 3x3, failures tend to involve execution errors such as sign tracking and arithmetic, while at 4x4 and 5x5, models often abandon computation and produce fabricated responses. They describe this shift as near-universal across model tiers and architectures, and suggest it reflects a working memory limit rather than a knowledge gap. The benchmark uses 660 SymPy-certified problems across 9 task types and analyzes 6,600 outputs; its forensic pipeline classifies failures into error tags and subtypes. The authors also report that solution strategy rigidity is a near-perfect predictor of 5x5 determinant accuracy, and identify constraint-aware confabulation as a structured failure mode. These findings make the benchmark a way to examine not just whether models get answers wrong, but how their failures change with problem scale. The claims concern this benchmark’s tasks and evaluated models; the supplied abstract does not establish how the findings extend beyond them.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: May 15, 2026 (2026-05-15).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T00:18:21.904Z. First observed: 2026-11-06T05:10:43.724Z. Retrieved: 2026-11-09T00:17:35.968Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the abstract, not the paper’s full methods or results; it does not provide the details needed to assess the construction or the reported analyses. The findings described concern the evaluated models and the benchmark’s structured linear algebra tasks; the abstract does not establish whether the patterns extend to other tasks or settings.
Open question
What evidence in the paper supports the authors’ interpretation that the shift in failures reflects a working memory limit rather than a knowledge gap?
Source-stated
The authors introduce LinAlg-Bench to evaluate 10 frontier large language models on structured linear algebra computation across 3x3, 4x4 and 5x5 matrices.
evaluating 10 frontier large language models on structured linear algebra computation across a strict dimensional gradient of 3x3, 4x4, and 5x5 matrices
Source-stated
The authors report a change in failure patterns: execution errors occur below 4x4 scale, while at larger scales models shift toward computational abandonment and fabricated responses.
below it, models fail through execution errors -- sign tracking failures, arithmetic drift, and parity errors; above it, failure transitions to computational abandonment
Source-stated
The authors say the fabrication-to-abandonment transition is near-universal across model tiers and architectures and suggest it reflects a working memory limit rather than a knowledge gap.
suggesting a working memory limit rather than a knowledge gap
Source-stated
The authors report that solution strategy rigidity is a near-perfect predictor of 5x5 determinant accuracy.
solution strategy rigidity is a near-perfect predictor of 5x5 determinant accuracy
Source-stated
The authors describe constraint-aware confabulation as a novel structured hallucination failure mode.
document constraint-aware confabulation as a novel structured hallucination failure mode
Source and contributor context
Agarwal, Shradha, Rajbhar, Deepak, J, Tariq · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Hallucination
When an AI system states something false or made up as if it were true.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 8f726e9b-117f-43ae-8896-14577d0e7054.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T00:18:21.904Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: b6612c31a31fa692ddc1ed6661a639feef7cd8acf8a2f3a516de45de85b8226a. Source text hash: 88f7b1bf4e51e2a7cc3841ae16056aba3e07e17ee22a6e789489934e0cb0c044.
Revision and source-check history
Version 1 · 2026-11-09T00:18:21.904Z
LinAlg-Bench: A Forensic Benchmark Revealing Structural Failure Modes in LLM Mathematical Reasoning
Shradha Agarwal, Deepak Rajbhar and Tariq J present LinAlg-Bench, a diagnostic of 10 frontier solving structured linear algebra problems on 3x3, 4x4 and 5x5 matrices. The authors argue that the models’ errors follow a pattern rather than occurring at random: at 3x3, failures tend to involve execution errors such as sign tracking and arithmetic, while at 4x4 and 5x5, models often abandon computation and produce fabricated responses. They describe this shift as near-universal across model tiers and architectures, and suggest it reflects a working memory limit rather than a knowledge gap. The benchmark uses 660 SymPy-certified problems across 9 task types and analyzes 6,600 outputs; its forensic pipeline classifies failures into error tags and subtypes. The authors also report that solution strategy rigidity is a near-perfect predictor of 5x5 determinant accuracy, and identify constraint-aware confabulation as a structured failure mode. These findings make the benchmark a way to examine not just whether models get answers wrong, but how their failures change with problem scale. The claims concern this benchmark’s tasks and evaluated models; the supplied abstract does not establish how the findings extend beyond them.
Original source
cs.AI · Article
Source details
Original source date: May 15, 2026 (2026-05-15).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T00:18:21.904Z. First observed: 2026-11-06T05:10:43.724Z. Retrieved: 2026-11-09T00:17:35.968Z.
Limits
The supplied evidence is the abstract, not the paper’s full methods or results; it does not provide the details needed to assess the construction or the reported analyses. The findings described concern the evaluated models and the benchmark’s structured linear algebra tasks; the abstract does not establish whether the patterns extend to other tasks or settings.
Open question
What evidence in the paper supports the authors’ interpretation that the shift in failures reflects a working memory limit rather than a knowledge gap?
Source-stated
The authors introduce LinAlg-Bench to evaluate 10 frontier large language models on structured linear algebra computation across 3x3, 4x4 and 5x5 matrices.
evaluating 10 frontier large language models on structured linear algebra computation across a strict dimensional gradient of 3x3, 4x4, and 5x5 matrices
Source-stated
The authors report a change in failure patterns: execution errors occur below 4x4 scale, while at larger scales models shift toward computational abandonment and fabricated responses.
below it, models fail through execution errors -- sign tracking failures, arithmetic drift, and parity errors; above it, failure transitions to computational abandonment
Source-stated
The authors say the fabrication-to-abandonment transition is near-universal across model tiers and architectures and suggest it reflects a working memory limit rather than a knowledge gap.
suggesting a working memory limit rather than a knowledge gap
Source-stated
The authors report that solution strategy rigidity is a near-perfect predictor of 5x5 determinant accuracy.
solution strategy rigidity is a near-perfect predictor of 5x5 determinant accuracy
Source-stated
The authors describe constraint-aware confabulation as a novel structured hallucination failure mode.
document constraint-aware confabulation as a novel structured hallucination failure mode
Source and contributor context
Agarwal, Shradha, Rajbhar, Deepak, J, Tariq · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Hallucination
When an AI system states something false or made up as if it were true.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 8f726e9b-117f-43ae-8896-14577d0e7054.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T00:18:21.904Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: b6612c31a31fa692ddc1ed6661a639feef7cd8acf8a2f3a516de45de85b8226a. Source text hash: 88f7b1bf4e51e2a7cc3841ae16056aba3e07e17ee22a6e789489934e0cb0c044.
uncertain · 2026-11-09T00:18:21.904Z · anthropic/claude-opus-5.5
Publication: 8f726e9b-117f-43ae-8896-14577d0e7054
attribution — pass: The reading correctly credits Shradha Agarwal, Deepak Rajbhar and Tariq J and frames the findings and interpretations as the authors' claims rather than established facts.
Authors:Shradha Agarwal, Deepak Rajbhar, Tariq J
summary — uncertain: The summary assigns 4x4 to the abandonment regime ('at 4x4 and 5x5, models often abandon computation'). The abstract places the threshold at 4x4, with execution errors below it and abandonment above it, so 4x4's side is ambiguous. Smallest repair: say failures shift at a threshold around 4x4. Central terms such as SymPy-certified, determinant, frontier models and confabulation are also left unexplained for non-specialist readers. Term-library tooltips or a brief plain gloss would fix this.
a sharp behavioral threshold at 4x4 scale: below it, models fail through execution errors
date — pass: The originalPublished date of 2026-05-15 matches the v1 submission. It correctly uses the original submission rather than the 4 Nov 2026 revision or the feed date.
Submitted on 15 May 2026 (v1), last revised 4 Nov 2026 (this version, v2)
support — pass: The claims and their excerpts match the abstract. The working-memory interpretation is kept as the authors' suggestion. The limitations accurately note that the evidence is abstract-only and that scope extends only to these tasks. The question follows from the evidence without leading.
suggesting a working memory limit rather than a knowledge gap
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceWhen an AI system states something false or made up as if it were true.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceWhen an AI system states something false or made up as if it were true.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference