Koriagin, Nikita, Balagansky, Nikita, Gavrilov, Daniil · Nov 5, 2026
How Much Data SAE Training Needs
Markdown · JSON · Original source
Authors Nikita Koriagin, Nikita Balagansky and Daniil Gavrilov study how much activation data sparse autoencoder (SAE) training needs when its total training budget is fixed. They report that quality improves as the activation grows, then reaches a plateau: replaying that buffer is nearly as effective as using fresh activations throughout training. Beyond the plateau, they say, adding unique activations brings little benefit to reconstruction or . The authors describe this diversity-versus-repetition tradeoff with a for limited data and report testing it across model layers and sizes, SAE types, token budgets and activation sources. Their practical rule is to use the smallest buffer that reaches the quality plateau, then replay it for the remaining budget. In their experiments, this reduced activation storage by 8–64× with negligible quality loss. The result could help reduce a storage bottleneck in SAE training, but the reported savings and quality outcome are experimental findings, not a guarantee for other settings.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 5, 2026 (2026-11-05).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T17:46:56.079Z. First observed: 2026-11-06T05:00:59.872Z. Retrieved: 2026-11-08T17:46:15.004Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied abstract gives no numeric sizes or per-condition results, so it does not show how the reported plateau or storage reduction varies across the tested settings.
Open question
How does the quality plateau change when the activation source changes?
Source-stated
The authors report that SAE quality improves as the activation buffer grows, then reaches a plateau where replaying the buffer is nearly as effective as using fresh activations throughout training.
Quality improves rapidly as the buffer grows, then reaches a plateau where replaying the buffer is nearly as effective as using fresh activations throughout training.
Source-stated
The authors say additional unique activations beyond the plateau bring little benefit to reconstruction or interpretability.
Beyond this point, additional unique activations bring little benefit to reconstruction or interpretability
Source-stated
The authors recommend choosing the smallest buffer that reaches the quality plateau and replaying it for the remaining training budget.
choose the smallest buffer that reaches the quality plateau and replay it for the remaining budget.
Source-stated
In their experiments, the authors report reducing activation storage by 8–64× with negligible quality loss.
In our experiments, this reduces activation storage by $8\text{--}64\times$ with negligible quality loss.
Source-stated
The authors report testing the tradeoff across dictionary sizes, token budgets, model layers and sizes, SAE types, and activation sources.
We validate it across dictionary sizes and token budgets, early, middle, and late layers of Llama-3.1, Qwen3 model sizes, and BatchTopK, TopK, and JumpReLU SAEs.
Source and contributor context
Koriagin, Nikita, Balagansky, Nikita, Gavrilov, Daniil · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Interpretability
Research into how a model reaches its outputs, often by examining its internal workings.
Scaling laws
Observed patterns in which model performance improves predictably as data, model size and computation grow.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Buffer
Buffer makes social media management software and publishes company accounts of customer support, team practices and online publishing.
Editorial provenance
Version 1. Publication ID: 1e4588f7-3a93-4627-8fec-168b41da258a.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T17:46:56.079Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 1d57713265820b7b1359184088c4e2708a09ba79c33cd54ea6e83f3edd361f33. Source text hash: 7cebd910327edf450b7058546d3f48ef01d6dbdf31e5d205fd533f4752ea9f8d.
Revision and source-check history
Version 1 · 2026-11-08T17:46:56.079Z
How Much Data SAE Training Needs
Authors Nikita Koriagin, Nikita Balagansky and Daniil Gavrilov study how much activation data sparse autoencoder (SAE) training needs when its total training budget is fixed. They report that quality improves as the activation grows, then reaches a plateau: replaying that buffer is nearly as effective as using fresh activations throughout training. Beyond the plateau, they say, adding unique activations brings little benefit to reconstruction or . The authors describe this diversity-versus-repetition tradeoff with a for limited data and report testing it across model layers and sizes, SAE types, token budgets and activation sources. Their practical rule is to use the smallest buffer that reaches the quality plateau, then replay it for the remaining budget. In their experiments, this reduced activation storage by 8–64× with negligible quality loss. The result could help reduce a storage bottleneck in SAE training, but the reported savings and quality outcome are experimental findings, not a guarantee for other settings.
Original source
cs.AI · Article
Source details
Original source date: Nov 5, 2026 (2026-11-05).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-08T17:46:56.079Z. First observed: 2026-11-06T05:00:59.872Z. Retrieved: 2026-11-08T17:46:15.004Z.
Limits
The supplied abstract gives no numeric sizes or per-condition results, so it does not show how the reported plateau or storage reduction varies across the tested settings.
Open question
How does the quality plateau change when the activation source changes?
Source-stated
The authors report that SAE quality improves as the activation buffer grows, then reaches a plateau where replaying the buffer is nearly as effective as using fresh activations throughout training.
Quality improves rapidly as the buffer grows, then reaches a plateau where replaying the buffer is nearly as effective as using fresh activations throughout training.
Source-stated
The authors say additional unique activations beyond the plateau bring little benefit to reconstruction or interpretability.
Beyond this point, additional unique activations bring little benefit to reconstruction or interpretability
Source-stated
The authors recommend choosing the smallest buffer that reaches the quality plateau and replaying it for the remaining training budget.
choose the smallest buffer that reaches the quality plateau and replay it for the remaining budget.
Source-stated
In their experiments, the authors report reducing activation storage by 8–64× with negligible quality loss.
In our experiments, this reduces activation storage by $8\text{--}64\times$ with negligible quality loss.
Source-stated
The authors report testing the tradeoff across dictionary sizes, token budgets, model layers and sizes, SAE types, and activation sources.
We validate it across dictionary sizes and token budgets, early, middle, and late layers of Llama-3.1, Qwen3 model sizes, and BatchTopK, TopK, and JumpReLU SAEs.
Source and contributor context
Koriagin, Nikita, Balagansky, Nikita, Gavrilov, Daniil · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Interpretability
Research into how a model reaches its outputs, often by examining its internal workings.
Scaling laws
Observed patterns in which model performance improves predictably as data, model size and computation grow.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Buffer
Buffer makes social media management software and publishes company accounts of customer support, team practices and online publishing.
Editorial provenance
Version 1. Publication ID: 1e4588f7-3a93-4627-8fec-168b41da258a.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-08T17:46:56.079Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 1d57713265820b7b1359184088c4e2708a09ba79c33cd54ea6e83f3edd361f33. Source text hash: 7cebd910327edf450b7058546d3f48ef01d6dbdf31e5d205fd533f4752ea9f8d.
uncertain · 2026-11-08T17:46:56.079Z · anthropic/claude-opus-5.5
Publication: 1e4588f7-3a93-4627-8fec-168b41da258a
attribution — pass: The draft correctly attributes the study to Nikita Koriagin, Nikita Balagansky and Daniil Gavrilov. It frames the findings, rule and storage savings as the authors' reports rather than established facts.
Authors:Nikita Koriagin, Nikita Balagansky, Daniil Gavrilov
summary — uncertain: The summary is faithful to the abstract, but central jargon is unexplained for non-specialists: 'sparse autoencoder (SAE)', 'activations', 'activation buffer', 'replaying', 'interpretability' and 'scaling law'. A reader is not told that SAEs are tools for inspecting what happens inside language models, so the significance is unclear. Smallest repair: add a plain-language gloss of SAE and activations (or term-library tooltips) and one sentence on why SAE training matters. The title and question are otherwise fine; the question follows from the source's activation-source sweeps, whose results the abstract does not report.
Activation storage is often a bottleneck in sparse autoencoder (SAE) training.
date — pass: originalPublished 2026-11-05 matches the arXiv v1 submission date. The feed date of Nov 6 is a later announcement.
[v1] Thu, 5 Nov 2026 10:00:55 UTC (89 KB)
support — pass: Every claim matches the abstract, including activation sources, which are covered by the math text and Python code sweeps. The storage-bottleneck significance is grounded in the abstract. The 8–64x figure is scoped to the authors' experiments and is caveated as not generalizing, so nothing is overstated. The limitation note accurately says the abstract lacks per-condition numbers.
Sweeps using mathematical text and Python code test the effect of changing the activation source.
media — pass: The selected media is the verified arXiv abstract page for this paper. It is the direct original source and the most relevant link.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Buffer makes social media management software and publishes company accounts of customer support, team practices and online publishing.
Explore BufferResearch into how a model reaches its outputs, often by examining its internal workings.
Definition referenceObserved patterns in which model performance improves predictably as data, model size and computation grow.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivBuffer makes social media management software and publishes company accounts of customer support, team practices and online publishing.
Explore BufferBuffer makes social media management software and publishes company accounts of customer support, team practices and online publishing.
Explore BufferResearch into how a model reaches its outputs, often by examining its internal workings.
Definition referenceObserved patterns in which model performance improves predictably as data, model size and computation grow.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivBuffer makes social media management software and publishes company accounts of customer support, team practices and online publishing.
Explore Buffer