Biggs, Scott, Soro, Bedionita, Gupta, Saumya, Andreis, Bruno, Hwang, Sung Ju · Nov 4, 2026
arXiv cs.AI: A Night at the LLM Zoo—Generating pretrained LLM weights from data mixtures
Markdown · JSON · Original source
The authors of “A Night at the Zoo” argue that a curated population of large language models can help extend Weight Space Learning—the study and generation of models through their weights—to LLMs. They trained populations of GPT-2 models at three parameter sizes, varying the proportions of web text, books, mathematics, coding and French in the training data. The authors report that variation among these models is structured by those data mixtures. They then conditioned a Flow Matching model on mixture composition to generate complete LLM weights in Gram-PCA compressed weight space. On held-out mixtures, the generator showed generalization; the authors also report that generated models performed similarly to models in their zoo, with minimal changes in downstream perplexity and general capability scores. The work offers a way to study generating models for data mixtures not used to train the generator, but its evidence comes from the described GPT-2 populations and evaluations. The abstract does not give the size or range of the held-out mixtures, or the detailed results behind its performance comparisons. The authors say they release source code and artifacts to support further work in Weight Space Learning and Generation.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 4, 2026 (2026-11-04).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T03:54:04.011Z. First observed: 2026-11-06T05:07:13.370Z. Retrieved: 2026-11-09T03:53:11.499Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract does not specify the number or range of held-out mixtures, or provide detailed results for the reported performance comparisons. The reported study uses GPT-2 model populations at the parameter sizes and with the data-mixture categories described in the abstract; the abstract does not establish how well the findings extend beyond them.
Open question
How broadly does the reported generalization hold across mixtures beyond those evaluated in this study?
Source-stated
The authors created GPT-2 model populations with varied proportions of five kinds of training data, at three parameter sizes.
systematically varying the data mixture composition between web text, books, mathematics, coding, and French across populations of 51M, 124M, and 355M parameter GPT-2 Models
Source-stated
The authors report that model variation in their zoo is structured by training mixture composition.
the zoo contains diverse models whose variation is structured by training mixture composition
Source-stated
The authors conditioned a Flow Matching model on training mixture composition to generate pretrained LLM weights in a compressed weight space.
We embed the training mixture composition to condition a Flow Matching model of complete pretrained LLM weights in Gram-PCA compressed weight space.
Source-stated
The authors report zero-shot generalization to held-out mixtures and similar performance to models in their zoo, with minimal changes on the stated evaluations.
By demonstrating zero-shot generalization to held out mixtures, measuring Continuous Ranked Probability Score skill, as well as finding minimal changes in downstream test-set perplexity and general capability benchmark scores
Source and contributor context
Biggs, Scott, Soro, Bedionita, Gupta, Saumya, Andreis, Bruno, Hwang, Sung Ju · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Pre-training
A model's initial training on a large dataset, before further adaptation for particular uses.
Zero-shot
Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Held-out set
Data kept aside during training and used only to test how well a model works on examples it has not seen.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 7793042a-593e-4c12-8c90-ad00f8b9be73.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T03:54:04.011Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 71f154218c80a43177a09f603a97d518517b32b259f19e9b8bd772b58272dd29. Source text hash: 41ec07656ad44df7bcbdd10cdc4bac6f699b5c0e9fde1ff9d07aa526a3a56d88.
Revision and source-check history
Version 1 · 2026-11-09T03:54:04.011Z
arXiv cs.AI: A Night at the LLM Zoo—Generating pretrained LLM weights from data mixtures
The authors of “A Night at the Zoo” argue that a curated population of large language models can help extend Weight Space Learning—the study and generation of models through their weights—to LLMs. They trained populations of GPT-2 models at three parameter sizes, varying the proportions of web text, books, mathematics, coding and French in the training data. The authors report that variation among these models is structured by those data mixtures. They then conditioned a Flow Matching model on mixture composition to generate complete LLM weights in Gram-PCA compressed weight space. On held-out mixtures, the generator showed generalization; the authors also report that generated models performed similarly to models in their zoo, with minimal changes in downstream perplexity and general capability scores. The work offers a way to study generating models for data mixtures not used to train the generator, but its evidence comes from the described GPT-2 populations and evaluations. The abstract does not give the size or range of the held-out mixtures, or the detailed results behind its performance comparisons. The authors say they release source code and artifacts to support further work in Weight Space Learning and Generation.
Original source
cs.AI · Article
Source details
Original source date: Nov 4, 2026 (2026-11-04).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T03:54:04.011Z. First observed: 2026-11-06T05:07:13.370Z. Retrieved: 2026-11-09T03:53:11.499Z.
Limits
The abstract does not specify the number or range of held-out mixtures, or provide detailed results for the reported performance comparisons. The reported study uses GPT-2 model populations at the parameter sizes and with the data-mixture categories described in the abstract; the abstract does not establish how well the findings extend beyond them.
Open question
How broadly does the reported generalization hold across mixtures beyond those evaluated in this study?
Source-stated
The authors created GPT-2 model populations with varied proportions of five kinds of training data, at three parameter sizes.
systematically varying the data mixture composition between web text, books, mathematics, coding, and French across populations of 51M, 124M, and 355M parameter GPT-2 Models
Source-stated
The authors report that model variation in their zoo is structured by training mixture composition.
the zoo contains diverse models whose variation is structured by training mixture composition
Source-stated
The authors conditioned a Flow Matching model on training mixture composition to generate pretrained LLM weights in a compressed weight space.
We embed the training mixture composition to condition a Flow Matching model of complete pretrained LLM weights in Gram-PCA compressed weight space.
Source-stated
The authors report zero-shot generalization to held-out mixtures and similar performance to models in their zoo, with minimal changes on the stated evaluations.
By demonstrating zero-shot generalization to held out mixtures, measuring Continuous Ranked Probability Score skill, as well as finding minimal changes in downstream test-set perplexity and general capability benchmark scores
Source and contributor context
Biggs, Scott, Soro, Bedionita, Gupta, Saumya, Andreis, Bruno, Hwang, Sung Ju · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Pre-training
A model's initial training on a large dataset, before further adaptation for particular uses.
Zero-shot
Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Held-out set
Data kept aside during training and used only to test how well a model works on examples it has not seen.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 7793042a-593e-4c12-8c90-ad00f8b9be73.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T03:54:04.011Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 71f154218c80a43177a09f603a97d518517b32b259f19e9b8bd772b58272dd29. Source text hash: 41ec07656ad44df7bcbdd10cdc4bac6f699b5c0e9fde1ff9d07aa526a3a56d88.
uncertain · 2026-11-09T03:54:04.011Z · anthropic/claude-opus-5.5
Publication: 7793042a-593e-4c12-8c90-ad00f8b9be73
attribution — pass: The reading credits the arXiv preprint's named authors and publisher, and mostly frames results as what the authors report. One sentence states 'the generator showed zero-shot generalization' without attribution; the summary finding addresses this.
Authors:Scott Biggs, Bedionita Soro, Saumya Gupta, Bruno Andreis, Sung Ju Hwang
summary — uncertain: The draft is faithful but hard for a non-specialist to follow. It never says what model 'weights' are (the learned numerical parameters). It leaves Flow Matching, Gram-PCA, zero-shot and perplexity unexplained. It also drops the abstract's central 'ex nihilo' idea: models are generated without being trained. Smallest repairs: define weights in plain words, add term tooltips, state that the generated models are produced without training, and attribute the zero-shot sentence to the authors ('the authors report').
we show that our generator generalizes on the simplex of data mixtures to produce models that perform similarly to those in the zoo ex nihilo
date — pass: originalPublished 2026-11-04 matches the v1 submission date on arXiv. It is not the later feed date.
[Submitted on 4 Nov 2026]
support — pass: Each claim and excerpt matches the abstract. The limitations correctly note what the abstract does not specify. The question is open rather than leading, and no unsupported benefits or costs are implied.
finding minimal changes in downstream test-set perplexity and general capability benchmark scores
media — pass: The selected media is the verified link to the arXiv abstract page itself, which is the most direct way to reach the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceA model's initial training on a large dataset, before further adaptation for particular uses.
Definition referenceDoing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Definition referenceData kept aside during training and used only to test how well a model works on examples it has not seen.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivDoing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Definition referenceData kept aside during training and used only to test how well a model works on examples it has not seen.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceA model's initial training on a large dataset, before further adaptation for particular uses.
Definition referenceDoing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Definition referenceData kept aside during training and used only to test how well a model works on examples it has not seen.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivDoing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).
Definition referenceData kept aside during training and used only to test how well a model works on examples it has not seen.
Definition reference