{"id":"8c1c925c5a285aaaea408ae827e2d5daf6d046e7a402ee4dca4c075acf278faf","title":"What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models","argument":"Chang Su, Bin Wu, Emine Yilmaz and Yao Lu study how language models learn knowledge during pretraining using models trained on web snapshots from different years. They define temporal variability score (TVS) as the coefficient of variation of an item's negative log-likelihood across models; a higher score suggests the knowledge is less consistently represented across years of web text. Across six benchmarks and six models, they report that high-TVS items improve faster later in training than low-TVS items. At the end of pretraining, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. The authors say the pattern is most robust on knowledge-intensive tasks, including comparisons between items of similar difficulty. Their results suggest TVS may help diagnose how knowledge is learned beyond aggregate benchmark scores, but the abstract does not establish whether the pattern generalizes beyond the models and benchmarks studied. How well does this pattern hold for models trained on other web corpora or with different training setups?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence provided is the paper's abstract; it does not give details about the benchmarks, models, training setup, or the size and composition of the evaluated items. The reported benchmark results do not establish whether the pattern generalizes to other models, corpora, or training setups.","speaker":{"name":"Su, Chang, Wu, Bin, Yilmaz, Emine, Lu, Yao","roleAtSource":"Article author"},"source":{"title":"What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.37631","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors define temporal variability score (TVS) as the coefficient of variation of an item's negative log-likelihood across models; higher TVS suggests knowledge is less consistently represented across years of web text.","excerpt":"we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text.","url":"https://arxiv.org/abs/2610.37631","relationship":"Source-stated"},{"text":"Across six benchmarks and six models, the authors report that high- and low-TVS items are learned similarly early in training, but high-TVS items improve faster later.","excerpt":"We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on.","url":"https://arxiv.org/abs/2610.37631","relationship":"Source-stated"},{"text":"At the end of pretraining, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy.","excerpt":"At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy.","url":"https://arxiv.org/abs/2610.37631","relationship":"Source-stated"},{"text":"The authors report the pattern is most robust on knowledge-intensive tasks, including when comparing items of similar difficulty.","excerpt":"This pattern is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty.","url":"https://arxiv.org/abs/2610.37631","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.37631","checkedAt":"2026-11-06T21:04:06.339Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well does this pattern hold for models trained on other web corpora or with different training setups?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models","summary":"Chang Su, Bin Wu, Emine Yilmaz and Yao Lu study how language models learn knowledge during pretraining using models trained on web snapshots from different years. They define temporal variability score (TVS) as the coefficient of variation of an item's negative log-likelihood across models; a higher score suggests the knowledge is less consistently represented across years of web text. Across six benchmarks and six models, they report that high-TVS items improve faster later in training than low-TVS items. At the end of pretraining, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. The authors say the pattern is most robust on knowledge-intensive tasks, including comparisons between items of similar difficulty. Their results suggest TVS may help diagnose how knowledge is learned beyond aggregate benchmark scores, but the abstract does not establish whether the pattern generalizes beyond the models and benchmarks studied. How well does this pattern hold for models trained on other web corpora or with different training setups?","question":"How well does this pattern hold for models trained on other web corpora or with different training setups?","claims":[{"text":"The authors define temporal variability score (TVS) as the coefficient of variation of an item's negative log-likelihood across models; higher TVS suggests knowledge is less consistently represented across years of web text.","excerpt":"we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text.","support":"source-stated"},{"text":"Across six benchmarks and six models, the authors report that high- and low-TVS items are learned similarly early in training, but high-TVS items improve faster later.","excerpt":"We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on.","support":"source-stated"},{"text":"At the end of pretraining, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy.","excerpt":"At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy.","support":"source-stated"},{"text":"The authors report the pattern is most robust on knowledge-intensive tasks, including when comparing items of similar difficulty.","excerpt":"This pattern is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty.","support":"source-stated"}],"limitations":["The evidence provided is the paper's abstract; it does not give details about the benchmarks, models, training setup, or the size and composition of the evaluated items.","The reported benchmark results do not establish whether the pattern generalizes to other models, corpora, or training setups."],"mediaId":"b0365f12d87e1bf756f4456b9d18f6845c364f6d2775192209fd67c7bda75eb0","correctionReason":null,"id":"8c1c925c5a285aaaea408ae827e2d5daf6d046e7a402ee4dca4c075acf278faf","publicationId":"544b68cf-c975-4d55-89f3-638b5271fc71","version":1,"sourceUrl":"https://arxiv.org/abs/2610.37631","sourceTitle":"What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T08:30:33.480Z","createdAt":"2026-11-06T20:01:54.657Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"b0365f12d87e1bf756f4456b9d18f6845c364f6d2775192209fd67c7bda75eb0","kind":"article","url":"https://arxiv.org/abs/2610.37631","sourceUrl":"https://arxiv.org/abs/2610.37631","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T03:33:24.245Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.37631","checkedAt":"2026-11-06T21:04:06.339Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"b0365f12d87e1bf756f4456b9d18f6845c364f6d2775192209fd67c7bda75eb0","confidence":0.81,"provider":"typesafe","model":"jev-1.13.0","inputHash":"2f1bff57cc240279d5032a1271d4c49a5578e384e665ba3c6938fd24ff39be5e","decidedAt":"2026-10-30T03:33:25.331Z"},"provenance":{"textHash":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","observationId":"6d0247897e1dbfa5fb49bc092aa7498271f0e506074563ea30953d7f82556299","retrievedAt":"2026-10-30T03:33:24.083Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"504508bb-589c-4950-84f6-9b1133409b54"},"reviewer":{"model":"gpt-6-luna","runId":"6f887167-5736-4035-82ba-12b0c976c06c"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-06T20:01:54.657Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Su, Chang","url":null,"kind":"unknown"},{"name":"Wu, Bin","url":null,"kind":"unknown"},{"name":"Yilmaz, Emine","url":null,"kind":"unknown"},{"name":"Lu, Yao","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Su, Chang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Su, Chang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37631","locator":"head/meta[name=author|citation_author]"},"assertionId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:33:24.083Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0","sourceUrl":"https://arxiv.org/abs/2610.37631","contentSha256":"84efcd409664c77ea8c7d3d30fc436bc6ed4a1b5659e1e0d0322855973c86dbf","textHash":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","observedAt":"2026-10-30T03:33:24.083Z"}},{"actor":{"name":"Wu, Bin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Bin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37631","locator":"head/meta[name=author|citation_author]"},"assertionId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:33:24.083Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0","sourceUrl":"https://arxiv.org/abs/2610.37631","contentSha256":"84efcd409664c77ea8c7d3d30fc436bc6ed4a1b5659e1e0d0322855973c86dbf","textHash":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","observedAt":"2026-10-30T03:33:24.083Z"}},{"actor":{"name":"Yilmaz, Emine","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yilmaz, Emine","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37631","locator":"head/meta[name=author|citation_author]"},"assertionId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:33:24.083Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0","sourceUrl":"https://arxiv.org/abs/2610.37631","contentSha256":"84efcd409664c77ea8c7d3d30fc436bc6ed4a1b5659e1e0d0322855973c86dbf","textHash":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","observedAt":"2026-10-30T03:33:24.083Z"}},{"actor":{"name":"Lu, Yao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lu, Yao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37631","locator":"head/meta[name=author|citation_author]"},"assertionId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:33:24.083Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0","sourceUrl":"https://arxiv.org/abs/2610.37631","contentSha256":"84efcd409664c77ea8c7d3d30fc436bc6ed4a1b5659e1e0d0322855973c86dbf","textHash":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","observedAt":"2026-10-30T03:33:24.083Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.37631","snapshotId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0","contentSha256":"84efcd409664c77ea8c7d3d30fc436bc6ed4a1b5659e1e0d0322855973c86dbf","textHash":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","observedAt":"2026-10-30T03:33:24.083Z","basis":"publisher-page-metadata","sourceLinks":42,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/8c1c925c5a285aaaea408ae827e2d5daf6d046e7a402ee4dca4c075acf278faf/source"}},"canonicalUrl":"https://escapemeridian.com/readings/what-improves-during-pretraining-tracking-knowledge-acquisition-8c1c925c","formats":{"html":"/readings/what-improves-during-pretraining-tracking-knowledge-acquisition-8c1c925c","markdown":"/readings/what-improves-during-pretraining-tracking-knowledge-acquisition-8c1c925c.md","json":"/readings/what-improves-during-pretraining-tracking-knowledge-acquisition-8c1c925c.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-pre-training","name":"Pre-training","explanation":"A model's initial training on a large dataset, before further adaptation for particular uses.","referenceUrl":"https://developers.google.com/machine-learning/glossary#pre-training"},{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"time","name":"TIME","kind":"organization","context":"TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.","referenceUrls":["https://time.com/"],"path":"/entities/time","json":"/entities/time.json","markdown":"/entities/time.md"},{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"8c1c925c5a285aaaea408ae827e2d5daf6d046e7a402ee4dca4c075acf278faf","revision":"544b68cf-c975-4d55-89f3-638b5271fc71","revisionKind":"publication-id","title":"What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models","url":"https://escapemeridian.com/readings/what-improves-during-pretraining-tracking-knowledge-acquisition-8c1c925c"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.37631","revision":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","revisionKind":"sha256-source-text","title":"What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models","url":"https://arxiv.org/abs/2610.37631"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"8c1c925c5a285aaaea408ae827e2d5daf6d046e7a402ee4dca4c075acf278faf","title":"What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models","argument":"Chang Su, Bin Wu, Emine Yilmaz and Yao Lu study how language models learn knowledge during pretraining using models trained on web snapshots from different years. They define temporal variability score (TVS) as the coefficient of variation of an item's negative log-likelihood across models; a higher score suggests the knowledge is less consistently represented across years of web text. Across six benchmarks and six models, they report that high-TVS items improve faster later in training than low-TVS items. At the end of pretraining, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. The authors say the pattern is most robust on knowledge-intensive tasks, including comparisons between items of similar difficulty. Their results suggest TVS may help diagnose how knowledge is learned beyond aggregate benchmark scores, but the abstract does not establish whether the pattern generalizes beyond the models and benchmarks studied. How well does this pattern hold for models trained on other web corpora or with different training setups?","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence provided is the paper's abstract; it does not give details about the benchmarks, models, training setup, or the size and composition of the evaluated items. The reported benchmark results do not establish whether the pattern generalizes to other models, corpora, or training setups.","speaker":{"name":"Su, Chang, Wu, Bin, Yilmaz, Emine, Lu, Yao","roleAtSource":"Article author"},"source":{"title":"What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.37631","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors define temporal variability score (TVS) as the coefficient of variation of an item's negative log-likelihood across models; higher TVS suggests knowledge is less consistently represented across years of web text.","excerpt":"we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text.","url":"https://arxiv.org/abs/2610.37631","relationship":"Source-stated"},{"text":"Across six benchmarks and six models, the authors report that high- and low-TVS items are learned similarly early in training, but high-TVS items improve faster later.","excerpt":"We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on.","url":"https://arxiv.org/abs/2610.37631","relationship":"Source-stated"},{"text":"At the end of pretraining, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy.","excerpt":"At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy.","url":"https://arxiv.org/abs/2610.37631","relationship":"Source-stated"},{"text":"The authors report the pattern is most robust on knowledge-intensive tasks, including when comparing items of similar difficulty.","excerpt":"This pattern is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty.","url":"https://arxiv.org/abs/2610.37631","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.37631","checkedAt":"2026-11-06T21:04:06.339Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well does this pattern hold for models trained on other web corpora or with different training setups?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models","summary":"Chang Su, Bin Wu, Emine Yilmaz and Yao Lu study how language models learn knowledge during pretraining using models trained on web snapshots from different years. They define temporal variability score (TVS) as the coefficient of variation of an item's negative log-likelihood across models; a higher score suggests the knowledge is less consistently represented across years of web text. Across six benchmarks and six models, they report that high-TVS items improve faster later in training than low-TVS items. At the end of pretraining, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. The authors say the pattern is most robust on knowledge-intensive tasks, including comparisons between items of similar difficulty. Their results suggest TVS may help diagnose how knowledge is learned beyond aggregate benchmark scores, but the abstract does not establish whether the pattern generalizes beyond the models and benchmarks studied. How well does this pattern hold for models trained on other web corpora or with different training setups?","question":"How well does this pattern hold for models trained on other web corpora or with different training setups?","claims":[{"text":"The authors define temporal variability score (TVS) as the coefficient of variation of an item's negative log-likelihood across models; higher TVS suggests knowledge is less consistently represented across years of web text.","excerpt":"we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text.","support":"source-stated"},{"text":"Across six benchmarks and six models, the authors report that high- and low-TVS items are learned similarly early in training, but high-TVS items improve faster later.","excerpt":"We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on.","support":"source-stated"},{"text":"At the end of pretraining, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy.","excerpt":"At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy.","support":"source-stated"},{"text":"The authors report the pattern is most robust on knowledge-intensive tasks, including when comparing items of similar difficulty.","excerpt":"This pattern is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty.","support":"source-stated"}],"limitations":["The evidence provided is the paper's abstract; it does not give details about the benchmarks, models, training setup, or the size and composition of the evaluated items.","The reported benchmark results do not establish whether the pattern generalizes to other models, corpora, or training setups."],"mediaId":"b0365f12d87e1bf756f4456b9d18f6845c364f6d2775192209fd67c7bda75eb0","correctionReason":null,"id":"8c1c925c5a285aaaea408ae827e2d5daf6d046e7a402ee4dca4c075acf278faf","publicationId":"544b68cf-c975-4d55-89f3-638b5271fc71","version":1,"sourceUrl":"https://arxiv.org/abs/2610.37631","sourceTitle":"What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-28T08:30:33.480Z","createdAt":"2026-11-06T20:01:54.657Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"b0365f12d87e1bf756f4456b9d18f6845c364f6d2775192209fd67c7bda75eb0","kind":"article","url":"https://arxiv.org/abs/2610.37631","sourceUrl":"https://arxiv.org/abs/2610.37631","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T03:33:24.245Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.37631","checkedAt":"2026-11-06T21:04:06.339Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"b0365f12d87e1bf756f4456b9d18f6845c364f6d2775192209fd67c7bda75eb0","confidence":0.81,"provider":"typesafe","model":"jev-1.13.0","inputHash":"2f1bff57cc240279d5032a1271d4c49a5578e384e665ba3c6938fd24ff39be5e","decidedAt":"2026-10-30T03:33:25.331Z"},"provenance":{"textHash":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","observationId":"6d0247897e1dbfa5fb49bc092aa7498271f0e506074563ea30953d7f82556299","retrievedAt":"2026-10-30T03:33:24.083Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"504508bb-589c-4950-84f6-9b1133409b54"},"reviewer":{"model":"gpt-6-luna","runId":"6f887167-5736-4035-82ba-12b0c976c06c"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-06T20:01:54.657Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Su, Chang","url":null,"kind":"unknown"},{"name":"Wu, Bin","url":null,"kind":"unknown"},{"name":"Yilmaz, Emine","url":null,"kind":"unknown"},{"name":"Lu, Yao","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Su, Chang","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Su, Chang","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37631","locator":"head/meta[name=author|citation_author]"},"assertionId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:33:24.083Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0","sourceUrl":"https://arxiv.org/abs/2610.37631","contentSha256":"84efcd409664c77ea8c7d3d30fc436bc6ed4a1b5659e1e0d0322855973c86dbf","textHash":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","observedAt":"2026-10-30T03:33:24.083Z"}},{"actor":{"name":"Wu, Bin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Bin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37631","locator":"head/meta[name=author|citation_author]"},"assertionId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:33:24.083Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0","sourceUrl":"https://arxiv.org/abs/2610.37631","contentSha256":"84efcd409664c77ea8c7d3d30fc436bc6ed4a1b5659e1e0d0322855973c86dbf","textHash":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","observedAt":"2026-10-30T03:33:24.083Z"}},{"actor":{"name":"Yilmaz, Emine","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yilmaz, Emine","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37631","locator":"head/meta[name=author|citation_author]"},"assertionId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:33:24.083Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0","sourceUrl":"https://arxiv.org/abs/2610.37631","contentSha256":"84efcd409664c77ea8c7d3d30fc436bc6ed4a1b5659e1e0d0322855973c86dbf","textHash":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","observedAt":"2026-10-30T03:33:24.083Z"}},{"actor":{"name":"Lu, Yao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Lu, Yao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37631","locator":"head/meta[name=author|citation_author]"},"assertionId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T03:33:24.083Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0","sourceUrl":"https://arxiv.org/abs/2610.37631","contentSha256":"84efcd409664c77ea8c7d3d30fc436bc6ed4a1b5659e1e0d0322855973c86dbf","textHash":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","observedAt":"2026-10-30T03:33:24.083Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.37631","snapshotId":"c49c786280cea4620fbfd0f00f52b8b4b8f501fce08f76e02d594f791537d3c0","contentSha256":"84efcd409664c77ea8c7d3d30fc436bc6ed4a1b5659e1e0d0322855973c86dbf","textHash":"bc5dfd491526590c0a2b7eb6479c212691082f37307967bd2e6b0046e8339202","observedAt":"2026-10-30T03:33:24.083Z","basis":"publisher-page-metadata","sourceLinks":42,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/8c1c925c5a285aaaea408ae827e2d5daf6d046e7a402ee4dca4c075acf278faf/source"}}}],"audits":[{"publicationId":"544b68cf-c975-4d55-89f3-638b5271fc71","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading correctly credits the paper to Chang Su, Bin Wu, Emine Yilmaz and Yao Lu. It keeps the findings attributed to them ('they report', 'the authors say').","excerpt":"Authors:Chang Su, Bin Wu, Emine Yilmaz, Yao Lu"},{"dimension":"summary","status":"uncertain","explanation":"The central measure is defined only through unexplained jargon. A non-specialist cannot tell what TVS captures or what a 7.3-point advantage means. The terms are 'coefficient of variation', 'negative log-likelihood', 'pretraining', 'point-in-time models' and 'chance-normalized accuracy'. Smallest repair: keep the terms exact and add term-library tooltips (suggest additions if missing), plus one plain line on what high TVS means. The summary also drops the source hedge 'may be less consistently represented'; restore 'may'. Separately, the question is repeated verbatim at the end of the summary.","excerpt":"A higher score suggests that the knowledge may be less consistently represented across years of web text."},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-10-27, which matches the v1 submission date. The feed date of 2026-10-29 is later and correctly not used.","excerpt":"[Submitted on 27 Oct 2026]"},{"dimension":"support","status":"pass","explanation":"Every claim matches the abstract verbatim in substance: the TVS definition, six benchmarks and six models, similar early learning followed by faster later gains, 34 of 36 settings with a 7.3-point average, and robustness on knowledge-intensive tasks. The diagnostic framing is hedged ('may help'), and the generalization question is flagged as unknown rather than asserted.","excerpt":"At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified arXiv abstract page for this paper. It is the direct original source.","excerpt":null}],"createdAt":"2026-11-06T20:01:54.657Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}