{"id":"487b80422c3f5824904b7054967bfec67590cf522159dbb7441e68644744b5cb","title":"Authors report signs of non-public books in OpenAI's GPT-4o responses","argument":"In “Beyond Public Access in LLM Pre-Training Data,” Sruly Rosenblat, Tim O'Reilly and Ilan Strauss test whether OpenAI models recognize copyrighted books that were not licensed for training. They use the DE-COP membership inference attack, a method for testing whether a model behaves as if it recognizes examples in a dataset, on 34 legally obtained O'Reilly Media books.\n\nThe authors report that GPT-4o’s results are consistent with recognition of the non-public books, while GPT-3.5 Turbo and GPT-4o Mini score near chance. The reported AUROC score measures how well a test distinguishes examples; the authors give a score of 0.82 for GPT-4o, with a wide 95% bootstrapped CI, or confidence interval, of 0.60–0.96. They say their results support disclosure requirements for pre-training data provenance and formal licensing markets for training content.\n\nThe result matters to the authors because it concerns possible access violations, not only whether copyrighted material appeared in training data. But they describe the findings as preliminary: the book sample is small, post-cutoff controls may be contaminated, and the models differ in size, architecture and training composition. An open question is whether larger studies can separate recognition of non-public content from these limitations.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The authors describe the results as preliminary, based on a small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture and training composition. The evidence provided is the paper's abstract, not its full methods or results; it does not establish that the models were trained on the books or that any legal violation occurred.","speaker":{"name":"Rosenblat, Sruly, O'Reilly, Tim, Strauss, Ilan","roleAtSource":"Article author"},"source":{"title":"Beyond Public Access in LLM Pre-Training Data","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2505.00020","publishedAt":"2025-04-24","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors test OpenAI models for recognition of non-public copyrighted books using DE-COP membership inference attack on 34 legally obtained O'Reilly Media books.","excerpt":"Using a legally obtained dataset of 34 copyrighted O'Reilly Media books (13,962 paragraphs), we apply the DE-COP membership inference attack to test whether OpenAI models show recognition of non-public (pay-walled) content they were not licensed to train on.","url":"https://arxiv.org/abs/2505.00020","relationship":"Source-stated"},{"text":"The authors report GPT-4o results consistent with recognition of non-public books, with an AUROC score of 0.82 and a 95% bootstrapped CI of 0.60–0.96.","excerpt":"Our results suggest that GPT-4o, OpenAI's more recent and capable model in our sample, exhibits patterns consistent with recognition of non-public books, with an AUROC score of 0.82 (95% bootstrapped CI: 0.60-0.96)","url":"https://arxiv.org/abs/2505.00020","relationship":"Source-stated"},{"text":"The authors say GPT-3.5 Turbo and GPT-4o Mini score near chance recognition on non-public data.","excerpt":"GPT-3.5 Turbo (trained two years prior) and the smaller GPT-4o Mini both score near chance recognition on non-public data (0.54 and 0.56, respectively).","url":"https://arxiv.org/abs/2505.00020","relationship":"Source-stated"},{"text":"The authors say their findings support disclosure requirements for pre-training data provenance and formal licensing markets for training content.","excerpt":"These findings support disclosure requirements for pre-training data provenance and formal licensing markets for training content.","url":"https://arxiv.org/abs/2505.00020","relationship":"Source-stated"},{"text":"The authors call the results preliminary, citing sample size, possible control contamination and differences across models.","excerpt":"Results remain preliminary given the small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture, and training composition.","url":"https://arxiv.org/abs/2505.00020","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2505.00020","checkedAt":"2026-11-10T11:48:15.076Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Can larger studies distinguish recognition of non-public books from possible contamination of post-cutoff controls and differences in model size, architecture and training composition?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Authors report signs of non-public books in OpenAI's GPT-4o responses","summary":"In “Beyond Public Access in LLM Pre-Training Data,” Sruly Rosenblat, Tim O'Reilly and Ilan Strauss test whether OpenAI models recognize copyrighted books that were not licensed for training. They use the DE-COP membership inference attack, a method for testing whether a model behaves as if it recognizes examples in a dataset, on 34 legally obtained O'Reilly Media books.\n\nThe authors report that GPT-4o’s results are consistent with recognition of the non-public books, while GPT-3.5 Turbo and GPT-4o Mini score near chance. The reported AUROC score measures how well a test distinguishes examples; the authors give a score of 0.82 for GPT-4o, with a wide 95% bootstrapped CI, or confidence interval, of 0.60–0.96. They say their results support disclosure requirements for pre-training data provenance and formal licensing markets for training content.\n\nThe result matters to the authors because it concerns possible access violations, not only whether copyrighted material appeared in training data. But they describe the findings as preliminary: the book sample is small, post-cutoff controls may be contaminated, and the models differ in size, architecture and training composition. An open question is whether larger studies can separate recognition of non-public content from these limitations.","question":"Can larger studies distinguish recognition of non-public books from possible contamination of post-cutoff controls and differences in model size, architecture and training composition?","claims":[{"text":"The authors test OpenAI models for recognition of non-public copyrighted books using DE-COP membership inference attack on 34 legally obtained O'Reilly Media books.","excerpt":"Using a legally obtained dataset of 34 copyrighted O'Reilly Media books (13,962 paragraphs), we apply the DE-COP membership inference attack to test whether OpenAI models show recognition of non-public (pay-walled) content they were not licensed to train on.","support":"source-stated"},{"text":"The authors report GPT-4o results consistent with recognition of non-public books, with an AUROC score of 0.82 and a 95% bootstrapped CI of 0.60–0.96.","excerpt":"Our results suggest that GPT-4o, OpenAI's more recent and capable model in our sample, exhibits patterns consistent with recognition of non-public books, with an AUROC score of 0.82 (95% bootstrapped CI: 0.60-0.96)","support":"source-stated"},{"text":"The authors say GPT-3.5 Turbo and GPT-4o Mini score near chance recognition on non-public data.","excerpt":"GPT-3.5 Turbo (trained two years prior) and the smaller GPT-4o Mini both score near chance recognition on non-public data (0.54 and 0.56, respectively).","support":"source-stated"},{"text":"The authors say their findings support disclosure requirements for pre-training data provenance and formal licensing markets for training content.","excerpt":"These findings support disclosure requirements for pre-training data provenance and formal licensing markets for training content.","support":"source-stated"},{"text":"The authors call the results preliminary, citing sample size, possible control contamination and differences across models.","excerpt":"Results remain preliminary given the small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture, and training composition.","support":"source-stated"}],"limitations":["The authors describe the results as preliminary, based on a small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture and training composition.","The evidence provided is the paper's abstract, not its full methods or results; it does not establish that the models were trained on the books or that any legal violation occurred."],"mediaId":"a7172f676882f31c3ce0ff83f0601b27e4e0ebcfac381ab7fc1cb7b6da5091b8","correctionReason":null,"id":"487b80422c3f5824904b7054967bfec67590cf522159dbb7441e68644744b5cb","publicationId":"c548b352-561d-4b60-8cdb-89188886709d","version":1,"sourceUrl":"https://arxiv.org/abs/2505.00020","sourceTitle":"Beyond Public Access in LLM Pre-Training Data","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2025-04-24","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-02T05:01:03.527Z","createdAt":"2026-11-10T11:17:10.360Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"a7172f676882f31c3ce0ff83f0601b27e4e0ebcfac381ab7fc1cb7b6da5091b8","kind":"article","url":"https://arxiv.org/abs/2505.00020","sourceUrl":"https://arxiv.org/abs/2505.00020","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-03T04:17:51.265Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2505.00020","checkedAt":"2026-11-10T11:48:15.076Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"a7172f676882f31c3ce0ff83f0601b27e4e0ebcfac381ab7fc1cb7b6da5091b8","confidence":0.8800000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"5da56c805d4fbc48530bfbfe19e2c2a086392c60b36e7254c64e03c0647541b4","decidedAt":"2026-11-03T04:17:53.698Z"},"provenance":{"textHash":"a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb","observationId":"039c72213927620573d22d3673c818acf5e328106d9169e6e46cfb41263110aa","retrievedAt":"2026-11-03T04:17:50.963Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"2a65fe87-ad67-4390-8d0e-6fb17c441d5b"},"reviewer":{"model":"gpt-6-luna","runId":"fd4d720c-6968-4e44-8861-ede438b67a11"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-10T11:17:10.360Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Rosenblat, Sruly","url":null,"kind":"unknown"},{"name":"O'Reilly, Tim","url":null,"kind":"unknown"},{"name":"Strauss, Ilan","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Rosenblat, Sruly","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Rosenblat, Sruly","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2505.00020","locator":"head/meta[name=author|citation_author]"},"assertionId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T11:48:15.076Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721","sourceUrl":"https://arxiv.org/abs/2505.00020","contentSha256":"ab5a77fff461f17738e38541cd7803ebf7c1f3a1ec05163108ef2c0b06f9c3e5","textHash":"a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb","observedAt":"2026-11-10T11:48:15.076Z"}},{"actor":{"name":"O'Reilly, Tim","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"O'Reilly, Tim","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2505.00020","locator":"head/meta[name=author|citation_author]"},"assertionId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T11:48:15.076Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721","sourceUrl":"https://arxiv.org/abs/2505.00020","contentSha256":"ab5a77fff461f17738e38541cd7803ebf7c1f3a1ec05163108ef2c0b06f9c3e5","textHash":"a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb","observedAt":"2026-11-10T11:48:15.076Z"}},{"actor":{"name":"Strauss, Ilan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Strauss, Ilan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2505.00020","locator":"head/meta[name=author|citation_author]"},"assertionId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T11:48:15.076Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721","sourceUrl":"https://arxiv.org/abs/2505.00020","contentSha256":"ab5a77fff461f17738e38541cd7803ebf7c1f3a1ec05163108ef2c0b06f9c3e5","textHash":"a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb","observedAt":"2026-11-10T11:48:15.076Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2505.00020","snapshotId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721","contentSha256":"ab5a77fff461f17738e38541cd7803ebf7c1f3a1ec05163108ef2c0b06f9c3e5","textHash":"a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb","observedAt":"2026-11-10T11:48:15.076Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/487b80422c3f5824904b7054967bfec67590cf522159dbb7441e68644744b5cb/source"}},"canonicalUrl":"https://escapemeridian.com/readings/authors-report-signs-of-non-public-books-in-openai-s-gpt-4o-487b8042","formats":{"html":"/readings/authors-report-signs-of-non-public-books-in-openai-s-gpt-4o-487b8042","markdown":"/readings/authors-report-signs-of-non-public-books-in-openai-s-gpt-4o-487b8042.md","json":"/readings/authors-report-signs-of-non-public-books-in-openai-s-gpt-4o-487b8042.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-pre-training","name":"Pre-training","explanation":"A model's initial training on a large dataset, before further adaptation for particular uses.","referenceUrl":"https://developers.google.com/machine-learning/glossary#pre-training"},{"id":"term-confidence-interval","name":"Confidence interval","explanation":"A range of values consistent with the data at a stated level of confidence.","referenceUrl":"https://en.wikipedia.org/wiki/Confidence_interval"}],"entities":[{"id":"openai","name":"OpenAI","kind":"organization","context":"OpenAI develops AI models and products and publishes research, release announcements and evaluations.","referenceUrls":["https://openai.com/news/"],"path":"/entities/openai","json":"/entities/openai.json","markdown":"/entities/openai.md"},{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"487b80422c3f5824904b7054967bfec67590cf522159dbb7441e68644744b5cb","revision":"c548b352-561d-4b60-8cdb-89188886709d","revisionKind":"publication-id","title":"Authors report signs of non-public books in OpenAI's GPT-4o responses","url":"https://escapemeridian.com/readings/authors-report-signs-of-non-public-books-in-openai-s-gpt-4o-487b8042"},"source":{"type":"source","id":"https://arxiv.org/abs/2505.00020","revision":"a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb","revisionKind":"sha256-source-text","title":"Beyond Public Access in LLM Pre-Training Data","url":"https://arxiv.org/abs/2505.00020"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"487b80422c3f5824904b7054967bfec67590cf522159dbb7441e68644744b5cb","title":"Authors report signs of non-public books in OpenAI's GPT-4o responses","argument":"In “Beyond Public Access in LLM Pre-Training Data,” Sruly Rosenblat, Tim O'Reilly and Ilan Strauss test whether OpenAI models recognize copyrighted books that were not licensed for training. They use the DE-COP membership inference attack, a method for testing whether a model behaves as if it recognizes examples in a dataset, on 34 legally obtained O'Reilly Media books.\n\nThe authors report that GPT-4o’s results are consistent with recognition of the non-public books, while GPT-3.5 Turbo and GPT-4o Mini score near chance. The reported AUROC score measures how well a test distinguishes examples; the authors give a score of 0.82 for GPT-4o, with a wide 95% bootstrapped CI, or confidence interval, of 0.60–0.96. They say their results support disclosure requirements for pre-training data provenance and formal licensing markets for training content.\n\nThe result matters to the authors because it concerns possible access violations, not only whether copyrighted material appeared in training data. But they describe the findings as preliminary: the book sample is small, post-cutoff controls may be contaminated, and the models differ in size, architecture and training composition. An open question is whether larger studies can separate recognition of non-public content from these limitations.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The authors describe the results as preliminary, based on a small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture and training composition. The evidence provided is the paper's abstract, not its full methods or results; it does not establish that the models were trained on the books or that any legal violation occurred.","speaker":{"name":"Rosenblat, Sruly, O'Reilly, Tim, Strauss, Ilan","roleAtSource":"Article author"},"source":{"title":"Beyond Public Access in LLM Pre-Training Data","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2505.00020","publishedAt":"2025-04-24","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors test OpenAI models for recognition of non-public copyrighted books using DE-COP membership inference attack on 34 legally obtained O'Reilly Media books.","excerpt":"Using a legally obtained dataset of 34 copyrighted O'Reilly Media books (13,962 paragraphs), we apply the DE-COP membership inference attack to test whether OpenAI models show recognition of non-public (pay-walled) content they were not licensed to train on.","url":"https://arxiv.org/abs/2505.00020","relationship":"Source-stated"},{"text":"The authors report GPT-4o results consistent with recognition of non-public books, with an AUROC score of 0.82 and a 95% bootstrapped CI of 0.60–0.96.","excerpt":"Our results suggest that GPT-4o, OpenAI's more recent and capable model in our sample, exhibits patterns consistent with recognition of non-public books, with an AUROC score of 0.82 (95% bootstrapped CI: 0.60-0.96)","url":"https://arxiv.org/abs/2505.00020","relationship":"Source-stated"},{"text":"The authors say GPT-3.5 Turbo and GPT-4o Mini score near chance recognition on non-public data.","excerpt":"GPT-3.5 Turbo (trained two years prior) and the smaller GPT-4o Mini both score near chance recognition on non-public data (0.54 and 0.56, respectively).","url":"https://arxiv.org/abs/2505.00020","relationship":"Source-stated"},{"text":"The authors say their findings support disclosure requirements for pre-training data provenance and formal licensing markets for training content.","excerpt":"These findings support disclosure requirements for pre-training data provenance and formal licensing markets for training content.","url":"https://arxiv.org/abs/2505.00020","relationship":"Source-stated"},{"text":"The authors call the results preliminary, citing sample size, possible control contamination and differences across models.","excerpt":"Results remain preliminary given the small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture, and training composition.","url":"https://arxiv.org/abs/2505.00020","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2505.00020","checkedAt":"2026-11-10T11:48:15.076Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Can larger studies distinguish recognition of non-public books from possible contamination of post-cutoff controls and differences in model size, architecture and training composition?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Authors report signs of non-public books in OpenAI's GPT-4o responses","summary":"In “Beyond Public Access in LLM Pre-Training Data,” Sruly Rosenblat, Tim O'Reilly and Ilan Strauss test whether OpenAI models recognize copyrighted books that were not licensed for training. They use the DE-COP membership inference attack, a method for testing whether a model behaves as if it recognizes examples in a dataset, on 34 legally obtained O'Reilly Media books.\n\nThe authors report that GPT-4o’s results are consistent with recognition of the non-public books, while GPT-3.5 Turbo and GPT-4o Mini score near chance. The reported AUROC score measures how well a test distinguishes examples; the authors give a score of 0.82 for GPT-4o, with a wide 95% bootstrapped CI, or confidence interval, of 0.60–0.96. They say their results support disclosure requirements for pre-training data provenance and formal licensing markets for training content.\n\nThe result matters to the authors because it concerns possible access violations, not only whether copyrighted material appeared in training data. But they describe the findings as preliminary: the book sample is small, post-cutoff controls may be contaminated, and the models differ in size, architecture and training composition. An open question is whether larger studies can separate recognition of non-public content from these limitations.","question":"Can larger studies distinguish recognition of non-public books from possible contamination of post-cutoff controls and differences in model size, architecture and training composition?","claims":[{"text":"The authors test OpenAI models for recognition of non-public copyrighted books using DE-COP membership inference attack on 34 legally obtained O'Reilly Media books.","excerpt":"Using a legally obtained dataset of 34 copyrighted O'Reilly Media books (13,962 paragraphs), we apply the DE-COP membership inference attack to test whether OpenAI models show recognition of non-public (pay-walled) content they were not licensed to train on.","support":"source-stated"},{"text":"The authors report GPT-4o results consistent with recognition of non-public books, with an AUROC score of 0.82 and a 95% bootstrapped CI of 0.60–0.96.","excerpt":"Our results suggest that GPT-4o, OpenAI's more recent and capable model in our sample, exhibits patterns consistent with recognition of non-public books, with an AUROC score of 0.82 (95% bootstrapped CI: 0.60-0.96)","support":"source-stated"},{"text":"The authors say GPT-3.5 Turbo and GPT-4o Mini score near chance recognition on non-public data.","excerpt":"GPT-3.5 Turbo (trained two years prior) and the smaller GPT-4o Mini both score near chance recognition on non-public data (0.54 and 0.56, respectively).","support":"source-stated"},{"text":"The authors say their findings support disclosure requirements for pre-training data provenance and formal licensing markets for training content.","excerpt":"These findings support disclosure requirements for pre-training data provenance and formal licensing markets for training content.","support":"source-stated"},{"text":"The authors call the results preliminary, citing sample size, possible control contamination and differences across models.","excerpt":"Results remain preliminary given the small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture, and training composition.","support":"source-stated"}],"limitations":["The authors describe the results as preliminary, based on a small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture and training composition.","The evidence provided is the paper's abstract, not its full methods or results; it does not establish that the models were trained on the books or that any legal violation occurred."],"mediaId":"a7172f676882f31c3ce0ff83f0601b27e4e0ebcfac381ab7fc1cb7b6da5091b8","correctionReason":null,"id":"487b80422c3f5824904b7054967bfec67590cf522159dbb7441e68644744b5cb","publicationId":"c548b352-561d-4b60-8cdb-89188886709d","version":1,"sourceUrl":"https://arxiv.org/abs/2505.00020","sourceTitle":"Beyond Public Access in LLM Pre-Training Data","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2025-04-24","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-02T05:01:03.527Z","createdAt":"2026-11-10T11:17:10.360Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"a7172f676882f31c3ce0ff83f0601b27e4e0ebcfac381ab7fc1cb7b6da5091b8","kind":"article","url":"https://arxiv.org/abs/2505.00020","sourceUrl":"https://arxiv.org/abs/2505.00020","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-03T04:17:51.265Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2505.00020","checkedAt":"2026-11-10T11:48:15.076Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"a7172f676882f31c3ce0ff83f0601b27e4e0ebcfac381ab7fc1cb7b6da5091b8","confidence":0.8800000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"5da56c805d4fbc48530bfbfe19e2c2a086392c60b36e7254c64e03c0647541b4","decidedAt":"2026-11-03T04:17:53.698Z"},"provenance":{"textHash":"a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb","observationId":"039c72213927620573d22d3673c818acf5e328106d9169e6e46cfb41263110aa","retrievedAt":"2026-11-03T04:17:50.963Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"2a65fe87-ad67-4390-8d0e-6fb17c441d5b"},"reviewer":{"model":"gpt-6-luna","runId":"fd4d720c-6968-4e44-8861-ede438b67a11"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-10T11:17:10.360Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Rosenblat, Sruly","url":null,"kind":"unknown"},{"name":"O'Reilly, Tim","url":null,"kind":"unknown"},{"name":"Strauss, Ilan","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Rosenblat, Sruly","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Rosenblat, Sruly","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2505.00020","locator":"head/meta[name=author|citation_author]"},"assertionId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T11:48:15.076Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721","sourceUrl":"https://arxiv.org/abs/2505.00020","contentSha256":"ab5a77fff461f17738e38541cd7803ebf7c1f3a1ec05163108ef2c0b06f9c3e5","textHash":"a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb","observedAt":"2026-11-10T11:48:15.076Z"}},{"actor":{"name":"O'Reilly, Tim","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"O'Reilly, Tim","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2505.00020","locator":"head/meta[name=author|citation_author]"},"assertionId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T11:48:15.076Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721","sourceUrl":"https://arxiv.org/abs/2505.00020","contentSha256":"ab5a77fff461f17738e38541cd7803ebf7c1f3a1ec05163108ef2c0b06f9c3e5","textHash":"a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb","observedAt":"2026-11-10T11:48:15.076Z"}},{"actor":{"name":"Strauss, Ilan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Strauss, Ilan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2505.00020","locator":"head/meta[name=author|citation_author]"},"assertionId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T11:48:15.076Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721","sourceUrl":"https://arxiv.org/abs/2505.00020","contentSha256":"ab5a77fff461f17738e38541cd7803ebf7c1f3a1ec05163108ef2c0b06f9c3e5","textHash":"a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb","observedAt":"2026-11-10T11:48:15.076Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2505.00020","snapshotId":"1d0801cf9c335aac81de4ef25dcb0c6d16e07cbe6541664f63a1299da72ca721","contentSha256":"ab5a77fff461f17738e38541cd7803ebf7c1f3a1ec05163108ef2c0b06f9c3e5","textHash":"a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb","observedAt":"2026-11-10T11:48:15.076Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/487b80422c3f5824904b7054967bfec67590cf522159dbb7441e68644744b5cb/source"}}}],"audits":[{"publicationId":"c548b352-561d-4b60-8cdb-89188886709d","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The draft correctly credits Sruly Rosenblat, Tim O'Reilly and Ilan Strauss. It keeps the findings and policy conclusions attributed to the authors rather than presenting them as established fact.","excerpt":"Authors:Sruly Rosenblat, Tim O'Reilly, Ilan Strauss"},{"dimension":"summary","status":"uncertain","explanation":"The body is faithful, but two fidelity issues remain. First, the title says 'signs of non-public books in OpenAI's GPT-4o responses,' which could imply the model reproduces the books. The source reports a membership-inference test showing 'patterns consistent with recognition.' Smallest repair: 'signs GPT-4o recognizes non-public books.' Second, the draft never says 0.5 (50%) is chance level and omits the 0.54/0.56 scores, so a lay reader cannot judge 0.82 against 'near chance.' Smallest repair: add the source's 'above 50% is better than random chance' baseline. The draft could also note that non-public means paywalled.","excerpt":"with an AUROC score of 0.82 (95% bootstrapped CI: 0.60-0.96) -- where above 50% is better than random chance"},{"dimension":"date","status":"pass","explanation":"originalPublished 2025-04-24 matches the v1 submission date. The later revisions (v2 May 2026, v3 Oct 2026) and the 2026 feed date are correctly not used as the original publication date.","excerpt":"[Submitted on 24 Apr 2025 (v1), last revised 30 Oct 2026 (this version, v3)]"},{"dimension":"support","status":"pass","explanation":"Every claim matches its abstract excerpt. The access-violation framing, the policy implications and all three stated caveats are source-stated. The open question follows directly from the authors' own limitations. The added limitation correctly notes that the evidence is only the abstract and does not establish training or a legal violation.","excerpt":"Our main contribution is examining non-public books, which allows us to consider potential access violations and not just training inclusion."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-10T11:17:10.360Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}