Rosenblat, Sruly, O'Reilly, Tim, Strauss, Ilan · Apr 24, 2025
Authors report signs of non-public books in OpenAI's GPT-4o responses
Markdown · JSON · Original source
In “Beyond Public Access in Data,” Sruly Rosenblat, Tim O'Reilly and Ilan Strauss test whether models recognize copyrighted books that were not licensed for training. They use the DE-COP membership inference attack, a method for testing whether a model behaves as if it recognizes examples in a dataset, on 34 legally obtained O'Reilly Media books. The authors report that GPT-4o’s results are consistent with recognition of the non-public books, while GPT-3.5 Turbo and GPT-4o Mini score near chance. The reported AUROC score measures how well a test distinguishes examples; the authors give a score of 0.82 for GPT-4o, with a wide 95% bootstrapped CI, or , of 0.60–0.96. They say their results support disclosure requirements for pre-training data provenance and formal licensing markets for training content. The result matters to the authors because it concerns possible access violations, not only whether copyrighted material appeared in training data. But they describe the findings as preliminary: the book sample is small, post-cutoff controls may be contaminated, and the models differ in size, architecture and training composition. An open question is whether larger studies can separate recognition of non-public content from these limitations.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Apr 24, 2025 (2025-04-24).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T11:17:10.360Z. First observed: 2026-11-02T05:01:03.527Z. Retrieved: 2026-11-03T04:17:50.963Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The authors describe the results as preliminary, based on a small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture and training composition. The evidence provided is the paper's abstract, not its full methods or results; it does not establish that the models were trained on the books or that any legal violation occurred.
Open question
Can larger studies distinguish recognition of non-public books from possible contamination of post-cutoff controls and differences in model size, architecture and training composition?
Source-stated
The authors test OpenAI models for recognition of non-public copyrighted books using DE-COP membership inference attack on 34 legally obtained O'Reilly Media books.
Using a legally obtained dataset of 34 copyrighted O'Reilly Media books (13,962 paragraphs), we apply the DE-COP membership inference attack to test whether OpenAI models show recognition of non-public (pay-walled) content they were not licensed to train on.
Source-stated
The authors report GPT-4o results consistent with recognition of non-public books, with an AUROC score of 0.82 and a 95% bootstrapped CI of 0.60–0.96.
Our results suggest that GPT-4o, OpenAI's more recent and capable model in our sample, exhibits patterns consistent with recognition of non-public books, with an AUROC score of 0.82 (95% bootstrapped CI: 0.60-0.96)
Source-stated
The authors say GPT-3.5 Turbo and GPT-4o Mini score near chance recognition on non-public data.
GPT-3.5 Turbo (trained two years prior) and the smaller GPT-4o Mini both score near chance recognition on non-public data (0.54 and 0.56, respectively).
Source-stated
The authors say their findings support disclosure requirements for pre-training data provenance and formal licensing markets for training content.
These findings support disclosure requirements for pre-training data provenance and formal licensing markets for training content.
Source-stated
The authors call the results preliminary, citing sample size, possible control contamination and differences across models.
Results remain preliminary given the small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture, and training composition.
Source and contributor context
Rosenblat, Sruly, O'Reilly, Tim, Strauss, Ilan · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Pre-training
A model's initial training on a large dataset, before further adaptation for particular uses.
Confidence interval
A range of values consistent with the data at a stated level of confidence.
OpenAI
OpenAI develops AI models and products and publishes research, release announcements and evaluations.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: c548b352-561d-4b60-8cdb-89188886709d.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T11:17:10.360Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 039c72213927620573d22d3673c818acf5e328106d9169e6e46cfb41263110aa. Source text hash: a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb.
Revision and source-check history
Version 1 · 2026-11-10T11:17:10.360Z
Authors report signs of non-public books in OpenAI's GPT-4o responses
In “Beyond Public Access in Data,” Sruly Rosenblat, Tim O'Reilly and Ilan Strauss test whether models recognize copyrighted books that were not licensed for training. They use the DE-COP membership inference attack, a method for testing whether a model behaves as if it recognizes examples in a dataset, on 34 legally obtained O'Reilly Media books. The authors report that GPT-4o’s results are consistent with recognition of the non-public books, while GPT-3.5 Turbo and GPT-4o Mini score near chance. The reported AUROC score measures how well a test distinguishes examples; the authors give a score of 0.82 for GPT-4o, with a wide 95% bootstrapped CI, or , of 0.60–0.96. They say their results support disclosure requirements for pre-training data provenance and formal licensing markets for training content. The result matters to the authors because it concerns possible access violations, not only whether copyrighted material appeared in training data. But they describe the findings as preliminary: the book sample is small, post-cutoff controls may be contaminated, and the models differ in size, architecture and training composition. An open question is whether larger studies can separate recognition of non-public content from these limitations.
Original source
cs.AI · Article
Source details
Original source date: Apr 24, 2025 (2025-04-24).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T11:17:10.360Z. First observed: 2026-11-02T05:01:03.527Z. Retrieved: 2026-11-03T04:17:50.963Z.
Limits
The authors describe the results as preliminary, based on a small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture and training composition. The evidence provided is the paper's abstract, not its full methods or results; it does not establish that the models were trained on the books or that any legal violation occurred.
Open question
Can larger studies distinguish recognition of non-public books from possible contamination of post-cutoff controls and differences in model size, architecture and training composition?
Source-stated
The authors test OpenAI models for recognition of non-public copyrighted books using DE-COP membership inference attack on 34 legally obtained O'Reilly Media books.
Using a legally obtained dataset of 34 copyrighted O'Reilly Media books (13,962 paragraphs), we apply the DE-COP membership inference attack to test whether OpenAI models show recognition of non-public (pay-walled) content they were not licensed to train on.
Source-stated
The authors report GPT-4o results consistent with recognition of non-public books, with an AUROC score of 0.82 and a 95% bootstrapped CI of 0.60–0.96.
Our results suggest that GPT-4o, OpenAI's more recent and capable model in our sample, exhibits patterns consistent with recognition of non-public books, with an AUROC score of 0.82 (95% bootstrapped CI: 0.60-0.96)
Source-stated
The authors say GPT-3.5 Turbo and GPT-4o Mini score near chance recognition on non-public data.
GPT-3.5 Turbo (trained two years prior) and the smaller GPT-4o Mini both score near chance recognition on non-public data (0.54 and 0.56, respectively).
Source-stated
The authors say their findings support disclosure requirements for pre-training data provenance and formal licensing markets for training content.
These findings support disclosure requirements for pre-training data provenance and formal licensing markets for training content.
Source-stated
The authors call the results preliminary, citing sample size, possible control contamination and differences across models.
Results remain preliminary given the small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture, and training composition.
Source and contributor context
Rosenblat, Sruly, O'Reilly, Tim, Strauss, Ilan · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Pre-training
A model's initial training on a large dataset, before further adaptation for particular uses.
Confidence interval
A range of values consistent with the data at a stated level of confidence.
OpenAI
OpenAI develops AI models and products and publishes research, release announcements and evaluations.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: c548b352-561d-4b60-8cdb-89188886709d.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T11:17:10.360Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 039c72213927620573d22d3673c818acf5e328106d9169e6e46cfb41263110aa. Source text hash: a0e8f77c5928906d607079aabd30c0dfb739e85239ab963c6d1156e8136e23cb.
uncertain · 2026-11-10T11:17:10.360Z · anthropic/claude-opus-5.5
Publication: c548b352-561d-4b60-8cdb-89188886709d
attribution — pass: The draft correctly credits Sruly Rosenblat, Tim O'Reilly and Ilan Strauss. It keeps the findings and policy conclusions attributed to the authors rather than presenting them as established fact.
Authors:Sruly Rosenblat, Tim O'Reilly, Ilan Strauss
summary — uncertain: The body is faithful, but two fidelity issues remain. First, the title says 'signs of non-public books in OpenAI's GPT-4o responses,' which could imply the model reproduces the books. The source reports a membership-inference test showing 'patterns consistent with recognition.' Smallest repair: 'signs GPT-4o recognizes non-public books.' Second, the draft never says 0.5 (50%) is chance level and omits the 0.54/0.56 scores, so a lay reader cannot judge 0.82 against 'near chance.' Smallest repair: add the source's 'above 50% is better than random chance' baseline. The draft could also note that non-public means paywalled.
with an AUROC score of 0.82 (95% bootstrapped CI: 0.60-0.96) -- where above 50% is better than random chance
date — pass: originalPublished 2025-04-24 matches the v1 submission date. The later revisions (v2 May 2026, v3 Oct 2026) and the 2026 feed date are correctly not used as the original publication date.
[Submitted on 24 Apr 2025 (v1), last revised 30 Oct 2026 (this version, v3)]
support — pass: Every claim matches its abstract excerpt. The access-violation framing, the policy implications and all three stated caveats are source-stated. The open question follows directly from the authors' own limitations. The added limitation correctly notes that the evidence is only the abstract and does not establish training or a legal violation.
Our main contribution is examining non-public books, which allows us to consider potential access violations and not just training inclusion.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceA model's initial training on a large dataset, before further adaptation for particular uses.
Definition referenceOpenAI develops AI models and products and publishes research, release announcements and evaluations.
Explore OpenAIA range of values consistent with the data at a stated level of confidence.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceA model's initial training on a large dataset, before further adaptation for particular uses.
Definition referenceOpenAI develops AI models and products and publishes research, release announcements and evaluations.
Explore OpenAIA range of values consistent with the data at a stated level of confidence.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv