# arXiv cs.AI paper tests how feedback changes language models’ data science projects [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-paper-tests-how-feedback-changes-language-models-f31194c7) Rohan Alexander’s paper evaluates whether language models can complete an end-to-end data science project and how their work changes with feedback. A “writer” model downloads, cleans and analyzes a dataset, then writes a short paper; a “grader” model evaluates the project, and the writer can revise it over three rounds. Across 147 initial submissions, 107 ran and rendered, while the average score rose from 59 per cent to 76 per cent after feedback. But the scores need careful reading: when a model graded its own work, its score tended to be about 9 percentage points higher than other models’ average, and about half the round-to-round difference was within the grader’s own noise. Alexander also reports that revision opportunities or additional time or tool calls, even without feedback, helped some models considerably but not others. The paper’s central qualification is that the effect depends on which model writes and which grades; weaker models generally did not improve stronger ones. The results describe this project-and-grading setup, not a general measure of data science reliability. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 3, 2026 (2026-11-03). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-09T13:19:41.381Z. First observed: 2026-11-05T05:06:33.695Z. Retrieved: 2026-11-09T13:18:48.621Z. - [Iterated creation, grading, and revision of data science projects by language models](https://arxiv.org/abs/2611.05548) ## Limits The evidence supplied is the paper’s abstract, not its full methods or results; it does not provide the details needed to assess the grading noise or interpret the scores beyond the described setup. The abstract’s results concern the data science project and grading procedure it describes; they do not establish how models perform on other tasks or in real-world use. ## Open question How would the reported gains change across further runs, given that about half the round-to-round difference was within the grader’s own noise? ## Source-stated The paper evaluates language models completing an end-to-end data science project, with a writer creating and revising the project and a grader evaluating it. > A 'writer' model with specific tools creates a complete data science project including downloading, cleaning, and analyzing a dataset, and writing a short paper about it. - [Supporting source](https://arxiv.org/abs/2611.05548) ## Source-stated The experiments used 147 initial submissions; 107 ran and rendered, and the average score rose from 59 per cent to 76 per cent after the final feedback round. > Of the 147 initial submissions, 107 run and render, and the average score increases from 59 per cent to 76 per cent after the final round of feedback. - [Supporting source](https://arxiv.org/abs/2611.05548) ## Source-stated Self-grading tended to produce scores about 9 percentage points above the average given by other models, and about half of the round-to-round difference was within grader noise. > When a model grades its own writing the score tends to be about 9 percentage points higher than the average of what the other models give it. About half of the difference that we see from round to round is within the grader's own noise. - [Supporting source](https://arxiv.org/abs/2611.05548) ## Source-stated The paper reports that revision or additional time or tool calls without feedback helped some models considerably, but not others. > even just having the chance to revise, or having the additional budget, in terms of time or tool calls, without any feedback, results in considerable increases for some of the models but not for others. - [Supporting source](https://arxiv.org/abs/2611.05548) ## Source-stated The reported benefit depends on the models’ roles; weaker models generally did not improve stronger models. > how much such a loop helps depends on which model plays which role, but weaker models generally do not improve stronger models. - [Supporting source](https://arxiv.org/abs/2611.05548) ## Source and contributor context Alexander, Rohan · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## Editorial provenance Version 1. Publication ID: 285480c9-0179-4655-84c7-7c03acdda503. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T13:19:41.381Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: a3eea8f015cbbd6c368f07466ef4eb62b1961ecc874ddac990a8fa84c77ee5b2. Source text hash: 9b547d75f42a16eb37cf71c1d4c1ad1abfdce0628335fd1c7366e83089e21b23. ## Revision and source-check history ### Version 1 — 2026-11-09T13:19:41.381Z Rohan Alexander’s paper evaluates whether language models can complete an end-to-end data science project and how their work changes with feedback. A “writer” model downloads, cleans and analyzes a dataset, then writes a short paper; a “grader” model evaluates the project, and the writer can revise it over three rounds. Across 147 initial submissions, 107 ran and rendered, while the average score rose from 59 per cent to 76 per cent after feedback. But the scores need careful reading: when a model graded its own work, its score tended to be about 9 percentage points higher than other models’ average, and about half the round-to-round difference was within the grader’s own noise. Alexander also reports that revision opportunities or additional time or tool calls, even without feedback, helped some models considerably but not others. The paper’s central qualification is that the effect depends on which model writes and which grades; weaker models generally did not improve stronger ones. The results describe this project-and-grading setup, not a general measure of data science reliability. ## Original source arXiv cs.AI · Article Original source date: Nov 3, 2026 (2026-11-03). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-09T13:19:41.381Z. First observed: 2026-11-05T05:06:33.695Z. Retrieved: 2026-11-09T13:18:48.621Z. - [Iterated creation, grading, and revision of data science projects by language models](https://arxiv.org/abs/2611.05548) ## Limits The evidence supplied is the paper’s abstract, not its full methods or results; it does not provide the details needed to assess the grading noise or interpret the scores beyond the described setup. The abstract’s results concern the data science project and grading procedure it describes; they do not establish how models perform on other tasks or in real-world use. ## Open question How would the reported gains change across further runs, given that about half the round-to-round difference was within the grader’s own noise? ## Source-stated The paper evaluates language models completing an end-to-end data science project, with a writer creating and revising the project and a grader evaluating it. > A 'writer' model with specific tools creates a complete data science project including downloading, cleaning, and analyzing a dataset, and writing a short paper about it. - [Supporting source](https://arxiv.org/abs/2611.05548) ## Source-stated The experiments used 147 initial submissions; 107 ran and rendered, and the average score rose from 59 per cent to 76 per cent after the final feedback round. > Of the 147 initial submissions, 107 run and render, and the average score increases from 59 per cent to 76 per cent after the final round of feedback. - [Supporting source](https://arxiv.org/abs/2611.05548) ## Source-stated Self-grading tended to produce scores about 9 percentage points above the average given by other models, and about half of the round-to-round difference was within grader noise. > When a model grades its own writing the score tends to be about 9 percentage points higher than the average of what the other models give it. About half of the difference that we see from round to round is within the grader's own noise. - [Supporting source](https://arxiv.org/abs/2611.05548) ## Source-stated The paper reports that revision or additional time or tool calls without feedback helped some models considerably, but not others. > even just having the chance to revise, or having the additional budget, in terms of time or tool calls, without any feedback, results in considerable increases for some of the models but not for others. - [Supporting source](https://arxiv.org/abs/2611.05548) ## Source-stated The reported benefit depends on the models’ roles; weaker models generally did not improve stronger models. > how much such a loop helps depends on which model plays which role, but weaker models generally do not improve stronger models. - [Supporting source](https://arxiv.org/abs/2611.05548) ## Source and contributor context Alexander, Rohan · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## Editorial provenance Version 1. Publication ID: 285480c9-0179-4655-84c7-7c03acdda503. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T13:19:41.381Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: a3eea8f015cbbd6c368f07466ef4eb62b1961ecc874ddac990a8fa84c77ee5b2. Source text hash: 9b547d75f42a16eb37cf71c1d4c1ad1abfdce0628335fd1c7366e83089e21b23. ### passed — 2026-11-09T13:19:41.381Z — anthropic/claude-opus-5.5 Publication: 285480c9-0179-4655-84c7-7c03acdda503 attribution — pass: The draft attributes the paper and its findings to Rohan Alexander on arXiv cs.AI, matching the listed author. Results are framed as what the paper reports, not as established fact. > Authors:Rohan Alexander summary — pass: The summary faithfully conveys the writer/grader loop, three rounds, 147 submissions with 107 running, the 59 to 76 per cent rise, the self-grading bias, grader noise, gains without feedback for some models, and the role-dependence finding. It explains the end-to-end project in plain terms. It omits the per-model ranges and the seven models from five developers, which is acceptable. Small optional repairs: add a brief gloss for 'ran and rendered' and 'grader's own noise' (scoring variation by the same grader). Neither gap blocks understanding. The question follows directly from the noise finding. > About half of the difference that we see from round to round is within the grader's own noise. date — pass: originalPublished is 2026-11-03, which matches the v1 submission date. The feed date of 2026-11-05 is a later announcement. > \[Submitted on 3 Nov 2026\] support — pass: Each claim quotes the abstract accurately. The limitation confining the results to the described setup is a fair caution, not an added claim. No benefits, costs or forecasts are invented. > We find that how much such a loop helps depends on which model plays which role, but weaker models generally do not improve stronger models. media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/f31194c74d1c5af586f2c07dda3e27fde94d87f6843f74e82be8cf15f0a2402d/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-paper-tests-how-feedback-changes-language-models-f31194c7.json)