Alexander, Rohan · Nov 3, 2026
arXiv cs.AI paper tests how feedback changes language models’ data science projects
Markdown · JSON · Original source
Rohan Alexander’s paper evaluates whether language models can complete an end-to-end data science project and how their work changes with feedback. A “writer” model downloads, cleans and analyzes a dataset, then writes a short paper; a “grader” model evaluates the project, and the writer can revise it over three rounds. Across 147 initial submissions, 107 ran and rendered, while the average score rose from 59 per cent to 76 per cent after feedback. But the scores need careful reading: when a model graded its own work, its score tended to be about 9 percentage points higher than other models’ average, and about half the round-to-round difference was within the grader’s own noise. Alexander also reports that revision opportunities or additional or tool calls, even without feedback, helped some models considerably but not others. The paper’s central qualification is that the effect depends on which model writes and which grades; weaker models generally did not improve stronger ones. The results describe this project-and-grading setup, not a general measure of data science reliability.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 3, 2026 (2026-11-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T13:19:41.381Z. First observed: 2026-11-05T05:06:33.695Z. Retrieved: 2026-11-09T13:18:48.621Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence supplied is the paper’s abstract, not its full methods or results; it does not provide the details needed to assess the grading noise or interpret the scores beyond the described setup. The abstract’s results concern the data science project and grading procedure it describes; they do not establish how models perform on other tasks or in real-world use.
Open question
How would the reported gains change across further runs, given that about half the round-to-round difference was within the grader’s own noise?
Source-stated
The paper evaluates language models completing an end-to-end data science project, with a writer creating and revising the project and a grader evaluating it.
A 'writer' model with specific tools creates a complete data science project including downloading, cleaning, and analyzing a dataset, and writing a short paper about it.
Source-stated
The experiments used 147 initial submissions; 107 ran and rendered, and the average score rose from 59 per cent to 76 per cent after the final feedback round.
Of the 147 initial submissions, 107 run and render, and the average score increases from 59 per cent to 76 per cent after the final round of feedback.
Source-stated
Self-grading tended to produce scores about 9 percentage points above the average given by other models, and about half of the round-to-round difference was within grader noise.
When a model grades its own writing the score tends to be about 9 percentage points higher than the average of what the other models give it. About half of the difference that we see from round to round is within the grader's own noise.
Source-stated
The paper reports that revision or additional time or tool calls without feedback helped some models considerably, but not others.
even just having the chance to revise, or having the additional budget, in terms of time or tool calls, without any feedback, results in considerable increases for some of the models but not for others.
Source-stated
The reported benefit depends on the models’ roles; weaker models generally did not improve stronger models.
how much such a loop helps depends on which model plays which role, but weaker models generally do not improve stronger models.
Source and contributor context
Alexander, Rohan · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 285480c9-0179-4655-84c7-7c03acdda503.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T13:19:41.381Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: a3eea8f015cbbd6c368f07466ef4eb62b1961ecc874ddac990a8fa84c77ee5b2. Source text hash: 9b547d75f42a16eb37cf71c1d4c1ad1abfdce0628335fd1c7366e83089e21b23.
Revision and source-check history
Version 1 · 2026-11-09T13:19:41.381Z
arXiv cs.AI paper tests how feedback changes language models’ data science projects
Rohan Alexander’s paper evaluates whether language models can complete an end-to-end data science project and how their work changes with feedback. A “writer” model downloads, cleans and analyzes a dataset, then writes a short paper; a “grader” model evaluates the project, and the writer can revise it over three rounds. Across 147 initial submissions, 107 ran and rendered, while the average score rose from 59 per cent to 76 per cent after feedback. But the scores need careful reading: when a model graded its own work, its score tended to be about 9 percentage points higher than other models’ average, and about half the round-to-round difference was within the grader’s own noise. Alexander also reports that revision opportunities or additional or tool calls, even without feedback, helped some models considerably but not others. The paper’s central qualification is that the effect depends on which model writes and which grades; weaker models generally did not improve stronger ones. The results describe this project-and-grading setup, not a general measure of data science reliability.
Original source
cs.AI · Article
Source details
Original source date: Nov 3, 2026 (2026-11-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T13:19:41.381Z. First observed: 2026-11-05T05:06:33.695Z. Retrieved: 2026-11-09T13:18:48.621Z.
Limits
The evidence supplied is the paper’s abstract, not its full methods or results; it does not provide the details needed to assess the grading noise or interpret the scores beyond the described setup. The abstract’s results concern the data science project and grading procedure it describes; they do not establish how models perform on other tasks or in real-world use.
Open question
How would the reported gains change across further runs, given that about half the round-to-round difference was within the grader’s own noise?
Source-stated
The paper evaluates language models completing an end-to-end data science project, with a writer creating and revising the project and a grader evaluating it.
A 'writer' model with specific tools creates a complete data science project including downloading, cleaning, and analyzing a dataset, and writing a short paper about it.
Source-stated
The experiments used 147 initial submissions; 107 ran and rendered, and the average score rose from 59 per cent to 76 per cent after the final feedback round.
Of the 147 initial submissions, 107 run and render, and the average score increases from 59 per cent to 76 per cent after the final round of feedback.
Source-stated
Self-grading tended to produce scores about 9 percentage points above the average given by other models, and about half of the round-to-round difference was within grader noise.
When a model grades its own writing the score tends to be about 9 percentage points higher than the average of what the other models give it. About half of the difference that we see from round to round is within the grader's own noise.
Source-stated
The paper reports that revision or additional time or tool calls without feedback helped some models considerably, but not others.
even just having the chance to revise, or having the additional budget, in terms of time or tool calls, without any feedback, results in considerable increases for some of the models but not for others.
Source-stated
The reported benefit depends on the models’ roles; weaker models generally did not improve stronger models.
how much such a loop helps depends on which model plays which role, but weaker models generally do not improve stronger models.
Source and contributor context
Alexander, Rohan · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 285480c9-0179-4655-84c7-7c03acdda503.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T13:19:41.381Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: a3eea8f015cbbd6c368f07466ef4eb62b1961ecc874ddac990a8fa84c77ee5b2. Source text hash: 9b547d75f42a16eb37cf71c1d4c1ad1abfdce0628335fd1c7366e83089e21b23.
passed · 2026-11-09T13:19:41.381Z · anthropic/claude-opus-5.5
Publication: 285480c9-0179-4655-84c7-7c03acdda503
attribution — pass: The draft attributes the paper and its findings to Rohan Alexander on arXiv cs.AI, matching the listed author. Results are framed as what the paper reports, not as established fact.
Authors:Rohan Alexander
summary — pass: The summary faithfully conveys the writer/grader loop, three rounds, 147 submissions with 107 running, the 59 to 76 per cent rise, the self-grading bias, grader noise, gains without feedback for some models, and the role-dependence finding. It explains the end-to-end project in plain terms. It omits the per-model ranges and the seven models from five developers, which is acceptable. Small optional repairs: add a brief gloss for 'ran and rendered' and 'grader's own noise' (scoring variation by the same grader). Neither gap blocks understanding. The question follows directly from the noise finding.
About half of the difference that we see from round to round is within the grader's own noise.
date — pass: originalPublished is 2026-11-03, which matches the v1 submission date. The feed date of 2026-11-05 is a later announcement.
[Submitted on 3 Nov 2026]
support — pass: Each claim quotes the abstract accurately. The limitation confining the results to the described setup is a fair caution, not an added claim. No benefits, costs or forecasts are invented.
We find that how much such a loop helps depends on which model plays which role, but weaker models generally do not improve stronger models.
media — pass: The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEarXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv