# arXiv cs.AI: CuraEval tests whether curated robot demonstrations beat simple baselines [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-curaeval-tests-whether-curated-robot-demonstrations-b6041664) Authors Jiuyi Xu, Meida Chen, Yang Ye and Yangming Shi introduce CuraEval, a framework for comparing ways to select robot demonstrations—recordings of operators performing tasks. They argue that differing selection and evaluation protocols make it hard to tell whether curation beats simple baselines, so CuraEval compares fixed size-matched random subsets and full-data training on mixed-skill robomimic data. In their reported tests, the PSD score largely tracks episode length on square, and selecting the shortest half of demonstrations improves square success over random selection with Diffusion Policy. But results vary with task and evaluation protocol. The adapted geometric-entropy LOO selector performs worse than random selection in reported comparisons, and changing checkpoint reporting, rollout count and evaluation seed jointly reverses the direction of a length-based selection result against full-data training. After Holm correction, no evaluated curated condition significantly outperforms full-data training, though the authors say confidence intervals still permit useful gains. Escape Meridian interpretation: These results make curation a question of careful comparison, not a demonstrated general advantage; the abstract reports specific tasks, methods and conditions, rather than establishing how the findings transfer beyond them. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Oct 28, 2026 (2026-10-28). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T12:41:00.634Z. First observed: 2026-10-30T06:45:28.187Z. Retrieved: 2026-10-31T09:32:19.281Z. - [CuraEval: Evaluating Demonstration Curation on Mixed-Skill Robot Data](https://arxiv.org/abs/2610.38221) ## Limits The evidence provided is the paper’s abstract, which reports results on mixed-skill robomimic data and named tasks; it does not establish how the results transfer beyond the tested conditions. The abstract says checkpoint reporting, rollout count and evaluation seed changed jointly, so it does not isolate which change accounts for the altered comparison. ## Open question Would CuraEval’s comparisons show a significant advantage for any curated condition under other tasks or evaluation protocols? ## Source-stated The authors introduce CuraEval to compare demonstration-selection methods against size-matched random subsets and full-data training. > We introduce \\textbf{CuraEval}, a framework combining mathematical analysis of demonstration scores with controlled comparisons against fixed size-matched random subsets and full-data training - [Supporting source](https://arxiv.org/abs/2610.38221) ## Source-stated On square, the evaluated PSD score produces nearly the same ranking as episode length. > produces nearly the same ranking as episode length on \\textit{square} ($\\rho=0.999$). - [Supporting source](https://arxiv.org/abs/2610.38221) ## Source-stated With Diffusion Policy, selecting the shortest half of demonstrations raises square success compared with random selection in the reported test. > selecting the shortest 50\\% of demonstrations improves \\textit{square} success from 70.7\\% under random selection to 76.5\\%. - [Supporting source](https://arxiv.org/abs/2610.38221) ## Source-stated The adapted geometric-entropy LOO selector performs worse than random selection in the reported comparisons on can and square. > reduces success at 25\\% retention from 94.7\\% to 74.0\\% on \\textit{can} and achieves 41.7\\% on \\textit{square}, below the mean success rates of all three independently sampled random subsets (59.8--62.5\\%). - [Supporting source](https://arxiv.org/abs/2610.38221) ## Source-stated Changing checkpoint reporting, rollout count and evaluation seed jointly changes the reported comparison between length-based selection and full-data training on square. > improves success over full-data training from 76.7\\% to 86.7\\% under the original protocol, but lowers it from 80.5\\% to 79.5\\% after checkpoint reporting, rollout count, and evaluation seed change jointly. - [Supporting source](https://arxiv.org/abs/2610.38221) ## Source-stated The authors report that no evaluated curated condition significantly outperforms full-data training after Holm correction, while confidence intervals still permit useful gains. > No evaluated curated condition significantly outperforms full-data training after Holm correction, although confidence intervals still permit useful gains. - [Supporting source](https://arxiv.org/abs/2610.38221) ## Source and contributor context Xu, Jiuyi, Chen, Meida, Ye, Yang, Shi, Yangming · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Confidence interval A range of values consistent with the data at a stated level of confidence. - [Definition reference](https://en.wikipedia.org/wiki/Confidence_interval) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 3ae0d1e8-d852-4e5a-86cb-e6c70689ff93. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T12:41:00.634Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 2de6b130e1bcad1be90ef80e177e84c7cb0901bee367dc50d0eaf590a2f8fe7f. Source text hash: ecb3ca3c9cefcfa94a0ebac8dc1abcbdbec0071f7d5abba0e5a668cd095eb52a. ## Revision and source-check history ### Version 1 — 2026-11-11T12:41:00.634Z Authors Jiuyi Xu, Meida Chen, Yang Ye and Yangming Shi introduce CuraEval, a framework for comparing ways to select robot demonstrations—recordings of operators performing tasks. They argue that differing selection and evaluation protocols make it hard to tell whether curation beats simple baselines, so CuraEval compares fixed size-matched random subsets and full-data training on mixed-skill robomimic data. In their reported tests, the PSD score largely tracks episode length on square, and selecting the shortest half of demonstrations improves square success over random selection with Diffusion Policy. But results vary with task and evaluation protocol. The adapted geometric-entropy LOO selector performs worse than random selection in reported comparisons, and changing checkpoint reporting, rollout count and evaluation seed jointly reverses the direction of a length-based selection result against full-data training. After Holm correction, no evaluated curated condition significantly outperforms full-data training, though the authors say confidence intervals still permit useful gains. Escape Meridian interpretation: These results make curation a question of careful comparison, not a demonstrated general advantage; the abstract reports specific tasks, methods and conditions, rather than establishing how the findings transfer beyond them. ## Original source arXiv cs.AI · Article Original source date: Oct 28, 2026 (2026-10-28). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-11T12:41:00.634Z. First observed: 2026-10-30T06:45:28.187Z. Retrieved: 2026-10-31T09:32:19.281Z. - [CuraEval: Evaluating Demonstration Curation on Mixed-Skill Robot Data](https://arxiv.org/abs/2610.38221) ## Limits The evidence provided is the paper’s abstract, which reports results on mixed-skill robomimic data and named tasks; it does not establish how the results transfer beyond the tested conditions. The abstract says checkpoint reporting, rollout count and evaluation seed changed jointly, so it does not isolate which change accounts for the altered comparison. ## Open question Would CuraEval’s comparisons show a significant advantage for any curated condition under other tasks or evaluation protocols? ## Source-stated The authors introduce CuraEval to compare demonstration-selection methods against size-matched random subsets and full-data training. > We introduce \\textbf{CuraEval}, a framework combining mathematical analysis of demonstration scores with controlled comparisons against fixed size-matched random subsets and full-data training - [Supporting source](https://arxiv.org/abs/2610.38221) ## Source-stated On square, the evaluated PSD score produces nearly the same ranking as episode length. > produces nearly the same ranking as episode length on \\textit{square} ($\\rho=0.999$). - [Supporting source](https://arxiv.org/abs/2610.38221) ## Source-stated With Diffusion Policy, selecting the shortest half of demonstrations raises square success compared with random selection in the reported test. > selecting the shortest 50\\% of demonstrations improves \\textit{square} success from 70.7\\% under random selection to 76.5\\%. - [Supporting source](https://arxiv.org/abs/2610.38221) ## Source-stated The adapted geometric-entropy LOO selector performs worse than random selection in the reported comparisons on can and square. > reduces success at 25\\% retention from 94.7\\% to 74.0\\% on \\textit{can} and achieves 41.7\\% on \\textit{square}, below the mean success rates of all three independently sampled random subsets (59.8--62.5\\%). - [Supporting source](https://arxiv.org/abs/2610.38221) ## Source-stated Changing checkpoint reporting, rollout count and evaluation seed jointly changes the reported comparison between length-based selection and full-data training on square. > improves success over full-data training from 76.7\\% to 86.7\\% under the original protocol, but lowers it from 80.5\\% to 79.5\\% after checkpoint reporting, rollout count, and evaluation seed change jointly. - [Supporting source](https://arxiv.org/abs/2610.38221) ## Source-stated The authors report that no evaluated curated condition significantly outperforms full-data training after Holm correction, while confidence intervals still permit useful gains. > No evaluated curated condition significantly outperforms full-data training after Holm correction, although confidence intervals still permit useful gains. - [Supporting source](https://arxiv.org/abs/2610.38221) ## Source and contributor context Xu, Jiuyi, Chen, Meida, Ye, Yang, Shi, Yangming · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Confidence interval A range of values consistent with the data at a stated level of confidence. - [Definition reference](https://en.wikipedia.org/wiki/Confidence_interval) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 3ae0d1e8-d852-4e5a-86cb-e6c70689ff93. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T12:41:00.634Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 2de6b130e1bcad1be90ef80e177e84c7cb0901bee367dc50d0eaf590a2f8fe7f. Source text hash: ecb3ca3c9cefcfa94a0ebac8dc1abcbdbec0071f7d5abba0e5a668cd095eb52a. ### uncertain — 2026-11-11T12:41:00.634Z — anthropic/claude-opus-5.5 Publication: 3ae0d1e8-d852-4e5a-86cb-e6c70689ff93 attribution — pass: The four named authors match the source, and CuraEval and its findings are attributed to them. The 'arXiv cs.AI' label matches the publisher metadata; the paper is primarily cs.RO and cross-listed to cs.AI. The interpretive closing is clearly labeled as Escape Meridian interpretation. > Authors:Jiuyi Xu, Meida Chen, Yang Ye, Yangming Shi summary — uncertain: The reading conveys the paper's main point: curation methods do not reliably beat simple baselines, and protocol choices change results. However, 'square' and 'can' appear without any cue that they are robomimic benchmark task names, so 'square success' is opaque to general readers. Smallest repair: say 'the square task' and 'the can task' on first use. PSD score, Diffusion Policy, LOO, Holm correction and episode length also lack plain explanation; tooltip entries should be confirmed. One phrase is also garbled: 'CuraEval compares fixed size-matched random subsets' should say it compares selection methods against those subsets. > controlled comparisons against fixed size-matched random subsets and full-data training on mixed-skill robomimic data date — pass: originalPublished 2026-10-28 matches the arXiv v1 submission date. The later feed entry (Oct 31) is correctly not used. > \[Submitted on 28 Oct 2026\] support — uncertain: Each listed claim matches its excerpt, and the Holm-correction caveat and confidence-interval condition are preserved. However, the sentence 'results vary with task and evaluation protocol' goes beyond the abstract. Protocol dependence is shown, but no method is reported to help on one task and hurt on another. The length-based gains are reported only on square, and the geometric-entropy selector underperforms on both tasks. Smallest repair: 'results vary with selection method and evaluation protocol.' The labeled EM interpretation is appropriately hedged and grounded in the Holm result. > No evaluated curated condition significantly outperforms full-data training after Holm correction, although confidence intervals still permit useful gains. media — pass: The selected media is the verified direct arXiv abstract page for this paper, which gives readers the easiest access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/b6041664238d9d075e8a2d15508d1493c8e2a8e4f2b17043cac204b5a008ee8a/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-curaeval-tests-whether-curated-robot-demonstrations-b6041664.json)