Yen, Thomson, Yang, Gilbert, Ye, Naimeng, Chen, Minghui, Xia, Dong, Shen, Tianyi, Jeon, Yechan, Chen, Yaqin, Mittal, Daksh, Wu, Fangyu, Namkoong, Hongseok · Oct 27, 2026
arXiv cs.AI authors introduce Seq@K for evaluating language models with feedback
Markdown · JSON · Original source
The authors introduce Seq@K, an evaluation protocol for language models that gives a model feedback after each failed attempt. It measures whether the model solves a problem within a set number of sequential attempts. They argue that common measures such as Pass@1 and Pass@K do not capture how models use feedback to improve their answers. The authors report that Seq@K can produce different model rankings from Pass@K. They also report that it can change which model configurations appear most efficient. The paper analyzes how models respond to feedback, including whether they address issues raised or hold onto their original answers. For practitioners choosing models for systems that refine outputs in response to feedback, the paper suggests that the evaluation method and the feedback itself may affect which models seem suitable. The abstract does not establish how well these findings predict performance in deployed systems. An open question is whether the reported differences persist across other tasks and real-world feedback loops.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 27, 2026 (2026-10-27).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-10-30T09:02:18.349Z. First observed: 2026-10-30T06:45:32.708Z. Retrieved: 2026-10-30T09:01:38.543Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The supplied evidence is the paper's abstract, not its full methods or results; it does not provide names, detailed measurements, or evidence about performance in deployed systems. The findings are reported for the benchmarks and models evaluated in the paper; the supplied abstract does not establish whether they generalize to other tasks or feedback loops.
Open question
Do the model-ranking differences reported with Seq@K persist on tasks and feedback loops beyond the evaluated in the paper?
Source-stated
The authors introduce Seq@K, which evaluates whether a model solves a problem within a set number of sequential attempts, receiving feedback after each failure.
We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure.
Source-stated
The authors say common measures such as Pass@1 or Pass@K do not capture how models use feedback on prior attempts.
the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts.
Source-stated
Across the evaluated benchmarks and models, the authors report that Seq@K often yields different model rankings and changes the relative gains from choosing an optimal model configuration.
under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K.
Source-stated
The authors say the way feedback is designed matters when evaluating iterative agentic systems.
showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems.
Source and contributor context
Yen, Thomson, Yang, Gilbert, Ye, Naimeng, Chen, Minghui, Xia, Dong, Shen, Tianyi, Jeon, Yechan, Chen, Yaqin, Mittal, Daksh, Wu, Fangyu, Namkoong, Hongseok · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 444b0e4b-3f60-440f-8aec-1c76aceb9b2b.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-10-30T09:02:18.349Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: c66189c63462c0123f9f685e718aaf298cef322f0162e7fefaa1a7c560d32f5b. Source text hash: c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217.
Revision and source-check history
Version 1 · 2026-10-30T09:02:18.349Z
arXiv cs.AI authors introduce Seq@K for evaluating language models with feedback
The authors introduce Seq@K, an evaluation protocol for language models that gives a model feedback after each failed attempt. It measures whether the model solves a problem within a set number of sequential attempts. They argue that common measures such as Pass@1 and Pass@K do not capture how models use feedback to improve their answers. The authors report that Seq@K can produce different model rankings from Pass@K. They also report that it can change which model configurations appear most efficient. The paper analyzes how models respond to feedback, including whether they address issues raised or hold onto their original answers. For practitioners choosing models for systems that refine outputs in response to feedback, the paper suggests that the evaluation method and the feedback itself may affect which models seem suitable. The abstract does not establish how well these findings predict performance in deployed systems. An open question is whether the reported differences persist across other tasks and real-world feedback loops.
Original source
cs.AI · Article
Source details
Original source date: Oct 27, 2026 (2026-10-27).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-10-30T09:02:18.349Z. First observed: 2026-10-30T06:45:32.708Z. Retrieved: 2026-10-30T09:01:38.543Z.
Limits
The supplied evidence is the paper's abstract, not its full methods or results; it does not provide names, detailed measurements, or evidence about performance in deployed systems. The findings are reported for the benchmarks and models evaluated in the paper; the supplied abstract does not establish whether they generalize to other tasks or feedback loops.
Open question
Do the model-ranking differences reported with Seq@K persist on tasks and feedback loops beyond the evaluated in the paper?
Source-stated
The authors introduce Seq@K, which evaluates whether a model solves a problem within a set number of sequential attempts, receiving feedback after each failure.
We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure.
Source-stated
The authors say common measures such as Pass@1 or Pass@K do not capture how models use feedback on prior attempts.
the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts.
Source-stated
Across the evaluated benchmarks and models, the authors report that Seq@K often yields different model rankings and changes the relative gains from choosing an optimal model configuration.
under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K.
Source-stated
The authors say the way feedback is designed matters when evaluating iterative agentic systems.
showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems.
Source and contributor context
Yen, Thomson, Yang, Gilbert, Ye, Naimeng, Chen, Minghui, Xia, Dong, Shen, Tianyi, Jeon, Yechan, Chen, Yaqin, Mittal, Daksh, Wu, Fangyu, Namkoong, Hongseok · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 444b0e4b-3f60-440f-8aec-1c76aceb9b2b.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-10-30T09:02:18.349Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: c66189c63462c0123f9f685e718aaf298cef322f0162e7fefaa1a7c560d32f5b. Source text hash: c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217.
passed · 2026-10-30T09:02:18.349Z · anthropic/claude-opus-5.5
Publication: 444b0e4b-3f60-440f-8aec-1c76aceb9b2b
attribution — pass: The draft attributes the protocol and its findings to the paper's authors, as posted on arXiv cs.AI. It does not name the authors in the title, but it consistently presents the claims as the authors' own rather than as established fact.
Authors:Thomson Yen, Gilbert Yang, Naimeng Ye, Minghui Chen, Dong Xia, Tianyi Shen, Yechan Jeon, Yaqin Chen, Daksh Mittal, Fangyu Wu, Hongseok Namkoong
summary — pass: The draft explains Seq@K in plain language: the model gets feedback after each failed attempt and is scored on solving the problem within K tries. It conveys the core contrast with common measures and reports the ranking and feedback-behavior findings. It also flags what the abstract does not establish. Pass@1 and Pass@K are not defined, but the reader can still follow the gist that they are common measures that ignore feedback. The question addresses a genuine generalization gap.
We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure.
date — pass: The originalPublished date of 2026-10-27 matches the v1 submission date shown on the arXiv page. It is not the later feed date of Oct 30.
[Submitted on 27 Oct 2026]
support — pass: Each claim is backed by the abstract. The point about practitioners is framed conditionally and follows the abstract's statement on feedback design. The draft explicitly says the abstract does not establish real-world predictive value, so it does not overstate the findings.
showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems.
media — pass: The selected media is the verified arXiv abstract page for this paper. It is the direct original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference