{"id":"028c6def544e6e515672273acf1ab372bce2ae9398d757d5cff578bffb3a8889","title":"arXiv cs.AI authors introduce Seq@K for evaluating language models with feedback","argument":"The authors introduce Seq@K, an evaluation protocol for language models that gives a model feedback after each failed attempt. It measures whether the model solves a problem within a set number of sequential attempts. They argue that common measures such as Pass@1 and Pass@K do not capture how models use feedback to improve their answers.\n\nThe authors report that Seq@K can produce different model rankings from Pass@K. They also report that it can change which model configurations appear most efficient. The paper analyzes how models respond to feedback, including whether they address issues raised or hold onto their original answers. For practitioners choosing models for systems that refine outputs in response to feedback, the paper suggests that the evaluation method and the feedback itself may affect which models seem suitable. The abstract does not establish how well these findings predict performance in deployed systems.\n\nAn open question is whether the reported differences persist across other tasks and real-world feedback loops.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence is the paper's abstract, not its full methods or results; it does not provide benchmark names, detailed measurements, or evidence about performance in deployed systems. The findings are reported for the benchmarks and models evaluated in the paper; the supplied abstract does not establish whether they generalize to other tasks or feedback loops.","speaker":{"name":"Yen, Thomson, Yang, Gilbert, Ye, Naimeng, Chen, Minghui, Xia, Dong, Shen, Tianyi, Jeon, Yechan, Chen, Yaqin, Mittal, Daksh, Wu, Fangyu, Namkoong, Hongseok","roleAtSource":"Article author"},"source":{"title":"Seq@K: Evaluating Language Models Under Sequential Feedback","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.37859","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce Seq@K, which evaluates whether a model solves a problem within a set number of sequential attempts, receiving feedback after each failure.","excerpt":"We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure.","url":"https://arxiv.org/abs/2610.37859","relationship":"Source-stated"},{"text":"The authors say common measures such as Pass@1 or Pass@K do not capture how models use feedback on prior attempts.","excerpt":"the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts.","url":"https://arxiv.org/abs/2610.37859","relationship":"Source-stated"},{"text":"Across the evaluated benchmarks and models, the authors report that Seq@K often yields different model rankings and changes the relative gains from choosing an optimal model configuration.","excerpt":"under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K.","url":"https://arxiv.org/abs/2610.37859","relationship":"Source-stated"},{"text":"The authors say the way feedback is designed matters when evaluating iterative agentic systems.","excerpt":"showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems.","url":"https://arxiv.org/abs/2610.37859","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.37859","checkedAt":"2026-10-30T11:10:34.688Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Do the model-ranking differences reported with Seq@K persist on tasks and feedback loops beyond the benchmarks evaluated in the paper?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI authors introduce Seq@K for evaluating language models with feedback","summary":"The authors introduce Seq@K, an evaluation protocol for language models that gives a model feedback after each failed attempt. It measures whether the model solves a problem within a set number of sequential attempts. They argue that common measures such as Pass@1 and Pass@K do not capture how models use feedback to improve their answers.\n\nThe authors report that Seq@K can produce different model rankings from Pass@K. They also report that it can change which model configurations appear most efficient. The paper analyzes how models respond to feedback, including whether they address issues raised or hold onto their original answers. For practitioners choosing models for systems that refine outputs in response to feedback, the paper suggests that the evaluation method and the feedback itself may affect which models seem suitable. The abstract does not establish how well these findings predict performance in deployed systems.\n\nAn open question is whether the reported differences persist across other tasks and real-world feedback loops.","question":"Do the model-ranking differences reported with Seq@K persist on tasks and feedback loops beyond the benchmarks evaluated in the paper?","claims":[{"text":"The authors introduce Seq@K, which evaluates whether a model solves a problem within a set number of sequential attempts, receiving feedback after each failure.","excerpt":"We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure.","support":"source-stated"},{"text":"The authors say common measures such as Pass@1 or Pass@K do not capture how models use feedback on prior attempts.","excerpt":"the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts.","support":"source-stated"},{"text":"Across the evaluated benchmarks and models, the authors report that Seq@K often yields different model rankings and changes the relative gains from choosing an optimal model configuration.","excerpt":"under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K.","support":"source-stated"},{"text":"The authors say the way feedback is designed matters when evaluating iterative agentic systems.","excerpt":"showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems.","support":"source-stated"}],"limitations":["The supplied evidence is the paper's abstract, not its full methods or results; it does not provide benchmark names, detailed measurements, or evidence about performance in deployed systems.","The findings are reported for the benchmarks and models evaluated in the paper; the supplied abstract does not establish whether they generalize to other tasks or feedback loops."],"mediaId":"f644284bc0118e4dc798bc6ff567a5cc751f9fd1939a3a4bc6ecde8a6bd58e9d","correctionReason":null,"id":"028c6def544e6e515672273acf1ab372bce2ae9398d757d5cff578bffb3a8889","publicationId":"444b0e4b-3f60-440f-8aec-1c76aceb9b2b","version":1,"sourceUrl":"https://arxiv.org/abs/2610.37859","sourceTitle":"Seq@K: Evaluating Language Models Under Sequential Feedback","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-30T06:45:32.708Z","createdAt":"2026-10-30T09:02:18.349Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"f644284bc0118e4dc798bc6ff567a5cc751f9fd1939a3a4bc6ecde8a6bd58e9d","kind":"article","url":"https://arxiv.org/abs/2610.37859","sourceUrl":"https://arxiv.org/abs/2610.37859","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T09:01:38.629Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.37859","checkedAt":"2026-10-30T11:10:34.688Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"f644284bc0118e4dc798bc6ff567a5cc751f9fd1939a3a4bc6ecde8a6bd58e9d","confidence":0.8099999999999999,"provider":"typesafe","model":"jev-1.13.0","inputHash":"042b70c9c9c726b854e8c2322c26260f534b02810518845e047b344e5c21d429","decidedAt":"2026-10-30T09:01:39.366Z"},"provenance":{"textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observationId":"c66189c63462c0123f9f685e718aaf298cef322f0162e7fefaa1a7c560d32f5b","retrievedAt":"2026-10-30T09:01:38.543Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"c7a2faf7-6f93-4671-8cba-b8d46ae52bcc"},"reviewer":{"model":"gpt-6-luna","runId":"4836a145-d5ca-4a1d-8c6b-d8891cea687d"}},"audit":{"status":"passed","reviewedAt":"2026-10-30T09:02:18.349Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Yen, Thomson","url":null,"kind":"unknown"},{"name":"Yang, Gilbert","url":null,"kind":"unknown"},{"name":"Ye, Naimeng","url":null,"kind":"unknown"},{"name":"Chen, Minghui","url":null,"kind":"unknown"},{"name":"Xia, Dong","url":null,"kind":"unknown"},{"name":"Shen, Tianyi","url":null,"kind":"unknown"},{"name":"Jeon, Yechan","url":null,"kind":"unknown"},{"name":"Chen, Yaqin","url":null,"kind":"unknown"},{"name":"Mittal, Daksh","url":null,"kind":"unknown"},{"name":"Wu, Fangyu","url":null,"kind":"unknown"},{"name":"Namkoong, Hongseok","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Yen, Thomson","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yen, Thomson","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Yang, Gilbert","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yang, Gilbert","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Ye, Naimeng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ye, Naimeng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Chen, Minghui","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chen, Minghui","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Xia, Dong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xia, Dong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Shen, Tianyi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shen, Tianyi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Jeon, Yechan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jeon, Yechan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Chen, Yaqin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chen, Yaqin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Mittal, Daksh","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mittal, Daksh","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Wu, Fangyu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Fangyu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:9","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Namkoong, Hongseok","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Namkoong, Hongseok","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:10","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}}],"participationCoverage":{"total":11,"returned":11,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.37859","snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z","basis":"publisher-page-metadata","sourceLinks":49,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/028c6def544e6e515672273acf1ab372bce2ae9398d757d5cff578bffb3a8889/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-authors-introduce-seq-k-for-evaluating-language-028c6def","formats":{"html":"/readings/arxiv-cs-ai-authors-introduce-seq-k-for-evaluating-language-028c6def","markdown":"/readings/arxiv-cs-ai-authors-introduce-seq-k-for-evaluating-language-028c6def.md","json":"/readings/arxiv-cs-ai-authors-introduce-seq-k-for-evaluating-language-028c6def.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"028c6def544e6e515672273acf1ab372bce2ae9398d757d5cff578bffb3a8889","revision":"444b0e4b-3f60-440f-8aec-1c76aceb9b2b","revisionKind":"publication-id","title":"arXiv cs.AI authors introduce Seq@K for evaluating language models with feedback","url":"https://escapemeridian.com/readings/arxiv-cs-ai-authors-introduce-seq-k-for-evaluating-language-028c6def"},"source":{"type":"source","id":"https://arxiv.org/abs/2610.37859","revision":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","revisionKind":"sha256-source-text","title":"Seq@K: Evaluating Language Models Under Sequential Feedback","url":"https://arxiv.org/abs/2610.37859"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"028c6def544e6e515672273acf1ab372bce2ae9398d757d5cff578bffb3a8889","title":"arXiv cs.AI authors introduce Seq@K for evaluating language models with feedback","argument":"The authors introduce Seq@K, an evaluation protocol for language models that gives a model feedback after each failed attempt. It measures whether the model solves a problem within a set number of sequential attempts. They argue that common measures such as Pass@1 and Pass@K do not capture how models use feedback to improve their answers.\n\nThe authors report that Seq@K can produce different model rankings from Pass@K. They also report that it can change which model configurations appear most efficient. The paper analyzes how models respond to feedback, including whether they address issues raised or hold onto their original answers. For practitioners choosing models for systems that refine outputs in response to feedback, the paper suggests that the evaluation method and the feedback itself may affect which models seem suitable. The abstract does not establish how well these findings predict performance in deployed systems.\n\nAn open question is whether the reported differences persist across other tasks and real-world feedback loops.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence is the paper's abstract, not its full methods or results; it does not provide benchmark names, detailed measurements, or evidence about performance in deployed systems. The findings are reported for the benchmarks and models evaluated in the paper; the supplied abstract does not establish whether they generalize to other tasks or feedback loops.","speaker":{"name":"Yen, Thomson, Yang, Gilbert, Ye, Naimeng, Chen, Minghui, Xia, Dong, Shen, Tianyi, Jeon, Yechan, Chen, Yaqin, Mittal, Daksh, Wu, Fangyu, Namkoong, Hongseok","roleAtSource":"Article author"},"source":{"title":"Seq@K: Evaluating Language Models Under Sequential Feedback","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2610.37859","publishedAt":"2026-10-27","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce Seq@K, which evaluates whether a model solves a problem within a set number of sequential attempts, receiving feedback after each failure.","excerpt":"We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure.","url":"https://arxiv.org/abs/2610.37859","relationship":"Source-stated"},{"text":"The authors say common measures such as Pass@1 or Pass@K do not capture how models use feedback on prior attempts.","excerpt":"the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts.","url":"https://arxiv.org/abs/2610.37859","relationship":"Source-stated"},{"text":"Across the evaluated benchmarks and models, the authors report that Seq@K often yields different model rankings and changes the relative gains from choosing an optimal model configuration.","excerpt":"under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K.","url":"https://arxiv.org/abs/2610.37859","relationship":"Source-stated"},{"text":"The authors say the way feedback is designed matters when evaluating iterative agentic systems.","excerpt":"showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems.","url":"https://arxiv.org/abs/2610.37859","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.37859","checkedAt":"2026-10-30T11:10:34.688Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Do the model-ranking differences reported with Seq@K persist on tasks and feedback loops beyond the benchmarks evaluated in the paper?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI authors introduce Seq@K for evaluating language models with feedback","summary":"The authors introduce Seq@K, an evaluation protocol for language models that gives a model feedback after each failed attempt. It measures whether the model solves a problem within a set number of sequential attempts. They argue that common measures such as Pass@1 and Pass@K do not capture how models use feedback to improve their answers.\n\nThe authors report that Seq@K can produce different model rankings from Pass@K. They also report that it can change which model configurations appear most efficient. The paper analyzes how models respond to feedback, including whether they address issues raised or hold onto their original answers. For practitioners choosing models for systems that refine outputs in response to feedback, the paper suggests that the evaluation method and the feedback itself may affect which models seem suitable. The abstract does not establish how well these findings predict performance in deployed systems.\n\nAn open question is whether the reported differences persist across other tasks and real-world feedback loops.","question":"Do the model-ranking differences reported with Seq@K persist on tasks and feedback loops beyond the benchmarks evaluated in the paper?","claims":[{"text":"The authors introduce Seq@K, which evaluates whether a model solves a problem within a set number of sequential attempts, receiving feedback after each failure.","excerpt":"We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure.","support":"source-stated"},{"text":"The authors say common measures such as Pass@1 or Pass@K do not capture how models use feedback on prior attempts.","excerpt":"the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts.","support":"source-stated"},{"text":"Across the evaluated benchmarks and models, the authors report that Seq@K often yields different model rankings and changes the relative gains from choosing an optimal model configuration.","excerpt":"under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K.","support":"source-stated"},{"text":"The authors say the way feedback is designed matters when evaluating iterative agentic systems.","excerpt":"showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems.","support":"source-stated"}],"limitations":["The supplied evidence is the paper's abstract, not its full methods or results; it does not provide benchmark names, detailed measurements, or evidence about performance in deployed systems.","The findings are reported for the benchmarks and models evaluated in the paper; the supplied abstract does not establish whether they generalize to other tasks or feedback loops."],"mediaId":"f644284bc0118e4dc798bc6ff567a5cc751f9fd1939a3a4bc6ecde8a6bd58e9d","correctionReason":null,"id":"028c6def544e6e515672273acf1ab372bce2ae9398d757d5cff578bffb3a8889","publicationId":"444b0e4b-3f60-440f-8aec-1c76aceb9b2b","version":1,"sourceUrl":"https://arxiv.org/abs/2610.37859","sourceTitle":"Seq@K: Evaluating Language Models Under Sequential Feedback","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-10-27","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-10-30T06:45:32.708Z","createdAt":"2026-10-30T09:02:18.349Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"f644284bc0118e4dc798bc6ff567a5cc751f9fd1939a3a4bc6ecde8a6bd58e9d","kind":"article","url":"https://arxiv.org/abs/2610.37859","sourceUrl":"https://arxiv.org/abs/2610.37859","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-30T09:01:38.629Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2610.37859","checkedAt":"2026-10-30T11:10:34.688Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"f644284bc0118e4dc798bc6ff567a5cc751f9fd1939a3a4bc6ecde8a6bd58e9d","confidence":0.8099999999999999,"provider":"typesafe","model":"jev-1.13.0","inputHash":"042b70c9c9c726b854e8c2322c26260f534b02810518845e047b344e5c21d429","decidedAt":"2026-10-30T09:01:39.366Z"},"provenance":{"textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observationId":"c66189c63462c0123f9f685e718aaf298cef322f0162e7fefaa1a7c560d32f5b","retrievedAt":"2026-10-30T09:01:38.543Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"c7a2faf7-6f93-4671-8cba-b8d46ae52bcc"},"reviewer":{"model":"gpt-6-luna","runId":"4836a145-d5ca-4a1d-8c6b-d8891cea687d"}},"audit":{"status":"passed","reviewedAt":"2026-10-30T09:02:18.349Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Yen, Thomson","url":null,"kind":"unknown"},{"name":"Yang, Gilbert","url":null,"kind":"unknown"},{"name":"Ye, Naimeng","url":null,"kind":"unknown"},{"name":"Chen, Minghui","url":null,"kind":"unknown"},{"name":"Xia, Dong","url":null,"kind":"unknown"},{"name":"Shen, Tianyi","url":null,"kind":"unknown"},{"name":"Jeon, Yechan","url":null,"kind":"unknown"},{"name":"Chen, Yaqin","url":null,"kind":"unknown"},{"name":"Mittal, Daksh","url":null,"kind":"unknown"},{"name":"Wu, Fangyu","url":null,"kind":"unknown"},{"name":"Namkoong, Hongseok","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Yen, Thomson","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yen, Thomson","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Yang, Gilbert","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yang, Gilbert","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Ye, Naimeng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ye, Naimeng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Chen, Minghui","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chen, Minghui","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Xia, Dong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Xia, Dong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Shen, Tianyi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Shen, Tianyi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Jeon, Yechan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jeon, Yechan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Chen, Yaqin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chen, Yaqin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Mittal, Daksh","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mittal, Daksh","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Wu, Fangyu","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Fangyu","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:9","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}},{"actor":{"name":"Namkoong, Hongseok","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Namkoong, Hongseok","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2610.37859","locator":"head/meta[name=author|citation_author]"},"assertionId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d:participation:10","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-30T11:10:34.688Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","sourceUrl":"https://arxiv.org/abs/2610.37859","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z"}}],"participationCoverage":{"total":11,"returned":11,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2610.37859","snapshotId":"dd4169e989a275bbb796b2cdb85c23fb87ecf177dbd73894e60bbe9dcd257f7d","contentSha256":"31a9e04299caf140fa6bce02302c8895e4b2d2a5d566ec897514c9b7fd9cc00f","textHash":"c441c7b83b019c4c28e3db4e6b07083cb2906922ff2f69e6de7cf45528a38217","observedAt":"2026-10-30T11:10:34.688Z","basis":"publisher-page-metadata","sourceLinks":49,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/028c6def544e6e515672273acf1ab372bce2ae9398d757d5cff578bffb3a8889/source"}}}],"audits":[{"publicationId":"444b0e4b-3f60-440f-8aec-1c76aceb9b2b","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The draft attributes the protocol and its findings to the paper's authors, as posted on arXiv cs.AI. It does not name the authors in the title, but it consistently presents the claims as the authors' own rather than as established fact.","excerpt":"Authors:Thomson Yen, Gilbert Yang, Naimeng Ye, Minghui Chen, Dong Xia, Tianyi Shen, Yechan Jeon, Yaqin Chen, Daksh Mittal, Fangyu Wu, Hongseok Namkoong"},{"dimension":"summary","status":"pass","explanation":"The draft explains Seq@K in plain language: the model gets feedback after each failed attempt and is scored on solving the problem within K tries. It conveys the core contrast with common measures and reports the ranking and feedback-behavior findings. It also flags what the abstract does not establish. Pass@1 and Pass@K are not defined, but the reader can still follow the gist that they are common measures that ignore feedback. The question addresses a genuine generalization gap.","excerpt":"We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure."},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-10-27 matches the v1 submission date shown on the arXiv page. It is not the later feed date of Oct 30.","excerpt":"[Submitted on 27 Oct 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim is backed by the abstract. The point about practitioners is framed conditionally and follows the abstract's statement on feedback design. The draft explicitly says the abstract does not establish real-world predictive value, so it does not overstate the findings.","excerpt":"showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified arXiv abstract page for this paper. It is the direct original source.","excerpt":null}],"createdAt":"2026-10-30T09:02:18.349Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}