Li, Mingxuan, Mao, Huanzhi, Zheng, Ruitao, James, Jarod Stanbury, Li, Yunze, Anderson, Natanya, Gonzalez, Joseph E., Dimakis, Alex · Nov 3, 2026
Mingxuan Li and coauthors introduce SlackBench for collaborative projects
Markdown · JSON · Original source
The SlackBench paper introduces a for testing whether can handle collaborative research and engineering work that draws on workplace conversations and changing code repositories. The authors argue that many existing benchmarks miss this complexity by using or isolated tasks. In their account, an agent may need to find relevant channels and direct messages, follow long discussions, inspect repository state, and reconcile stale or conflicting evidence. SlackBench uses privacy-preserving reenactments of real academic research projects, with dialogue, decisions, and code changes tied to GitHub repositories and Google Docs. Its 140 queries test search across sources, abstention when questions cannot be answered or rely on false premises, summarization, and inference from long conversations. The authors say these tasks cannot be reduced to static corpus question answering: agents must identify relevant conversations and combine evidence from messages and project artifacts. Across 33 and agent harnesses, the best system scores around 80%, while most score below 66%. The authors present this gap as substantial headroom for improvement and describe SlackBench as an extensible framework. These benchmark scores indicate performance on the stated tasks; the abstract does not establish how they translate to agents used in everyday work.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 3, 2026 (2026-11-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T00:09:03.449Z. First observed: 2026-11-05T05:03:34.268Z. Retrieved: 2026-11-09T15:01:16.346Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported scores are results; the supplied abstract does not establish how they translate to agents used in everyday collaborative work. The abstract reports overall scores but does not break performance down by the four task types.
Open question
Which of SlackBench’s four task types account for the largest share of the reported performance gap?
Source-stated
The authors introduce SlackBench to evaluate agents on research and engineering workflows that require reasoning across workplace communication and changing repository information.
a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state
Source-stated
SlackBench is built from privacy-preserving reenactments of real academic research projects, connecting dialogue and evolving decisions with code changes and documents.
privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs.
Source-stated
The benchmark contains 140 queries covering search across sources, abstention, summarization, and inference about circumstances in long conversations.
It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations.
Source-stated
The authors report that the strongest system scored around 80% across 33 frontier models and agent harnesses, and that most scored below 66%.
Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%
Source and contributor context
Li, Mingxuan, Mao, Huanzhi, Zheng, Ruitao, James, Jarod Stanbury, Li, Yunze, Anderson, Natanya, Gonzalez, Joseph E., Dimakis, Alex · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Synthetic data
Training or test data generated by software or another model rather than collected from the world.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 4a595803-4ee5-462a-8582-14a3d6353c65.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-10T00:09:03.449Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 57fce15a915471abad8dd7def5e741ef40ebca0dea7237e5ddfab1535fcd8563. Source text hash: f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5.
Revision and source-check history
Version 1 · 2026-11-10T00:09:03.449Z
Mingxuan Li and coauthors introduce SlackBench for collaborative projects
The SlackBench paper introduces a for testing whether can handle collaborative research and engineering work that draws on workplace conversations and changing code repositories. The authors argue that many existing benchmarks miss this complexity by using or isolated tasks. In their account, an agent may need to find relevant channels and direct messages, follow long discussions, inspect repository state, and reconcile stale or conflicting evidence. SlackBench uses privacy-preserving reenactments of real academic research projects, with dialogue, decisions, and code changes tied to GitHub repositories and Google Docs. Its 140 queries test search across sources, abstention when questions cannot be answered or rely on false premises, summarization, and inference from long conversations. The authors say these tasks cannot be reduced to static corpus question answering: agents must identify relevant conversations and combine evidence from messages and project artifacts. Across 33 and agent harnesses, the best system scores around 80%, while most score below 66%. The authors present this gap as substantial headroom for improvement and describe SlackBench as an extensible framework. These benchmark scores indicate performance on the stated tasks; the abstract does not establish how they translate to agents used in everyday work.
Original source
cs.AI · Article
Source details
Original source date: Nov 3, 2026 (2026-11-03).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-10T00:09:03.449Z. First observed: 2026-11-05T05:03:34.268Z. Retrieved: 2026-11-09T15:01:16.346Z.
Limits
The reported scores are results; the supplied abstract does not establish how they translate to agents used in everyday collaborative work. The abstract reports overall scores but does not break performance down by the four task types.
Open question
Which of SlackBench’s four task types account for the largest share of the reported performance gap?
Source-stated
The authors introduce SlackBench to evaluate agents on research and engineering workflows that require reasoning across workplace communication and changing repository information.
a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state
Source-stated
SlackBench is built from privacy-preserving reenactments of real academic research projects, connecting dialogue and evolving decisions with code changes and documents.
privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs.
Source-stated
The benchmark contains 140 queries covering search across sources, abstention, summarization, and inference about circumstances in long conversations.
It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations.
Source-stated
The authors report that the strongest system scored around 80% across 33 frontier models and agent harnesses, and that most scored below 66%.
Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%
Source and contributor context
Li, Mingxuan, Mao, Huanzhi, Zheng, Ruitao, James, Jarod Stanbury, Li, Yunze, Anderson, Natanya, Gonzalez, Joseph E., Dimakis, Alex · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
AI agent
An AI system that chooses and carries out actions toward a goal, often using software tools.
Synthetic data
Training or test data generated by software or another model rather than collected from the world.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 4a595803-4ee5-462a-8582-14a3d6353c65.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-10T00:09:03.449Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 57fce15a915471abad8dd7def5e741ef40ebca0dea7237e5ddfab1535fcd8563. Source text hash: f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5.
passed · 2026-11-10T00:09:03.449Z · anthropic/claude-opus-5.5
Publication: 4a595803-4ee5-462a-8582-14a3d6353c65
attribution — pass: The title credits Mingxuan Li and coauthors, matching the listed authors. The draft frames the benchmark's purpose, critique of existing benchmarks and results as the authors' claims ('The authors argue', 'In their account', 'The authors say').
Authors:Mingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis
summary — pass: The reading leads with what the benchmark is, explains the problem it targets in plain language and reports the results and the authors' framing. It closes with a caveat that the scores do not show everyday reliability. One fidelity nit: 'goal-directed summarization' becomes 'summarization' and 'circumstance inference' is paraphrased. The smallest repair is to keep the exact terms with tooltips. The question follows from the evidence and is not leading.
heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations
date — pass: originalPublished is 2026-11-03, which matches the v1 submission date. The feed date of 2026-11-05 is later and is correctly not used.
[Submitted on 3 Nov 2026]
support — pass: Every claim matches an abstract excerpt: 140 queries, 33 models and harnesses, best around 80%, most below 66%, and the privacy-preserving reenactments. The limitations correctly note there is no breakdown by task type and no evidence of real-world transfer. The draft adds no unsupported benefits or costs.
Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%, highlighting substantial headroom for improvement.
media — pass: The selected media is the verified direct link to the arXiv abstract page of the source itself, which is the most direct access to the original.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceAn AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceTraining or test data generated by software or another model rather than collected from the world.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceAn AI system that chooses and carries out actions toward a goal, often using software tools.
Definition referenceTraining or test data generated by software or another model rather than collected from the world.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference