{"id":"b8c5621c88151c7c2d6a5edb67d3bd30ec3f2c6de82fba8380151afb46061ab5","title":"Mingxuan Li and coauthors introduce SlackBench for collaborative projects","argument":"The SlackBench paper introduces a benchmark for testing whether AI agents can handle collaborative research and engineering work that draws on workplace conversations and changing code repositories. The authors argue that many existing benchmarks miss this complexity by using synthetic data or isolated tasks. In their account, an agent may need to find relevant channels and direct messages, follow long discussions, inspect repository state, and reconcile stale or conflicting evidence.\n\nSlackBench uses privacy-preserving reenactments of real academic research projects, with dialogue, decisions, and code changes tied to GitHub repositories and Google Docs. Its 140 queries test search across sources, abstention when questions cannot be answered or rely on false premises, summarization, and inference from long conversations. The authors say these tasks cannot be reduced to static corpus question answering: agents must identify relevant conversations and combine evidence from messages and project artifacts.\n\nAcross 33 frontier models and agent harnesses, the best system scores around 80%, while most score below 66%. The authors present this gap as substantial headroom for improvement and describe SlackBench as an extensible framework. These benchmark scores indicate performance on the stated tasks; the abstract does not establish how they translate to agents used in everyday work.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported scores are benchmark results; the supplied abstract does not establish how they translate to agents used in everyday collaborative work. The abstract reports overall scores but does not break performance down by the four task types.","speaker":{"name":"Li, Mingxuan, Mao, Huanzhi, Zheng, Ruitao, James, Jarod Stanbury, Li, Yunze, Anderson, Natanya, Gonzalez, Joseph E., Dimakis, Alex","roleAtSource":"Article author"},"source":{"title":"SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.05512","publishedAt":"2026-11-03","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce SlackBench to evaluate agents on research and engineering workflows that require reasoning across workplace communication and changing repository information.","excerpt":"a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state","url":"https://arxiv.org/abs/2611.05512","relationship":"Source-stated"},{"text":"SlackBench is built from privacy-preserving reenactments of real academic research projects, connecting dialogue and evolving decisions with code changes and documents.","excerpt":"privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs.","url":"https://arxiv.org/abs/2611.05512","relationship":"Source-stated"},{"text":"The benchmark contains 140 queries covering search across sources, abstention, summarization, and inference about circumstances in long conversations.","excerpt":"It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations.","url":"https://arxiv.org/abs/2611.05512","relationship":"Source-stated"},{"text":"The authors report that the strongest system scored around 80% across 33 frontier models and agent harnesses, and that most scored below 66%.","excerpt":"Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%","url":"https://arxiv.org/abs/2611.05512","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.05512","checkedAt":"2026-11-10T00:40:10.654Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Which of SlackBench’s four task types account for the largest share of the reported performance gap?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Mingxuan Li and coauthors introduce SlackBench for collaborative projects","summary":"The SlackBench paper introduces a benchmark for testing whether AI agents can handle collaborative research and engineering work that draws on workplace conversations and changing code repositories. The authors argue that many existing benchmarks miss this complexity by using synthetic data or isolated tasks. In their account, an agent may need to find relevant channels and direct messages, follow long discussions, inspect repository state, and reconcile stale or conflicting evidence.\n\nSlackBench uses privacy-preserving reenactments of real academic research projects, with dialogue, decisions, and code changes tied to GitHub repositories and Google Docs. Its 140 queries test search across sources, abstention when questions cannot be answered or rely on false premises, summarization, and inference from long conversations. The authors say these tasks cannot be reduced to static corpus question answering: agents must identify relevant conversations and combine evidence from messages and project artifacts.\n\nAcross 33 frontier models and agent harnesses, the best system scores around 80%, while most score below 66%. The authors present this gap as substantial headroom for improvement and describe SlackBench as an extensible framework. These benchmark scores indicate performance on the stated tasks; the abstract does not establish how they translate to agents used in everyday work.","question":"Which of SlackBench’s four task types account for the largest share of the reported performance gap?","claims":[{"text":"The authors introduce SlackBench to evaluate agents on research and engineering workflows that require reasoning across workplace communication and changing repository information.","excerpt":"a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state","support":"source-stated"},{"text":"SlackBench is built from privacy-preserving reenactments of real academic research projects, connecting dialogue and evolving decisions with code changes and documents.","excerpt":"privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs.","support":"source-stated"},{"text":"The benchmark contains 140 queries covering search across sources, abstention, summarization, and inference about circumstances in long conversations.","excerpt":"It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations.","support":"source-stated"},{"text":"The authors report that the strongest system scored around 80% across 33 frontier models and agent harnesses, and that most scored below 66%.","excerpt":"Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%","support":"source-stated"}],"limitations":["The reported scores are benchmark results; the supplied abstract does not establish how they translate to agents used in everyday collaborative work.","The abstract reports overall scores but does not break performance down by the four task types."],"mediaId":"bbb7f69db5debade679b62450c74a40067db910451c8096a1cbf3fcbac92522d","correctionReason":null,"id":"b8c5621c88151c7c2d6a5edb67d3bd30ec3f2c6de82fba8380151afb46061ab5","publicationId":"4a595803-4ee5-462a-8582-14a3d6353c65","version":1,"sourceUrl":"https://arxiv.org/abs/2611.05512","sourceTitle":"SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-03","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:03:34.268Z","createdAt":"2026-11-10T00:09:03.449Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"bbb7f69db5debade679b62450c74a40067db910451c8096a1cbf3fcbac92522d","kind":"article","url":"https://arxiv.org/abs/2611.05512","sourceUrl":"https://arxiv.org/abs/2611.05512","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T15:01:16.470Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.05512","checkedAt":"2026-11-10T00:40:10.654Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"bbb7f69db5debade679b62450c74a40067db910451c8096a1cbf3fcbac92522d","confidence":0.8400000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"67a84b5038e2cd6f29ade4bfc039702272d0ddc0b8163aff3aca678b629ec9b8","decidedAt":"2026-11-09T15:01:17.617Z"},"provenance":{"textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observationId":"57fce15a915471abad8dd7def5e741ef40ebca0dea7237e5ddfab1535fcd8563","retrievedAt":"2026-11-09T15:01:16.346Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"3ad86ba0-4507-40a7-88c8-ef711dd84c1e"},"reviewer":{"model":"gpt-6-luna","runId":"e9b09d00-676a-4477-8583-23efc562366b"}},"audit":{"status":"passed","reviewedAt":"2026-11-10T00:09:03.449Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Li, Mingxuan","url":null,"kind":"unknown"},{"name":"Mao, Huanzhi","url":null,"kind":"unknown"},{"name":"Zheng, Ruitao","url":null,"kind":"unknown"},{"name":"James, Jarod Stanbury","url":null,"kind":"unknown"},{"name":"Li, Yunze","url":null,"kind":"unknown"},{"name":"Anderson, Natanya","url":null,"kind":"unknown"},{"name":"Gonzalez, Joseph E.","url":null,"kind":"unknown"},{"name":"Dimakis, Alex","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Li, Mingxuan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Mingxuan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"Mao, Huanzhi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mao, Huanzhi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"Zheng, Ruitao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zheng, Ruitao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"James, Jarod Stanbury","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"James, Jarod Stanbury","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"Li, Yunze","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Yunze","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"Anderson, Natanya","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Anderson, Natanya","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"Gonzalez, Joseph E.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gonzalez, Joseph E.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"Dimakis, Alex","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Dimakis, Alex","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.05512","snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z","basis":"publisher-page-metadata","sourceLinks":48,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/b8c5621c88151c7c2d6a5edb67d3bd30ec3f2c6de82fba8380151afb46061ab5/source"}},"canonicalUrl":"https://escapemeridian.com/readings/mingxuan-li-and-coauthors-introduce-slackbench-for-b8c5621c","formats":{"html":"/readings/mingxuan-li-and-coauthors-introduce-slackbench-for-b8c5621c","markdown":"/readings/mingxuan-li-and-coauthors-introduce-slackbench-for-b8c5621c.md","json":"/readings/mingxuan-li-and-coauthors-introduce-slackbench-for-b8c5621c.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-ai-agent","name":"AI agent","explanation":"An AI system that chooses and carries out actions toward a goal, often using software tools.","referenceUrl":"https://www.ibm.com/think/topics/ai-agents"},{"id":"term-synthetic-data","name":"Synthetic data","explanation":"Training or test data generated by software or another model rather than collected from the world.","referenceUrl":"https://en.wikipedia.org/wiki/Synthetic_data"},{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"b8c5621c88151c7c2d6a5edb67d3bd30ec3f2c6de82fba8380151afb46061ab5","revision":"4a595803-4ee5-462a-8582-14a3d6353c65","revisionKind":"publication-id","title":"Mingxuan Li and coauthors introduce SlackBench for collaborative projects","url":"https://escapemeridian.com/readings/mingxuan-li-and-coauthors-introduce-slackbench-for-b8c5621c"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.05512","revision":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","revisionKind":"sha256-source-text","title":"SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories","url":"https://arxiv.org/abs/2611.05512"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"b8c5621c88151c7c2d6a5edb67d3bd30ec3f2c6de82fba8380151afb46061ab5","title":"Mingxuan Li and coauthors introduce SlackBench for collaborative projects","argument":"The SlackBench paper introduces a benchmark for testing whether AI agents can handle collaborative research and engineering work that draws on workplace conversations and changing code repositories. The authors argue that many existing benchmarks miss this complexity by using synthetic data or isolated tasks. In their account, an agent may need to find relevant channels and direct messages, follow long discussions, inspect repository state, and reconcile stale or conflicting evidence.\n\nSlackBench uses privacy-preserving reenactments of real academic research projects, with dialogue, decisions, and code changes tied to GitHub repositories and Google Docs. Its 140 queries test search across sources, abstention when questions cannot be answered or rely on false premises, summarization, and inference from long conversations. The authors say these tasks cannot be reduced to static corpus question answering: agents must identify relevant conversations and combine evidence from messages and project artifacts.\n\nAcross 33 frontier models and agent harnesses, the best system scores around 80%, while most score below 66%. The authors present this gap as substantial headroom for improvement and describe SlackBench as an extensible framework. These benchmark scores indicate performance on the stated tasks; the abstract does not establish how they translate to agents used in everyday work.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported scores are benchmark results; the supplied abstract does not establish how they translate to agents used in everyday collaborative work. The abstract reports overall scores but does not break performance down by the four task types.","speaker":{"name":"Li, Mingxuan, Mao, Huanzhi, Zheng, Ruitao, James, Jarod Stanbury, Li, Yunze, Anderson, Natanya, Gonzalez, Joseph E., Dimakis, Alex","roleAtSource":"Article author"},"source":{"title":"SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.05512","publishedAt":"2026-11-03","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce SlackBench to evaluate agents on research and engineering workflows that require reasoning across workplace communication and changing repository information.","excerpt":"a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state","url":"https://arxiv.org/abs/2611.05512","relationship":"Source-stated"},{"text":"SlackBench is built from privacy-preserving reenactments of real academic research projects, connecting dialogue and evolving decisions with code changes and documents.","excerpt":"privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs.","url":"https://arxiv.org/abs/2611.05512","relationship":"Source-stated"},{"text":"The benchmark contains 140 queries covering search across sources, abstention, summarization, and inference about circumstances in long conversations.","excerpt":"It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations.","url":"https://arxiv.org/abs/2611.05512","relationship":"Source-stated"},{"text":"The authors report that the strongest system scored around 80% across 33 frontier models and agent harnesses, and that most scored below 66%.","excerpt":"Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%","url":"https://arxiv.org/abs/2611.05512","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.05512","checkedAt":"2026-11-10T00:40:10.654Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Which of SlackBench’s four task types account for the largest share of the reported performance gap?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Mingxuan Li and coauthors introduce SlackBench for collaborative projects","summary":"The SlackBench paper introduces a benchmark for testing whether AI agents can handle collaborative research and engineering work that draws on workplace conversations and changing code repositories. The authors argue that many existing benchmarks miss this complexity by using synthetic data or isolated tasks. In their account, an agent may need to find relevant channels and direct messages, follow long discussions, inspect repository state, and reconcile stale or conflicting evidence.\n\nSlackBench uses privacy-preserving reenactments of real academic research projects, with dialogue, decisions, and code changes tied to GitHub repositories and Google Docs. Its 140 queries test search across sources, abstention when questions cannot be answered or rely on false premises, summarization, and inference from long conversations. The authors say these tasks cannot be reduced to static corpus question answering: agents must identify relevant conversations and combine evidence from messages and project artifacts.\n\nAcross 33 frontier models and agent harnesses, the best system scores around 80%, while most score below 66%. The authors present this gap as substantial headroom for improvement and describe SlackBench as an extensible framework. These benchmark scores indicate performance on the stated tasks; the abstract does not establish how they translate to agents used in everyday work.","question":"Which of SlackBench’s four task types account for the largest share of the reported performance gap?","claims":[{"text":"The authors introduce SlackBench to evaluate agents on research and engineering workflows that require reasoning across workplace communication and changing repository information.","excerpt":"a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state","support":"source-stated"},{"text":"SlackBench is built from privacy-preserving reenactments of real academic research projects, connecting dialogue and evolving decisions with code changes and documents.","excerpt":"privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs.","support":"source-stated"},{"text":"The benchmark contains 140 queries covering search across sources, abstention, summarization, and inference about circumstances in long conversations.","excerpt":"It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations.","support":"source-stated"},{"text":"The authors report that the strongest system scored around 80% across 33 frontier models and agent harnesses, and that most scored below 66%.","excerpt":"Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%","support":"source-stated"}],"limitations":["The reported scores are benchmark results; the supplied abstract does not establish how they translate to agents used in everyday collaborative work.","The abstract reports overall scores but does not break performance down by the four task types."],"mediaId":"bbb7f69db5debade679b62450c74a40067db910451c8096a1cbf3fcbac92522d","correctionReason":null,"id":"b8c5621c88151c7c2d6a5edb67d3bd30ec3f2c6de82fba8380151afb46061ab5","publicationId":"4a595803-4ee5-462a-8582-14a3d6353c65","version":1,"sourceUrl":"https://arxiv.org/abs/2611.05512","sourceTitle":"SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-03","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:03:34.268Z","createdAt":"2026-11-10T00:09:03.449Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"bbb7f69db5debade679b62450c74a40067db910451c8096a1cbf3fcbac92522d","kind":"article","url":"https://arxiv.org/abs/2611.05512","sourceUrl":"https://arxiv.org/abs/2611.05512","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T15:01:16.470Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.05512","checkedAt":"2026-11-10T00:40:10.654Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"bbb7f69db5debade679b62450c74a40067db910451c8096a1cbf3fcbac92522d","confidence":0.8400000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"67a84b5038e2cd6f29ade4bfc039702272d0ddc0b8163aff3aca678b629ec9b8","decidedAt":"2026-11-09T15:01:17.617Z"},"provenance":{"textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observationId":"57fce15a915471abad8dd7def5e741ef40ebca0dea7237e5ddfab1535fcd8563","retrievedAt":"2026-11-09T15:01:16.346Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"3ad86ba0-4507-40a7-88c8-ef711dd84c1e"},"reviewer":{"model":"gpt-6-luna","runId":"e9b09d00-676a-4477-8583-23efc562366b"}},"audit":{"status":"passed","reviewedAt":"2026-11-10T00:09:03.449Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Li, Mingxuan","url":null,"kind":"unknown"},{"name":"Mao, Huanzhi","url":null,"kind":"unknown"},{"name":"Zheng, Ruitao","url":null,"kind":"unknown"},{"name":"James, Jarod Stanbury","url":null,"kind":"unknown"},{"name":"Li, Yunze","url":null,"kind":"unknown"},{"name":"Anderson, Natanya","url":null,"kind":"unknown"},{"name":"Gonzalez, Joseph E.","url":null,"kind":"unknown"},{"name":"Dimakis, Alex","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Li, Mingxuan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Mingxuan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"Mao, Huanzhi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mao, Huanzhi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"Zheng, Ruitao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zheng, Ruitao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"James, Jarod Stanbury","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"James, Jarod Stanbury","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"Li, Yunze","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Yunze","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"Anderson, Natanya","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Anderson, Natanya","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"Gonzalez, Joseph E.","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gonzalez, Joseph E.","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}},{"actor":{"name":"Dimakis, Alex","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Dimakis, Alex","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.05512","locator":"head/meta[name=author|citation_author]"},"assertionId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T00:40:10.654Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","sourceUrl":"https://arxiv.org/abs/2611.05512","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.05512","snapshotId":"4a45cf168890c60393f0e6953fe2ad463a7e326feefcd34898688cd0bdc2fa67","contentSha256":"51d6a6559ebf45f0e09365ac5e8ec7c80f80777d1fc993b24647919c41fed7f2","textHash":"f5adfd9fb923744bef605d634efa8162042dd6f6c9e2ec97c4fd2c109aa860f5","observedAt":"2026-11-10T00:40:10.654Z","basis":"publisher-page-metadata","sourceLinks":48,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/b8c5621c88151c7c2d6a5edb67d3bd30ec3f2c6de82fba8380151afb46061ab5/source"}}}],"audits":[{"publicationId":"4a595803-4ee5-462a-8582-14a3d6353c65","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The title credits Mingxuan Li and coauthors, matching the listed authors. The draft frames the benchmark's purpose, critique of existing benchmarks and results as the authors' claims ('The authors argue', 'In their account', 'The authors say').","excerpt":"Authors:Mingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis"},{"dimension":"summary","status":"pass","explanation":"The reading leads with what the benchmark is, explains the problem it targets in plain language and reports the results and the authors' framing. It closes with a caveat that the scores do not show everyday reliability. One fidelity nit: 'goal-directed summarization' becomes 'summarization' and 'circumstance inference' is paraphrased. The smallest repair is to keep the exact terms with tooltips. The question follows from the evidence and is not leading.","excerpt":"heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations"},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-11-03, which matches the v1 submission date. The feed date of 2026-11-05 is later and is correctly not used.","excerpt":"[Submitted on 3 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Every claim matches an abstract excerpt: 140 queries, 33 models and harnesses, best around 80%, most below 66%, and the privacy-preserving reenactments. The limitations correctly note there is no breakdown by task type and no evidence of real-world transfer. The draft adds no unsupported benefits or costs.","excerpt":"Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%, highlighting substantial headroom for improvement."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the arXiv abstract page of the source itself, which is the most direct access to the original.","excerpt":null}],"createdAt":"2026-11-10T00:09:03.449Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}