Designing Benchmarks for Knowledge Work

Designing Benchmarks for Knowledge Work

Yining Hua, Hongbin Na, Cyrus Ayubcha, and Levi Lian argue that benchmarks for AI agents doing knowledge work often describe tasks, environments, and metrics while leaving other important design choices implicit. Their paper proposes a four-field record describing represented activity, tested setting, required work product, and evaluated result, so benchmarks designed differently can be compared side by side. To describe work activities, they group O*NET task statements into 18 shared activities using unsupervised learning and human review. Applying their framework to GDPval, OfficeQA Pro, and APEX-SWE, they report that the evaluated result covers less than the required work product in all three. In an audit of all 25 public APEX-SWE Integration tasks, they found that in 13, deliverables could change established source-data content and still pass the tests. The paper offers a worked example and fillable template; the abstract does not establish whether using the record improves comparisons or changes benchmark design in practice.

Publication: d35459d3-a6ce-4057-8627-ac664ca82f5b

attribution — pass: The reading correctly credits Yining Hua, Hongbin Na, Cyrus Ayubcha and Levi Lian as the paper's authors. It frames the benchmark-scope and APEX-SWE audit findings as their reported results, not as established fact or Escape Meridian's view.

Authors:Yining Hua, Hongbin Na, Cyrus Ayubcha, Levi Lian

summary — pass: The summary leads with the authors' argument, then covers the four-field proposal, the 18 O*NET-derived activities, the three benchmark cases and the 13-of-25 audit result. The audit finding is stated concretely enough for a non-specialist to grasp the stake: tests can pass work that alters source data. O*NET and unsupervised learning appear unexplained, but they are technical terms suited to term-library tooltips and do not block understanding. The question asks a genuine open issue and does not lead the reader. Optional additions, not required for fidelity: the theory sources the authors draw on and the two-coder audit design.

Starting from the definition of knowledge work, this paper derives four other design choices that often remain implicit.

date — pass: The displayed original publication date, 2026-05-22, matches the v1 submission date on arXiv. It is not the v3 revision date (3 Nov 2026) or the feed date (5 Nov 2026).

[Submitted on 22 May 2026 (v1), last revised 3 Nov 2026 (this version, v3)]

support — pass: Each claim matches its excerpt. 'Covers less than' faithfully paraphrases 'narrower scope'. 'Some deliverables' understates the 13-of-25 figure, which the summary states exactly. The limitations properly note that the evidence is only the abstract, that the audit covers only the 25 public tasks, and that no practical benefit is demonstrated. No benefits or costs are invented.

We audited the tests of all 25 public APEX-SWE Integration tasks, independently coded by two authors.

media — pass: The selected media is the verified direct arXiv abstract page for this paper, which matches the source URL. It is the most direct way for readers to reach the original.