Salapa, George Juraj · Oct 26, 2026
kw-bench: Breaking Frontier Harnesses on Real-World Tasks
Markdown · JSON · Original source
George Juraj Salapa introduces kw-bench, a for investigative knowledge work: recovering answers hidden in incomplete, inconsistent records rather than generating a document from a brief. Its first task asks a model working inside a harness—a loop with tools such as a shell and filesystem—to reconstruct VAT across three generations of SAP from raw system extracts. The paper says grading is deterministic, with no judge. In the reported two-hour runs, four harnesses used the same model. OpenCode and micro-cc scored 0.542 and 0.528; the two lab harnesses each scored 0.00. Salapa says “inertia”—keeping weak signals alive across a long session—defeated those harnesses. With Opus 5.5, micro-cc and Claude Code scored 0.9986 and 1.000 in 22 and 25 minutes. Salapa argues that below the frontier, the harness affects how far a model gets, while a makes this task much easier. He says a weeks-long engagement now costs minutes and a few dollars; the abstract does not establish that result across other engagements. How well would these results hold across other investigative tasks and records?
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 26, 2026 (2026-10-26).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-14T00:01:16.312Z. First observed: 2026-10-28T08:24:36.660Z. Retrieved: 2026-10-30T21:20:33.758Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract describes only the first task family, a VAT reconstruction, and does not report how results generalize to other investigative tasks. The abstract states that a weeks-long engagement costs minutes and a few dollars but does not specify the basis for that comparison or establish it across engagements. The reported results are for the listed harnesses, models, task, and run durations; the abstract alone does not show how performance changes under other conditions.
Open question
How well would these results hold across other investigative tasks and records?
Source-stated
Salapa presents kw-bench as a benchmark for investigative work, where answers must be recovered from incomplete and inconsistent evidence.
the answer already exists -- hidden in incomplete and inconsistent evidence spread across systems -- and has to be recovered and defended to the cent.
Source-stated
The first task reconstructs five years of VAT across three generations of SAP, using raw system extracts.
Its first task family is a five-year VAT reconstruction across three generations of SAP
Source-stated
The benchmark uses deterministic grading without an LLM judge.
Grading is deterministic, with no LLM judge.
Source-stated
In a two-hour run using DeepSeek v4.1 Flash, OpenCode scored 0.542 and micro-cc 0.528, while Claude Code and Codex each scored 0.00.
OpenCode 0.542, micro-cc 0.528); the two lab harnesses, away from their home model, delivered nothing (0.00 each).
Source-stated
The author attributes the weaker results to harnesses failing to keep weak signals active across a long session.
What defeated them was inertia: keeping many weak signals alive across a long session.
Source-stated
On Claude Opus 5.5, micro-cc and Claude Code scored 0.9986 and 1.000 in 22 and 25 minutes.
micro-cc and Claude Code on Claude Opus 5.5 scored 0.9986 and 1.000, in 22 and 25 minutes.
Source-stated
Salapa argues that below the frontier, the harness affects how far a model gets, and says a weeks-long engagement could take minutes and a few dollars.
A weeks-long engagement now costs minutes and a few dollars; below the frontier, the harness decides how far a model gets.
Source and contributor context
Salapa, George Juraj · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 2cb147b2-d1d0-4fcf-8fcc-beb6d0d2c66e.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-14T00:01:16.312Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: d589aeafcc1c0effa6ef79ec1fb4bb9e007970e7d6d4a958a643e3adf1ad6ddd. Source text hash: ac5f4ce184531093625e38898c84bdeca15b5e668f3af2d4ae2620b35cb8e9f5.
Revision and source-check history
Version 1 · 2026-11-14T00:01:16.312Z
kw-bench: Breaking Frontier Harnesses on Real-World Tasks
George Juraj Salapa introduces kw-bench, a for investigative knowledge work: recovering answers hidden in incomplete, inconsistent records rather than generating a document from a brief. Its first task asks a model working inside a harness—a loop with tools such as a shell and filesystem—to reconstruct VAT across three generations of SAP from raw system extracts. The paper says grading is deterministic, with no judge. In the reported two-hour runs, four harnesses used the same model. OpenCode and micro-cc scored 0.542 and 0.528; the two lab harnesses each scored 0.00. Salapa says “inertia”—keeping weak signals alive across a long session—defeated those harnesses. With Opus 5.5, micro-cc and Claude Code scored 0.9986 and 1.000 in 22 and 25 minutes. Salapa argues that below the frontier, the harness affects how far a model gets, while a makes this task much easier. He says a weeks-long engagement now costs minutes and a few dollars; the abstract does not establish that result across other engagements. How well would these results hold across other investigative tasks and records?
Original source
cs.AI · Article
Source details
Original source date: Oct 26, 2026 (2026-10-26).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-14T00:01:16.312Z. First observed: 2026-10-28T08:24:36.660Z. Retrieved: 2026-10-30T21:20:33.758Z.
Limits
The abstract describes only the first task family, a VAT reconstruction, and does not report how results generalize to other investigative tasks. The abstract states that a weeks-long engagement costs minutes and a few dollars but does not specify the basis for that comparison or establish it across engagements. The reported results are for the listed harnesses, models, task, and run durations; the abstract alone does not show how performance changes under other conditions.
Open question
How well would these results hold across other investigative tasks and records?
Source-stated
Salapa presents kw-bench as a benchmark for investigative work, where answers must be recovered from incomplete and inconsistent evidence.
the answer already exists -- hidden in incomplete and inconsistent evidence spread across systems -- and has to be recovered and defended to the cent.
Source-stated
The first task reconstructs five years of VAT across three generations of SAP, using raw system extracts.
Its first task family is a five-year VAT reconstruction across three generations of SAP
Source-stated
The benchmark uses deterministic grading without an LLM judge.
Grading is deterministic, with no LLM judge.
Source-stated
In a two-hour run using DeepSeek v4.1 Flash, OpenCode scored 0.542 and micro-cc 0.528, while Claude Code and Codex each scored 0.00.
OpenCode 0.542, micro-cc 0.528); the two lab harnesses, away from their home model, delivered nothing (0.00 each).
Source-stated
The author attributes the weaker results to harnesses failing to keep weak signals active across a long session.
What defeated them was inertia: keeping many weak signals alive across a long session.
Source-stated
On Claude Opus 5.5, micro-cc and Claude Code scored 0.9986 and 1.000 in 22 and 25 minutes.
micro-cc and Claude Code on Claude Opus 5.5 scored 0.9986 and 1.000, in 22 and 25 minutes.
Source-stated
Salapa argues that below the frontier, the harness affects how far a model gets, and says a weeks-long engagement could take minutes and a few dollars.
A weeks-long engagement now costs minutes and a few dollars; below the frontier, the harness decides how far a model gets.
Source and contributor context
Salapa, George Juraj · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Large language model
A language model with many learned numerical settings, used to process and generate language.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: 2cb147b2-d1d0-4fcf-8fcc-beb6d0d2c66e.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-14T00:01:16.312Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: d589aeafcc1c0effa6ef79ec1fb4bb9e007970e7d6d4a958a643e3adf1ad6ddd. Source text hash: ac5f4ce184531093625e38898c84bdeca15b5e668f3af2d4ae2620b35cb8e9f5.
uncertain · 2026-11-14T00:01:16.312Z · anthropic/claude-opus-5.5
Publication: 2cb147b2-d1d0-4fcf-8fcc-beb6d0d2c66e
attribution — pass: The paper is correctly attributed to George Juraj Salapa. The benchmark, the scores and the inertia diagnosis are kept as the author's reported results and claims.
Authors:George Juraj Salapa
summary — uncertain: The summary softens the author's force and drops a key condition. The source says the harness 'decides' how far a model gets; the draft says it 'affects'. The source says the task 'collapses' on a frontier model; the draft says 'much easier'. The summary never names DeepSeek v4.1 Flash. It also does not say that the zero-scoring lab harnesses were Claude Code and Codex running 'away from their home model'. Without that, a lay reader cannot see that Claude Code scored 0.00 on DeepSeek but 1.000 on Opus. Smallest repair: restore 'decides' and 'collapses', name the model and the two lab harnesses, and add the home-model condition. VAT, SAP and LLM judge may need term tooltips.
the two lab harnesses, away from their home model, delivered nothing (0.00 each)
date — pass: The originalPublished date of 2026-10-26 matches the v1 submission date. The later feed date was correctly not used.
[Submitted on 26 Oct 2026]
support — pass: Each claim is source-stated with a matching excerpt. The cost claim is attributed to the author and flagged as not established across engagements. The limitations and the question follow from the abstract's single task family and do not invent benefits or costs.
A weeks-long engagement now costs minutes and a few dollars; below the frontier, the harness decides how far a model gets.
media — pass: The selected media is the verified direct link to the original arXiv abstract page, which is the source itself.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv