arXiv cs.AI: LUCA: A Synthetic World for Experimental Reasoning

arXiv cs.AI: LUCA: A Synthetic World for Experimental Reasoning

Authors Anna Gogleva, Thomas Charman, Martin Balla and Karl Gemayel introduce LUCA, a configurable synthetic world for testing how agents investigate unknown systems—not just whether they reach the right outcome. They argue that an agent can succeed for the wrong reasons when evaluations judge outcomes but overlook experiments, evidence and inferences. In LUCA, agents use limited experiments to learn regulatory systems, predict unseen interventions, diagnose hidden faults and attempt rescues. The authors report that these abilities can diverge: agents may rescue a system while identifying the wrong fault, and collecting evidence does not guarantee that an agent uses it. They also find that models adopt different experimental strategies, while more exploration does not reliably improve performance. The authors present LUCA as a controlled way to measure where experimental agents succeed and fail, separating skills that an outcome-only score could blur. The abstract reports results across seven frontier LLMs and three baselines, but does not establish how well these findings transfer beyond the synthetic world. How do agents’ experimental and diagnostic failures compare when evaluated on real systems with known ground truth?

Publication: a77f577f-7f94-410e-8309-9985ad3ca326

attribution — pass: The draft correctly credits the four listed authors and keeps the findings framed as what the authors introduce, argue and report. The reading does not present them as independently verified facts.

Authors:Anna Gogleva, Thomas Charman, Martin Balla, Karl Gemayel

summary — uncertain: Two passages need repair. First, 'regulatory systems' is central but unexplained; a general reader could take it to mean legal or policy regulation. The abstract describes unknown systems that agents probe through experiments. Add a tooltip or plain-language gloss without inventing a domain. Second, the draft softens the authors' frequency claims: 'frequently' becomes 'may', and 'often perform the experiment... but still fail' becomes 'does not guarantee'. It also drops the striking 7x versus 1.4x contrast. Restoring 'frequently', 'often' and that ratio would preserve the authors' force. Terms such as frontier LLMs, baselines and ground truth would benefit from tooltips.

Agents frequently rescue malfunctioning systems while diagnosing the wrong fault; diagnosis varies 7x across models while prediction varies only 1.4x.

date — pass: The originalPublished date, 2026-11-02, matches the v1 submission date. The later feed date of November 3 is correctly not used.

[Submitted on 2 Nov 2026]

support — pass: Every claim matches a verbatim excerpt from the abstract. The stated limitation that transfer beyond the synthetic world is not established is a fair gap rather than an invented claim. The question follows from that gap and does not assert an answer.

In one failure mode, agents often perform the experiment needed to distinguish competing causes, but still fail to use the resulting evidence.

media — pass: The selected media is the verified arXiv abstract page for this paper. It is the direct original source.