Code Owns the Simulation, Jev Owns the Evaluation

Code Owns the Simulation, Jev Owns the Evaluation

The paper’s authors argue that Jev can judge options well when the needed information is already in its input, but struggles when it must first predict an unseen event or sequence. They call the first task evaluation and the second simulation. The authors report Jev solves 99% of the counterintuitive Cognitive Reflection Test questions, yet plays one-shot matrix games suboptimally when an opponent’s action is unknown and, in ALFWorld, takes an unwashed knife to the drawer instead of washing it first. The authors say these failures often persist even when Jev can answer separate questions about what an opponent will do or what action fits once that action is supplied. Their proposed division of labor is literal: code performs the prediction or simulation, while Jev evaluates the resulting options. They report that Jev becomes an expert controller when code supplies lookahead in ALFWorld or physics simulation in robot control. The distinction matters for systems that use a model to choose actions: judging a described option is not the same as working out what will happen first. The abstract does not establish how broadly this division of labor works beyond the tested settings.

Publication: a6185f1d-a386-4338-89d1-78f1173b751f

attribution — pass: The reading consistently attributes findings and the proposal to the paper's authors, as listed on the arXiv page. It labels results as reported rather than established fact.

Authors:Yaodong Yang, Hongyao Tang, Yi Ma, Xingyu Fan, Weixun Wang, Jinpeng Li, Tianpei Yang

summary — uncertain: The reading never says what Jev is. The abstract describes it as a judgment model that returns a probability for each described option in one call, without reasoning text. That is why it is attractive as an agent's action-selection layer, and general readers need this. ALFWorld (a text game) and one-shot matrix games also go unexplained. The key mechanism is underplayed: failures arise when one call must both simulate and evaluate. Smallest repair: add one plain sentence defining Jev, and state that single-call condition. Replacing 'literal' with 'suggested' would also match the source's tone.

Judgment models such as \jev{} return, in a single call and without reasoning text, a probability for each described option.

date — pass: The originalPublished date of 2026-10-01 matches the v1 submission date. It is not the v3 revision of 6 Nov or the feed date of 9 Nov.

[Submitted on 1 Oct 2026 (v1), last revised 6 Nov 2026 (this version, v3)]

support — pass: The claims are grounded in the abstract: the evaluation/simulation distinction, the 99% CRT result, the game and ALFWorld failures, the separate-question success, and expert control when code supplies a simulation. The stated significance and the open question stay within the evidence and avoid overgeneralizing. 'Often persist' is a mild paraphrase of 'many of these failures are not due to a lack of knowledge', but it is acceptable.

It fails when one call must both perform the simulation and evaluate based on it.

media — pass: The selected media is the verified arXiv abstract page, which is the direct original source.