# arXiv: Hou and Yamada introduce S1CogBench to test System One decision models [Canonical reading](https://escapemeridian.com/readings/arxiv-hou-and-yamada-introduce-s1cogbench-to-test-system-one-9f8aa843) Dengzhe Hou and Kazunori D Yamada introduce S1CogBench, a benchmark of System One decision models on cognitive experiments. These models return a single decision from the legal options, without text explaining it; the authors argue that a well-formed answer alone does not show whether a model used the evidence. The authors evaluate five open models and Jev on 24 cognitive tasks. They report that reliability varies widely, good performance does not necessarily predict human behavior, and aggregate scores can hide how decisions change with task conditions. They also say the models’ speed comes mainly from answering without generating text. The benchmark’s significance, as the authors frame it, is to test how decisions respond to evidence, feedback, memory load, and future opportunities before software relies on them for routing, guardrails, or automated gates. The abstract does not establish how any particular model would perform in those uses, or whether benchmark results predict real-world reliability. An open question is: across the 24 tasks, which models’ decisions change appropriately when evidence, feedback, memory load, or future opportunities change? Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv q-bio.NC · Article Original source date: Oct 31, 2026 (2026-10-31). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-04T05:47:29.926Z. First observed: 2026-11-03T08:51:24.119Z. Retrieved: 2026-11-04T05:46:32.737Z. - [S1CogBench: What System One Decision Models Can and Cannot Do on Cognitive Experiments](https://arxiv.org/abs/2611.00911) ## Limits The supplied evidence is the arXiv abstract, not the paper’s detailed methods, task-level results, or scoring; it does not establish how individual models performed. The abstract does not show whether benchmark results predict reliability in deployed software. ## Open question Across the 24 tasks, which models’ decisions change appropriately when evidence, feedback, memory load, or future opportunities change? ## Source-stated Hou and Yamada introduce S1CogBench to evaluate System One decision models on cognitive experiments. > We introduce S1CogBench to ask what these models can and cannot do on 24 cognitive tasks - [Supporting source](https://arxiv.org/abs/2611.00911) ## Source-stated The authors say that a well-formed decision does not establish that a model used the evidence. > a fast, well-formed decision does not show that the model used the evidence. - [Supporting source](https://arxiv.org/abs/2611.00911) ## Source-stated The authors report wide variation in reliability, a gap between acting well and predicting humans, and that aggregate scores can conceal responses to conditions. > We find that decision models vary widely in reliability, acting well does not imply predicting humans, aggregate scores hide how decisions respond to conditions - [Supporting source](https://arxiv.org/abs/2611.00911) ## Source-stated The authors say the models’ speed mainly comes from answering without generating text. > speed comes mainly from answering without generation. - [Supporting source](https://arxiv.org/abs/2611.00911) ## Source-stated The authors frame the benchmark as a shared test of whether decisions respond to evidence, feedback, memory load, and future opportunities before software relies on them. > it gives a shared test of whether their decisions respond to evidence, feedback, memory load, and future opportunities before software relies on them. - [Supporting source](https://arxiv.org/abs/2611.00911) ## Source and contributor context Hou, Dengzhe, Yamada, Kazunori D · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Jev Jev is TypeSafe AI’s model for fast evaluation and routing decisions inside AI systems. - [Explore this context](/entities/jev) - [Reference](https://typesafe.ai/blog/introducing-system-one-models-and-jev) - [Reference](https://typesafe.ai/) ## Editorial provenance Version 1. Publication ID: c8b0a06c-6695-4857-8790-13784ea522d7. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-04T05:47:29.926Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: fa3f6ef90a8025d6f798416cc6c661e260c5bd69f2f7aa9f23bda2e06197b686. Source text hash: 0b16ff3c2f2f5c15a9d57fbf6ce34d6e559636a99d487527ecdee8f775e55296. ## Revision and source-check history ### Version 1 — 2026-11-04T05:47:29.926Z Dengzhe Hou and Kazunori D Yamada introduce S1CogBench, a benchmark of System One decision models on cognitive experiments. These models return a single decision from the legal options, without text explaining it; the authors argue that a well-formed answer alone does not show whether a model used the evidence. The authors evaluate five open models and Jev on 24 cognitive tasks. They report that reliability varies widely, good performance does not necessarily predict human behavior, and aggregate scores can hide how decisions change with task conditions. They also say the models’ speed comes mainly from answering without generating text. The benchmark’s significance, as the authors frame it, is to test how decisions respond to evidence, feedback, memory load, and future opportunities before software relies on them for routing, guardrails, or automated gates. The abstract does not establish how any particular model would perform in those uses, or whether benchmark results predict real-world reliability. An open question is: across the 24 tasks, which models’ decisions change appropriately when evidence, feedback, memory load, or future opportunities change? ## Original source arXiv q-bio.NC · Article Original source date: Oct 31, 2026 (2026-10-31). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-04T05:47:29.926Z. First observed: 2026-11-03T08:51:24.119Z. Retrieved: 2026-11-04T05:46:32.737Z. - [S1CogBench: What System One Decision Models Can and Cannot Do on Cognitive Experiments](https://arxiv.org/abs/2611.00911) ## Limits The supplied evidence is the arXiv abstract, not the paper’s detailed methods, task-level results, or scoring; it does not establish how individual models performed. The abstract does not show whether benchmark results predict reliability in deployed software. ## Open question Across the 24 tasks, which models’ decisions change appropriately when evidence, feedback, memory load, or future opportunities change? ## Source-stated Hou and Yamada introduce S1CogBench to evaluate System One decision models on cognitive experiments. > We introduce S1CogBench to ask what these models can and cannot do on 24 cognitive tasks - [Supporting source](https://arxiv.org/abs/2611.00911) ## Source-stated The authors say that a well-formed decision does not establish that a model used the evidence. > a fast, well-formed decision does not show that the model used the evidence. - [Supporting source](https://arxiv.org/abs/2611.00911) ## Source-stated The authors report wide variation in reliability, a gap between acting well and predicting humans, and that aggregate scores can conceal responses to conditions. > We find that decision models vary widely in reliability, acting well does not imply predicting humans, aggregate scores hide how decisions respond to conditions - [Supporting source](https://arxiv.org/abs/2611.00911) ## Source-stated The authors say the models’ speed mainly comes from answering without generating text. > speed comes mainly from answering without generation. - [Supporting source](https://arxiv.org/abs/2611.00911) ## Source-stated The authors frame the benchmark as a shared test of whether decisions respond to evidence, feedback, memory load, and future opportunities before software relies on them. > it gives a shared test of whether their decisions respond to evidence, feedback, memory load, and future opportunities before software relies on them. - [Supporting source](https://arxiv.org/abs/2611.00911) ## Source and contributor context Hou, Dengzhe, Yamada, Kazunori D · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Jev Jev is TypeSafe AI’s model for fast evaluation and routing decisions inside AI systems. - [Explore this context](/entities/jev) - [Reference](https://typesafe.ai/blog/introducing-system-one-models-and-jev) - [Reference](https://typesafe.ai/) ## Editorial provenance Version 1. Publication ID: c8b0a06c-6695-4857-8790-13784ea522d7. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-04T05:47:29.926Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: fa3f6ef90a8025d6f798416cc6c661e260c5bd69f2f7aa9f23bda2e06197b686. Source text hash: 0b16ff3c2f2f5c15a9d57fbf6ce34d6e559636a99d487527ecdee8f775e55296. ### passed — 2026-11-04T05:47:29.926Z — anthropic/claude-opus-5.5 Publication: c8b0a06c-6695-4857-8790-13784ea522d7 attribution — pass: The draft correctly attributes S1CogBench to Dengzhe Hou and Kazunori D Yamada on arXiv. It keeps the findings and framing as the authors' claims rather than established facts. > Authors:Dengzhe Hou, Kazunori D Yamada summary — pass: The draft states the development and the central argument: a fast, well-formed decision does not show that the evidence was used. It explains System One decision models in plain language and accurately reports the four findings. The question follows from the stated aim. Optional repair: note that Jev is one of the source's examples of System One decision models, and briefly gloss routing, guardrails and automated gates. These gaps do not block understanding. > System One decision models, also called typed decision models, such as Jev and Laya, return one fast decision over the legal options, leaving no text to inspect. date — pass: The originalPublished date, 2026-10-31, matches the arXiv v1 submission date. The later feed date is correctly not used. > \[Submitted on 31 Oct 2026\] support — pass: Each claim is source-stated with an exact excerpt. The limitations correctly note that the abstract does not establish model-level results or deployed reliability. The draft avoids turning benchmark findings into real-world reliability and invents no stakeholder benefits or costs. > We find that decision models vary widely in reliability, acting well does not imply predicting humans, aggregate scores hide how decisions respond to conditions, and speed comes mainly from answering without generation. media — pass: The selected media is the verified arXiv abstract page for this paper. It is the direct original source and the most relevant access point. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/9f8aa843fc02e1af9f3e629ed1bfb30cb3ebff0d371b4ec9239085617fe2a90d/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-hou-and-yamada-introduce-s1cogbench-to-test-system-one-9f8aa843.json)