# Marginal Revolution on whether LLMs are good Bayesians [Canonical reading](https://escapemeridian.com/readings/marginal-revolution-on-whether-llms-are-good-bayesians-a12cefd3) Marginal Revolution presents an exchange in which Bryan Caplan argues that LLMs respond poorly to counterevidence. They seem to move toward the speaker or not change, rather than judge whether an argument was better or worse than expected. ChatGPT calls this a “genuine and important failure mode” and says LLMs treat a plausible counterargument as evidence for its conclusion, instead of comparing it with what they should have expected. ChatGPT agrees that an argument weaker than expected should cause a negative update, even if it is valid as far as it goes. It qualifies Caplan’s suggestion of roughly equal chances of moving toward or against a speaker: there is no general theorem requiring equal movement, though both directions should occur under appropriate calibration assumptions. The exchange makes the distinction between hearing an argument and assessing its strength central to whether LLMs reason like Bayesians. The post asks whether developers will try to fix this, whether they should, and whether prompting can undo the behavior. The supplied text does not include the full dialogue or examples beyond this exchange for judging how often the failure occurs or whether prompting can change it. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source Marginal Revolution · Article Original source date: Nov 1, 2026 (2026-11-01T04:17:10+00:00). Date basis: Date stated by the publisher. Added to Escape Meridian: 2026-11-11T03:10:57.455Z. First observed: 2026-11-01T04:21:30.284Z. Retrieved: 2026-11-01T04:31:23.441Z. - [Are LLMs good or bad Bayesians? - Marginal REVOLUTION](https://marginalrevolution.com/marginalrevolution/2026/11/are-llms-good-or-bad-bayesians.html) ## Limits The supplied text says a full dialogue exists but does not include it, limiting assessment of the exchange's context and examples. The excerpt raises whether prompting can undo the described behavior but does not report a test or answer. ## Open question What evidence would show whether prompting can undo the failure mode ChatGPT describes? ## Source-stated Caplan argues that LLMs seem to move toward a speaker or not change when presented with counterevidence, rather than respond according to whether the argument was better or worse than expected. > you seem to either move toward the speaker or just not change. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-llms-good-or-bad-bayesians.html) ## Source-stated ChatGPT describes treating a plausible counterargument as evidence for its conclusion, rather than comparing it with what should have been expected, as a failure mode in AI reasoning. > LLMs tend to treat a plausible counterargument as evidence in favor of its conclusion, rather than comparing the counterargument to what they should have expected to hear. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-llms-good-or-bad-bayesians.html) ## Source-stated ChatGPT qualifies the idea that updates toward and away from a speaker should happen with roughly equal probability, saying both directions should occur under appropriate calibration assumptions. > there’s no general theorem that updating toward and away from a speaker should happen with equal probability. But under appropriate calibration assumptions, both directions should occur - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-llms-good-or-bad-bayesians.html) ## Source-stated ChatGPT says an argument weaker than expected should cause a negative update, even if the argument is valid as far as it goes. > an argument that’s weaker than expected should cause a negative update—even if the argument is perfectly valid as far as it goes. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-llms-good-or-bad-bayesians.html) ## Source-stated The post asks whether the failure can be fixed or undone through prompting, but the supplied excerpt gives no answer. > But will they try to fix this? Should they try to fix this? Can you undo this with good enough prompting? - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-llms-good-or-bad-bayesians.html) ## Source and contributor context Tyler Cowen · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Marginal Revolution Economics publication by Tyler Cowen and Alex Tabarrok. - [Explore this context](/entities/marginal-revolution) - [Reference](https://marginalrevolution.com/) ## Tyler Cowen Economics, institutions and unexpected cross-domain questions; useful source discovery. - [Explore this context](/entities/tyler-cowen) - [Reference](https://www.dwarkesh.com/p/tyler-cowen-4) - [Reference](https://www.dwarkesh.com/p/tyler-cowen-3) - [Reference](https://www.dwarkesh.com/p/tyler-cowen-2) - [Reference](https://marginalrevolution.com/marginalrevolution/2026/10/saturday-assorted-links-583.html?utm_source=rss&utm_medium=rss&utm_campaign=saturday-assorted-links-583) - [Reference](https://tylercowen.com/) ## Editorial provenance Version 1. Publication ID: 35487807-8953-4709-8398-f16d40727d87. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-11T03:10:57.455Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: useful\_perspective. Observation: a4df7d76d596630014019add8f31775fc0224cba1f523f21dade76eb47af49bc. Source text hash: 25f76a5a59ed8641cb24a40d83e0e5724101eb49ea5a4d0b376c0adb24d3f289. ## Revision and source-check history ### Version 1 — 2026-11-11T03:10:57.455Z Marginal Revolution presents an exchange in which Bryan Caplan argues that LLMs respond poorly to counterevidence. They seem to move toward the speaker or not change, rather than judge whether an argument was better or worse than expected. ChatGPT calls this a “genuine and important failure mode” and says LLMs treat a plausible counterargument as evidence for its conclusion, instead of comparing it with what they should have expected. ChatGPT agrees that an argument weaker than expected should cause a negative update, even if it is valid as far as it goes. It qualifies Caplan’s suggestion of roughly equal chances of moving toward or against a speaker: there is no general theorem requiring equal movement, though both directions should occur under appropriate calibration assumptions. The exchange makes the distinction between hearing an argument and assessing its strength central to whether LLMs reason like Bayesians. The post asks whether developers will try to fix this, whether they should, and whether prompting can undo the behavior. The supplied text does not include the full dialogue or examples beyond this exchange for judging how often the failure occurs or whether prompting can change it. ## Original source Marginal Revolution · Article Original source date: Nov 1, 2026 (2026-11-01T04:17:10+00:00). Date basis: Date stated by the publisher. Added to Escape Meridian: 2026-11-11T03:10:57.455Z. First observed: 2026-11-01T04:21:30.284Z. Retrieved: 2026-11-01T04:31:23.441Z. - [Are LLMs good or bad Bayesians? - Marginal REVOLUTION](https://marginalrevolution.com/marginalrevolution/2026/11/are-llms-good-or-bad-bayesians.html) ## Limits The supplied text says a full dialogue exists but does not include it, limiting assessment of the exchange's context and examples. The excerpt raises whether prompting can undo the described behavior but does not report a test or answer. ## Open question What evidence would show whether prompting can undo the failure mode ChatGPT describes? ## Source-stated Caplan argues that LLMs seem to move toward a speaker or not change when presented with counterevidence, rather than respond according to whether the argument was better or worse than expected. > you seem to either move toward the speaker or just not change. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-llms-good-or-bad-bayesians.html) ## Source-stated ChatGPT describes treating a plausible counterargument as evidence for its conclusion, rather than comparing it with what should have been expected, as a failure mode in AI reasoning. > LLMs tend to treat a plausible counterargument as evidence in favor of its conclusion, rather than comparing the counterargument to what they should have expected to hear. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-llms-good-or-bad-bayesians.html) ## Source-stated ChatGPT qualifies the idea that updates toward and away from a speaker should happen with roughly equal probability, saying both directions should occur under appropriate calibration assumptions. > there’s no general theorem that updating toward and away from a speaker should happen with equal probability. But under appropriate calibration assumptions, both directions should occur - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-llms-good-or-bad-bayesians.html) ## Source-stated ChatGPT says an argument weaker than expected should cause a negative update, even if the argument is valid as far as it goes. > an argument that’s weaker than expected should cause a negative update—even if the argument is perfectly valid as far as it goes. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-llms-good-or-bad-bayesians.html) ## Source-stated The post asks whether the failure can be fixed or undone through prompting, but the supplied excerpt gives no answer. > But will they try to fix this? Should they try to fix this? Can you undo this with good enough prompting? - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-llms-good-or-bad-bayesians.html) ## Source and contributor context Tyler Cowen · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Marginal Revolution Economics publication by Tyler Cowen and Alex Tabarrok. - [Explore this context](/entities/marginal-revolution) - [Reference](https://marginalrevolution.com/) ## Tyler Cowen Economics, institutions and unexpected cross-domain questions; useful source discovery. - [Explore this context](/entities/tyler-cowen) - [Reference](https://www.dwarkesh.com/p/tyler-cowen-4) - [Reference](https://www.dwarkesh.com/p/tyler-cowen-3) - [Reference](https://www.dwarkesh.com/p/tyler-cowen-2) - [Reference](https://marginalrevolution.com/marginalrevolution/2026/10/saturday-assorted-links-583.html?utm_source=rss&utm_medium=rss&utm_campaign=saturday-assorted-links-583) - [Reference](https://tylercowen.com/) ## Editorial provenance Version 1. Publication ID: 35487807-8953-4709-8398-f16d40727d87. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-11T03:10:57.455Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: useful\_perspective. Observation: a4df7d76d596630014019add8f31775fc0224cba1f523f21dade76eb47af49bc. Source text hash: 25f76a5a59ed8641cb24a40d83e0e5724101eb49ea5a4d0b376c0adb24d3f289. ### passed — 2026-11-11T03:10:57.455Z — anthropic/claude-opus-5.5 Publication: 35487807-8953-4709-8398-f16d40727d87 attribution — pass: The reading correctly attributes the critique to Bryan Caplan and the agreement and qualification to ChatGPT, within a Marginal Revolution post. The closing questions are attributed to the post. Rendering 'they' as developers is a reasonable reading of the post's ambiguous pronoun and does not misattribute any claim. > But will they try to fix this? Should they try to fix this? Can you undo this with good enough prompting? summary — pass: The summary is understandable on its own. It names the actors and explains the core idea in plain language: judging whether an argument was better or worse than expected, rather than simply moving toward the speaker. It keeps ChatGPT's qualification and the post's open questions. Terms such as Bayesian, negative update and calibration are kept exact and are suitable for term tooltips. The question follows directly from the source's own prompting question. > It depends on whether the argument was better or worse than expected. date — pass: The originalPublished value of 2026-11-01T04:17:10Z comes from the publisher page and matches the feed entry. The draft makes no conflicting date claim. support — pass: Each claim matches a verbatim excerpt, including the equal-probability qualification and the point that a valid but weaker-than-expected argument should cause a negative update. The limitations correctly note that the full dialogue is absent and that no prompting test is reported. The reading adds no invented benefits, costs or forecasts. > an argument that’s weaker than expected should cause a negative update—even if the argument is perfectly valid as far as it goes. media — pass: The selected media is the verified direct link to the original Marginal Revolution article, which is the most direct way to reach the source. No other source media was found. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/a12cefd36e20b21203143aa4e314360aa8b89baeda59115dad4637a8880a577b/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/marginal-revolution-on-whether-llms-are-good-bayesians-a12cefd3.json)