# Marginal Revolution introduces a paper arguing many development-economics RCTs are underpowered [Canonical reading](https://escapemeridian.com/readings/marginal-revolution-introduces-a-paper-arguing-many-development-5c652c12) Marginal Revolution introduces a paper by Qingyuan Chai, Kevin Lang and Liang Zhong on the statistical credibility of randomized controlled trials (RCTs) in development economics. The authors report that actual t-statistics peak below the theoretical benchmark expected under 80 percent power, but above values associated with p-hacking; they say this suggests many RCTs are underpowered. Using an empirical Bayes mixture model that accounts for selective publication, they estimate that approximately 30 percent of published rejections are false discoveries, with average directional power of 40 percent. At conventional significance thresholds, they estimate a 66 percent posterior probability of a false discovery. The paper compares two changes: doubling original sample sizes reduces the average false discovery rate by five percentage points, while tightening the significance threshold to 0.005 yields a larger reduction at the cost of total discoveries. It also models replication funding under a fixed budget and evidence standard. Under standard regularity conditions, the authors’ model allocates replications to an intermediate interval of original test statistics, with optimal replication sample size single-peaked in the original t-statistic. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source Marginal Revolution · Article Original source date: Nov 10, 2026 (2026-11-10T20:05:26+00:00). Date basis: Date stated by the publisher. Added to Escape Meridian: 2026-11-10T21:02:41.306Z. First observed: 2026-11-10T20:09:23.668Z. Retrieved: 2026-11-10T21:01:10.873Z. - [Are RCTs underpowered? - Marginal REVOLUTION](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Limits The analysis described is limited to 745 principal-hypothesis tests from 190 articles in leading economics journals, in development economics. The source presents the replication-policy results as a model under standard regularity conditions; its reported post-replication rates are illustrative calibrations. The source does not specify the evidence standards behind the 1.8–7.8 percent calibration range. ## Open question How do the paper’s replication-policy results vary across evidence standards? ## Source-stated The authors report that t-statistics in their sample peak below the theoretical benchmark expected under 80 percent power, which they say suggests many RCTs are underpowered. > actual t-statistics peak well below the theoretical benchmark of 2.8 expected under 80 percent power, though above values associated with p-hacking. This suggests many RCTs are underpowered. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Source-stated Using an empirical Bayes mixture model that accounts for selective publication, the authors estimate that approximately 30 percent of published rejections are false discoveries and average directional power is 40 percent. > we find that approximately 30 percent of published rejections are false discoveries, with average directional power at only 40 percent. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Source-stated At conventional significance thresholds, the authors estimate a 66 percent posterior probability of a false discovery. > At conventional significance thresholds, the posterior probability of a false discovery reaches 66 percent. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Source-stated The authors report that doubling original sample sizes reduces average false discovery rates by five percentage points; tightening the significance threshold to 0.005 reduces them more, but results in fewer total discoveries. > Doubling original sample sizes lowers the average false discovery rate by five percentage points, whereas tightening the significance threshold to 0.005 yields a larger reduction at the cost of total discoveries. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Source-stated Their model of replication funding under a fixed budget and evidence standard places replications in an intermediate interval of original test statistics; optimal replication sample size is single-peaked in the original t-statistic. > replications concentrate in an intermediate interval of original test statistics, and the optimal replication sample size is single-peaked in the original t-statistic. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Source-stated Illustrative calibrations estimate that successful replications reduce aggregate false discovery rates to between 1.8 and 7.8 percent, depending on the evidence standard. > Illustrative calibrations show that successful replications reduce aggregate false discovery rates to 1.8–7.8 percent, depending on the evidence standard. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Source and contributor context Tyler Cowen · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Randomized controlled trial A study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment. - [Definition reference](https://en.wikipedia.org/wiki/Randomized_controlled_trial) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Marginal Revolution Economics publication by Tyler Cowen and Alex Tabarrok. - [Explore this context](/entities/marginal-revolution) - [Reference](https://marginalrevolution.com/) ## Tyler Cowen Economics, institutions and unexpected cross-domain questions; useful source discovery. - [Explore this context](/entities/tyler-cowen) - [Reference](https://www.dwarkesh.com/p/tyler-cowen-4) - [Reference](https://www.dwarkesh.com/p/tyler-cowen-3) - [Reference](https://www.dwarkesh.com/p/tyler-cowen-2) - [Reference](https://marginalrevolution.com/marginalrevolution/2026/10/saturday-assorted-links-583.html?utm_source=rss&utm_medium=rss&utm_campaign=saturday-assorted-links-583) - [Reference](https://tylercowen.com/) ## Editorial provenance Version 1. Publication ID: 5b8e4b7a-35fd-4280-8dc0-e76c304fabd4. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T21:02:41.306Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: useful\_perspective. Observation: 0d99035d657af4ff01ea8a35623c281493f8654f0a0250c70041dce916211e24. Source text hash: 83baf6db00a1fdc9ea872bb0391dbdc5e52e6b112fec42bfc6631c4088cd1d89. ## Revision and source-check history ### Version 1 — 2026-11-10T21:02:41.306Z Marginal Revolution introduces a paper by Qingyuan Chai, Kevin Lang and Liang Zhong on the statistical credibility of randomized controlled trials (RCTs) in development economics. The authors report that actual t-statistics peak below the theoretical benchmark expected under 80 percent power, but above values associated with p-hacking; they say this suggests many RCTs are underpowered. Using an empirical Bayes mixture model that accounts for selective publication, they estimate that approximately 30 percent of published rejections are false discoveries, with average directional power of 40 percent. At conventional significance thresholds, they estimate a 66 percent posterior probability of a false discovery. The paper compares two changes: doubling original sample sizes reduces the average false discovery rate by five percentage points, while tightening the significance threshold to 0.005 yields a larger reduction at the cost of total discoveries. It also models replication funding under a fixed budget and evidence standard. Under standard regularity conditions, the authors’ model allocates replications to an intermediate interval of original test statistics, with optimal replication sample size single-peaked in the original t-statistic. ## Original source Marginal Revolution · Article Original source date: Nov 10, 2026 (2026-11-10T20:05:26+00:00). Date basis: Date stated by the publisher. Added to Escape Meridian: 2026-11-10T21:02:41.306Z. First observed: 2026-11-10T20:09:23.668Z. Retrieved: 2026-11-10T21:01:10.873Z. - [Are RCTs underpowered? - Marginal REVOLUTION](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Limits The analysis described is limited to 745 principal-hypothesis tests from 190 articles in leading economics journals, in development economics. The source presents the replication-policy results as a model under standard regularity conditions; its reported post-replication rates are illustrative calibrations. The source does not specify the evidence standards behind the 1.8–7.8 percent calibration range. ## Open question How do the paper’s replication-policy results vary across evidence standards? ## Source-stated The authors report that t-statistics in their sample peak below the theoretical benchmark expected under 80 percent power, which they say suggests many RCTs are underpowered. > actual t-statistics peak well below the theoretical benchmark of 2.8 expected under 80 percent power, though above values associated with p-hacking. This suggests many RCTs are underpowered. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Source-stated Using an empirical Bayes mixture model that accounts for selective publication, the authors estimate that approximately 30 percent of published rejections are false discoveries and average directional power is 40 percent. > we find that approximately 30 percent of published rejections are false discoveries, with average directional power at only 40 percent. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Source-stated At conventional significance thresholds, the authors estimate a 66 percent posterior probability of a false discovery. > At conventional significance thresholds, the posterior probability of a false discovery reaches 66 percent. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Source-stated The authors report that doubling original sample sizes reduces average false discovery rates by five percentage points; tightening the significance threshold to 0.005 reduces them more, but results in fewer total discoveries. > Doubling original sample sizes lowers the average false discovery rate by five percentage points, whereas tightening the significance threshold to 0.005 yields a larger reduction at the cost of total discoveries. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Source-stated Their model of replication funding under a fixed budget and evidence standard places replications in an intermediate interval of original test statistics; optimal replication sample size is single-peaked in the original t-statistic. > replications concentrate in an intermediate interval of original test statistics, and the optimal replication sample size is single-peaked in the original t-statistic. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Source-stated Illustrative calibrations estimate that successful replications reduce aggregate false discovery rates to between 1.8 and 7.8 percent, depending on the evidence standard. > Illustrative calibrations show that successful replications reduce aggregate false discovery rates to 1.8–7.8 percent, depending on the evidence standard. - [Supporting source](https://marginalrevolution.com/marginalrevolution/2026/11/are-rcts-underpowered.html) ## Source and contributor context Tyler Cowen · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Randomized controlled trial A study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment. - [Definition reference](https://en.wikipedia.org/wiki/Randomized_controlled_trial) ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Marginal Revolution Economics publication by Tyler Cowen and Alex Tabarrok. - [Explore this context](/entities/marginal-revolution) - [Reference](https://marginalrevolution.com/) ## Tyler Cowen Economics, institutions and unexpected cross-domain questions; useful source discovery. - [Explore this context](/entities/tyler-cowen) - [Reference](https://www.dwarkesh.com/p/tyler-cowen-4) - [Reference](https://www.dwarkesh.com/p/tyler-cowen-3) - [Reference](https://www.dwarkesh.com/p/tyler-cowen-2) - [Reference](https://marginalrevolution.com/marginalrevolution/2026/10/saturday-assorted-links-583.html?utm_source=rss&utm_medium=rss&utm_campaign=saturday-assorted-links-583) - [Reference](https://tylercowen.com/) ## Editorial provenance Version 1. Publication ID: 5b8e4b7a-35fd-4280-8dc0-e76c304fabd4. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T21:02:41.306Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: useful\_perspective. Observation: 0d99035d657af4ff01ea8a35623c281493f8654f0a0250c70041dce916211e24. Source text hash: 83baf6db00a1fdc9ea872bb0391dbdc5e52e6b112fec42bfc6631c4088cd1d89. ### uncertain — 2026-11-10T21:02:41.306Z — anthropic/claude-opus-5.5 Publication: 5b8e4b7a-35fd-4280-8dc0-e76c304fabd4 attribution — pass: The draft correctly presents this as a Marginal Revolution post quoting a new paper by Qingyuan Chai, Kevin Lang and Liang Zhong. It attributes the findings to the authors throughout and does not present them as established fact or as Escape Meridian's view. > That is from a new paper by Qingyuan Chai, Kevin Lang & Liang Zhong. summary — uncertain: The draft is faithful but hard for non-specialists to follow. Central terms are never explained in plain language or flagged for tooltips: underpowered, t-statistics, 80 percent power, p-hacking, empirical Bayes mixture model, directional power, posterior probability, regularity conditions and single-peaked. It also drops the 2.8 benchmark and softens 'well below' to 'below'. It does not say why the findings matter, for example that published development-economics results may be less reliable than they appear. The smallest repair is to restore the 2.8 benchmark and 'well below', and to add plain explanations or tooltips for power, false discovery and p-hacking. > actual t-statistics peak well below the theoretical benchmark of 2.8 expected under 80 percent power date — pass: originalPublished (2026-11-10T20:05:26Z, publisher page) matches the feed entry for this Marginal Revolution post, which is the source being read. The post calls the paper new. No earlier first-publication date for the post appears in the evidence. support — pass: Each claim matches the quoted abstract: the 30 percent false discoveries, the 40 percent directional power, the 66 percent posterior probability, the sample-size and threshold comparison, the replication allocation, and the 1.8–7.8 percent calibrations. The limitations accurately reflect the scope and the model-based nature of the replication results. The question follows from the evidence and implies no unsupported benefits or costs. > Using 745 principal-hypothesis tests from 190 articles in leading economics journals media — pass: The selected media is the verified direct link to the original Marginal Revolution post, the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/5c652c1258b9669bdeb39e5a5ec5c425b7175a665c26b8baa588c8e5e238e456/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/marginal-revolution-introduces-a-paper-arguing-many-development-5c652c12.json)