· Nov 10, 2026
Marginal Revolution introduces a paper arguing many development-economics RCTs are underpowered
Markdown · JSON · Original source
introduces a paper by Qingyuan Chai, Kevin Lang and Liang Zhong on the statistical credibility of (RCTs) in development economics. The authors report that actual t-statistics peak below the theoretical expected under 80 percent power, but above values associated with p-hacking; they say this suggests many RCTs are underpowered. Using an empirical Bayes mixture model that accounts for selective publication, they estimate that approximately 30 percent of published rejections are false discoveries, with average directional power of 40 percent. At conventional significance thresholds, they estimate a 66 percent posterior probability of a false discovery. The paper compares two changes: doubling original sample sizes reduces the average false discovery rate by five percentage points, while tightening the significance threshold to 0.005 yields a larger reduction at the cost of total discoveries. It also models replication funding under a fixed budget and evidence standard. Under standard regularity conditions, the authors’ model allocates replications to an intermediate interval of original test statistics, with optimal replication sample size single-peaked in the original t-statistic.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Nov 10, 2026 (2026-11-10T20:05:26+00:00).
Date basis: Date stated by the publisher.
Added to Escape Meridian: 2026-11-10T21:02:41.306Z. First observed: 2026-11-10T20:09:23.668Z. Retrieved: 2026-11-10T21:01:10.873Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The analysis described is limited to 745 principal-hypothesis tests from 190 articles in leading economics journals, in development economics. The source presents the replication-policy results as a model under standard regularity conditions; its reported post-replication rates are illustrative calibrations. The source does not specify the evidence standards behind the 1.8–7.8 percent calibration range.
Open question
How do the paper’s replication-policy results vary across evidence standards?
Source-stated
The authors report that t-statistics in their sample peak below the theoretical benchmark expected under 80 percent power, which they say suggests many RCTs are underpowered.
actual t-statistics peak well below the theoretical benchmark of 2.8 expected under 80 percent power, though above values associated with p-hacking. This suggests many RCTs are underpowered.
Source-stated
Using an empirical Bayes mixture model that accounts for selective publication, the authors estimate that approximately 30 percent of published rejections are false discoveries and average directional power is 40 percent.
we find that approximately 30 percent of published rejections are false discoveries, with average directional power at only 40 percent.
Source-stated
At conventional significance thresholds, the authors estimate a 66 percent posterior probability of a false discovery.
At conventional significance thresholds, the posterior probability of a false discovery reaches 66 percent.
Source-stated
The authors report that doubling original sample sizes reduces average false discovery rates by five percentage points; tightening the significance threshold to 0.005 reduces them more, but results in fewer total discoveries.
Doubling original sample sizes lowers the average false discovery rate by five percentage points, whereas tightening the significance threshold to 0.005 yields a larger reduction at the cost of total discoveries.
Source-stated
Their model of replication funding under a fixed budget and evidence standard places replications in an intermediate interval of original test statistics; optimal replication sample size is single-peaked in the original t-statistic.
replications concentrate in an intermediate interval of original test statistics, and the optimal replication sample size is single-peaked in the original t-statistic.
Source-stated
Illustrative calibrations estimate that successful replications reduce aggregate false discovery rates to between 1.8 and 7.8 percent, depending on the evidence standard.
Illustrative calibrations show that successful replications reduce aggregate false discovery rates to 1.8–7.8 percent, depending on the evidence standard.
Source and contributor context
Tyler Cowen · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Randomized controlled trial
A study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Marginal Revolution
Economics publication by Tyler Cowen and Alex Tabarrok.
Tyler Cowen
Economics, institutions and unexpected cross-domain questions; useful source discovery.
Editorial provenance
Version 1. Publication ID: 5b8e4b7a-35fd-4280-8dc0-e76c304fabd4.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T21:02:41.306Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: useful_perspective.
Observation: 0d99035d657af4ff01ea8a35623c281493f8654f0a0250c70041dce916211e24. Source text hash: 83baf6db00a1fdc9ea872bb0391dbdc5e52e6b112fec42bfc6631c4088cd1d89.
Revision and source-check history
Version 1 · 2026-11-10T21:02:41.306Z
Marginal Revolution introduces a paper arguing many development-economics RCTs are underpowered
introduces a paper by Qingyuan Chai, Kevin Lang and Liang Zhong on the statistical credibility of (RCTs) in development economics. The authors report that actual t-statistics peak below the theoretical expected under 80 percent power, but above values associated with p-hacking; they say this suggests many RCTs are underpowered. Using an empirical Bayes mixture model that accounts for selective publication, they estimate that approximately 30 percent of published rejections are false discoveries, with average directional power of 40 percent. At conventional significance thresholds, they estimate a 66 percent posterior probability of a false discovery. The paper compares two changes: doubling original sample sizes reduces the average false discovery rate by five percentage points, while tightening the significance threshold to 0.005 yields a larger reduction at the cost of total discoveries. It also models replication funding under a fixed budget and evidence standard. Under standard regularity conditions, the authors’ model allocates replications to an intermediate interval of original test statistics, with optimal replication sample size single-peaked in the original t-statistic.
Original source
· Article
Source details
Original source date: Nov 10, 2026 (2026-11-10T20:05:26+00:00).
Date basis: Date stated by the publisher.
Added to Escape Meridian: 2026-11-10T21:02:41.306Z. First observed: 2026-11-10T20:09:23.668Z. Retrieved: 2026-11-10T21:01:10.873Z.
Limits
The analysis described is limited to 745 principal-hypothesis tests from 190 articles in leading economics journals, in development economics. The source presents the replication-policy results as a model under standard regularity conditions; its reported post-replication rates are illustrative calibrations. The source does not specify the evidence standards behind the 1.8–7.8 percent calibration range.
Open question
How do the paper’s replication-policy results vary across evidence standards?
Source-stated
The authors report that t-statistics in their sample peak below the theoretical benchmark expected under 80 percent power, which they say suggests many RCTs are underpowered.
actual t-statistics peak well below the theoretical benchmark of 2.8 expected under 80 percent power, though above values associated with p-hacking. This suggests many RCTs are underpowered.
Source-stated
Using an empirical Bayes mixture model that accounts for selective publication, the authors estimate that approximately 30 percent of published rejections are false discoveries and average directional power is 40 percent.
we find that approximately 30 percent of published rejections are false discoveries, with average directional power at only 40 percent.
Source-stated
At conventional significance thresholds, the authors estimate a 66 percent posterior probability of a false discovery.
At conventional significance thresholds, the posterior probability of a false discovery reaches 66 percent.
Source-stated
The authors report that doubling original sample sizes reduces average false discovery rates by five percentage points; tightening the significance threshold to 0.005 reduces them more, but results in fewer total discoveries.
Doubling original sample sizes lowers the average false discovery rate by five percentage points, whereas tightening the significance threshold to 0.005 yields a larger reduction at the cost of total discoveries.
Source-stated
Their model of replication funding under a fixed budget and evidence standard places replications in an intermediate interval of original test statistics; optimal replication sample size is single-peaked in the original t-statistic.
replications concentrate in an intermediate interval of original test statistics, and the optimal replication sample size is single-peaked in the original t-statistic.
Source-stated
Illustrative calibrations estimate that successful replications reduce aggregate false discovery rates to between 1.8 and 7.8 percent, depending on the evidence standard.
Illustrative calibrations show that successful replications reduce aggregate false discovery rates to 1.8–7.8 percent, depending on the evidence standard.
Source and contributor context
Tyler Cowen · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Randomized controlled trial
A study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Marginal Revolution
Economics publication by Tyler Cowen and Alex Tabarrok.
Tyler Cowen
Economics, institutions and unexpected cross-domain questions; useful source discovery.
Editorial provenance
Version 1. Publication ID: 5b8e4b7a-35fd-4280-8dc0-e76c304fabd4.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T21:02:41.306Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: useful_perspective.
Observation: 0d99035d657af4ff01ea8a35623c281493f8654f0a0250c70041dce916211e24. Source text hash: 83baf6db00a1fdc9ea872bb0391dbdc5e52e6b112fec42bfc6631c4088cd1d89.
uncertain · 2026-11-10T21:02:41.306Z · anthropic/claude-opus-5.5
Publication: 5b8e4b7a-35fd-4280-8dc0-e76c304fabd4
attribution — pass: The draft correctly presents this as a Marginal Revolution post quoting a new paper by Qingyuan Chai, Kevin Lang and Liang Zhong. It attributes the findings to the authors throughout and does not present them as established fact or as Escape Meridian's view.
That is from a new paper by Qingyuan Chai, Kevin Lang & Liang Zhong.
summary — uncertain: The draft is faithful but hard for non-specialists to follow. Central terms are never explained in plain language or flagged for tooltips: underpowered, t-statistics, 80 percent power, p-hacking, empirical Bayes mixture model, directional power, posterior probability, regularity conditions and single-peaked. It also drops the 2.8 benchmark and softens 'well below' to 'below'. It does not say why the findings matter, for example that published development-economics results may be less reliable than they appear. The smallest repair is to restore the 2.8 benchmark and 'well below', and to add plain explanations or tooltips for power, false discovery and p-hacking.
actual t-statistics peak well below the theoretical benchmark of 2.8 expected under 80 percent power
date — pass: originalPublished (2026-11-10T20:05:26Z, publisher page) matches the feed entry for this Marginal Revolution post, which is the source being read. The post calls the paper new. No earlier first-publication date for the post appears in the evidence.
support — pass: Each claim matches the quoted abstract: the 30 percent false discoveries, the 40 percent directional power, the 66 percent posterior probability, the sample-size and threshold comparison, the replication allocation, and the 1.8–7.8 percent calibrations. The limitations accurately reflect the scope and the model-based nature of the replication results. The question follows from the evidence and implies no unsupported benefits or costs.
Using 745 principal-hypothesis tests from 190 articles in leading economics journals
media — pass: The selected media is the verified direct link to the original Marginal Revolution post, the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Economics, institutions and unexpected cross-domain questions; useful source discovery.
Explore Tyler CowenEconomics publication by Tyler Cowen and Alex Tabarrok.
Explore Marginal RevolutionA study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceEconomics publication by Tyler Cowen and Alex Tabarrok.
Explore Marginal RevolutionEconomics publication by Tyler Cowen and Alex Tabarrok.
Explore Marginal RevolutionA study that randomly assigns people to groups so that differences in outcomes can be traced to the treatment.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceEconomics publication by Tyler Cowen and Alex Tabarrok.
Explore Marginal Revolution