# arXiv: When AI Analysts Disagree: Agentic Multiverse Analysis and the Robustness of Empirical Research

[Canonical reading](https://escapemeridian.com/readings/arxiv-when-ai-analysts-disagree-agentic-multiverse-analysis-and-d32a9f42)

Authors Ruijiang Gao and Steven Chong Xiao argue that agentic multiverse analysis (AMA) can make hidden analytical choices visible and help human experts assess how robust empirical findings are. They call those choices “forks”: defensible ways an analyst might conduct an analysis. In a pilot using one dataset and one set of research questions, 150 AI coding agents disagreed through methodological choices also made by human analysts, though across a narrower range. AI peer review barely changed their disagreement; exposure to highly rated example reports sharply reduced it, largely through imitation rather than analysis. The authors’ framework has agents map a paper’s possible forks, assess them against a validity rubric, run the validated specifications, and check one another’s work under separated duties. Across ten studies, it reproduced the original estimates and found that some conclusions held across all validated specifications, while others changed with individual choices the papers did not discuss. The authors suggest these assessments could act as robustness proxies. They report that the work takes hours of agent time and tens of dollars per paper, and say AI makes multiverse analysis practical where few research teams could previously afford it.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Nov 18, 2026 (2026-11-18).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-20T06:16:39.097Z. First observed: 2026-11-20T05:01:57.867Z. Retrieved: 2026-11-20T06:15:51.610Z.

- [When AI Analysts Disagree: Agentic Multiverse Analysis and the Robustness of Empirical Research](https://arxiv.org/abs/2611.25198)

## Limits

The pilot uses one dataset and one set of research questions; its findings do not establish how AI agents would disagree across other settings. The AMA case studies cover ten empirical studies from one leading management journal’s replication packages; the abstract does not establish how well the results generalize beyond them. The authors describe AMA-based measures as possible robustness proxies, not as established measures of real-world reliability.


## Open question

How well would AMA’s robustness assessments hold up when applied to empirical studies beyond the ten in these case studies?


## Source-stated

The authors propose AMA as a way to expose how defensible analytical choices affect empirical conclusions, for human experts’ decision support.

> We develop an agentic multiverse analysis (AMA) framework that makes a conclusion's sensitivity to those choices visible at scale, and we position it as a decision-support instrument for human experts.

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source-stated

In a pilot involving one dataset and one set of research questions, the AI agents disagreed over defensible methodological choices, but across a narrower range than human analysts.

> A pilot study first deploys 150 autonomous AI coding agents on one dataset and one set of research questions and finds that they disagree through the same defensible methodological choices human analysts make, though over a narrower range.

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source-stated

The pilot found that AI peer review scarcely changed disagreement, while exposure to highly rated example reports reduced it largely through imitation.

> AI peer review leaves their dispersion almost unchanged. Exposure to top-rated exemplar reports then collapses it, largely through imitation rather than analysis.

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source-stated

AMA maps possible analytical choices, assesses them, runs validated specifications, and has agents cross-examine one another under separated duties.

> AMA has agents enumerate a published paper's possible fork space, score every fork against a validity rubric, execute the validated specification grid, and cross-examine one another's work under strict separation of duties.

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source-stated

Across ten studies, the framework reproduced the original estimates; some conclusions held across all validated specifications, while others were sensitive to individual choices not discussed in the papers.

> Applied to ten empirical studies from a leading management journal's replication packages (1,755 specification estimates), the framework reproduces every paper's original estimates, and shows that some conclusions hold across every specification the assessment validates, while others are sensitive

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source-stated

The authors suggest AMA-based measures could serve as proxies for the robustness of empirical findings.

> The case studies suggest our AMA-based measures can serve as robustness proxies for empirical findings.

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source-stated

The authors report that their exercises take hours of agent time and tens of dollars per paper.

> our exercises take hours of agent time and tens of dollars per paper.

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source and contributor context

Gao, Ruijiang, Xiao, Steven Chong · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Agentic AI

AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.

- [Definition reference](https://developers.google.com/machine-learning/glossary#agentic)

## AI agent

An AI system that chooses and carries out actions toward a goal, often using software tools.

- [Definition reference](https://www.ibm.com/think/topics/ai-agents)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

- [Explore this context](/entities/time)
- [Reference](https://time.com/)

## Editorial provenance

Version 1. Publication ID: 52dbae03-1899-44c4-8b8f-05d52d6d0149.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-20T06:16:39.097Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 892195c2a69a6c9da479ef1035a890cc253b0634f230998ca1de91a4bd8fe5a2. Source text hash: decebb36513cbe26b734c274df8c1212db7acee176071d5ecd5cc5cd0bea8fff.


## Revision and source-check history

### Version 1 — 2026-11-20T06:16:39.097Z

Authors Ruijiang Gao and Steven Chong Xiao argue that agentic multiverse analysis (AMA) can make hidden analytical choices visible and help human experts assess how robust empirical findings are. They call those choices “forks”: defensible ways an analyst might conduct an analysis. In a pilot using one dataset and one set of research questions, 150 AI coding agents disagreed through methodological choices also made by human analysts, though across a narrower range. AI peer review barely changed their disagreement; exposure to highly rated example reports sharply reduced it, largely through imitation rather than analysis. The authors’ framework has agents map a paper’s possible forks, assess them against a validity rubric, run the validated specifications, and check one another’s work under separated duties. Across ten studies, it reproduced the original estimates and found that some conclusions held across all validated specifications, while others changed with individual choices the papers did not discuss. The authors suggest these assessments could act as robustness proxies. They report that the work takes hours of agent time and tens of dollars per paper, and say AI makes multiverse analysis practical where few research teams could previously afford it.

## Original source

arXiv cs.AI · Article

Original source date: Nov 18, 2026 (2026-11-18).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-20T06:16:39.097Z. First observed: 2026-11-20T05:01:57.867Z. Retrieved: 2026-11-20T06:15:51.610Z.

- [When AI Analysts Disagree: Agentic Multiverse Analysis and the Robustness of Empirical Research](https://arxiv.org/abs/2611.25198)

## Limits

The pilot uses one dataset and one set of research questions; its findings do not establish how AI agents would disagree across other settings. The AMA case studies cover ten empirical studies from one leading management journal’s replication packages; the abstract does not establish how well the results generalize beyond them. The authors describe AMA-based measures as possible robustness proxies, not as established measures of real-world reliability.


## Open question

How well would AMA’s robustness assessments hold up when applied to empirical studies beyond the ten in these case studies?


## Source-stated

The authors propose AMA as a way to expose how defensible analytical choices affect empirical conclusions, for human experts’ decision support.

> We develop an agentic multiverse analysis (AMA) framework that makes a conclusion's sensitivity to those choices visible at scale, and we position it as a decision-support instrument for human experts.

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source-stated

In a pilot involving one dataset and one set of research questions, the AI agents disagreed over defensible methodological choices, but across a narrower range than human analysts.

> A pilot study first deploys 150 autonomous AI coding agents on one dataset and one set of research questions and finds that they disagree through the same defensible methodological choices human analysts make, though over a narrower range.

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source-stated

The pilot found that AI peer review scarcely changed disagreement, while exposure to highly rated example reports reduced it largely through imitation.

> AI peer review leaves their dispersion almost unchanged. Exposure to top-rated exemplar reports then collapses it, largely through imitation rather than analysis.

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source-stated

AMA maps possible analytical choices, assesses them, runs validated specifications, and has agents cross-examine one another under separated duties.

> AMA has agents enumerate a published paper's possible fork space, score every fork against a validity rubric, execute the validated specification grid, and cross-examine one another's work under strict separation of duties.

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source-stated

Across ten studies, the framework reproduced the original estimates; some conclusions held across all validated specifications, while others were sensitive to individual choices not discussed in the papers.

> Applied to ten empirical studies from a leading management journal's replication packages (1,755 specification estimates), the framework reproduces every paper's original estimates, and shows that some conclusions hold across every specification the assessment validates, while others are sensitive

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source-stated

The authors suggest AMA-based measures could serve as proxies for the robustness of empirical findings.

> The case studies suggest our AMA-based measures can serve as robustness proxies for empirical findings.

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source-stated

The authors report that their exercises take hours of agent time and tens of dollars per paper.

> our exercises take hours of agent time and tens of dollars per paper.

- [Supporting source](https://arxiv.org/abs/2611.25198)

## Source and contributor context

Gao, Ruijiang, Xiao, Steven Chong · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Agentic AI

AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.

- [Definition reference](https://developers.google.com/machine-learning/glossary#agentic)

## AI agent

An AI system that chooses and carries out actions toward a goal, often using software tools.

- [Definition reference](https://www.ibm.com/think/topics/ai-agents)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## TIME

TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.

- [Explore this context](/entities/time)
- [Reference](https://time.com/)

## Editorial provenance

Version 1. Publication ID: 52dbae03-1899-44c4-8b8f-05d52d6d0149.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-20T06:16:39.097Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 892195c2a69a6c9da479ef1035a890cc253b0634f230998ca1de91a4bd8fe5a2. Source text hash: decebb36513cbe26b734c274df8c1212db7acee176071d5ecd5cc5cd0bea8fff.


### passed — 2026-11-20T06:16:39.097Z — anthropic/claude-opus-5.5

Publication: 52dbae03-1899-44c4-8b8f-05d52d6d0149

attribution — pass: The reading correctly credits Ruijiang Gao and Steven Chong Xiao. It frames the framework, findings, proxy suggestion and cost figures as the authors' claims and reports, not as independently established facts.

> Authors:Ruijiang Gao, Steven Chong Xiao

summary — pass: The summary faithfully follows the abstract: forks, the 150-agent pilot, peer review versus exemplar imitation, the AMA steps, the ten-study results and the cost claim. Forks are defined in plain language, and AMA's purpose is clear enough for a general reader. The question about generalization follows from the evidence. One small optional addition would be the authors' stated implications for replication policy and AI-assisted peer review; the reading does not need it.

> Every empirical analysis on observational data rests on dozens of defensible analytical choices ("forks") that its readers cannot see.

date — pass: The displayed originalPublished date of 2026-11-18 matches this arXiv entry's v1 submission date. The feed date of Nov 20 is later. The note that this paper subsumes an earlier arXiv ID does not change the first publication date of this entry.

> \[Submitted on 18 Nov 2026\]

support — pass: Each claim is supported by a verbatim abstract excerpt, and wording such as 'sharply reduced' fairly renders 'collapses'. The limitations correctly confine the pilot to one dataset and the case studies to one journal's replication packages. They also keep robustness proxies as a suggestion rather than proven reliability. The reading invents no beneficiaries, costs or forecasts.

> The case studies suggest our AMA-based measures can serve as robustness proxies for empirical findings.

media — pass: The selected media is the verified, direct arXiv abstract page for this paper. It is the most direct way to access the source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/d32a9f42dff07eb6175e09013e0369cd0d4391b05c71e7d88ef2296e4e159e23/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/arxiv-when-ai-analysts-disagree-agentic-multiverse-analysis-and-d32a9f42.json)
