# Sara Amar on MANGO Arena: same seeds, different chains

[Canonical reading](https://escapemeridian.com/readings/sara-amar-on-mango-arena-same-seeds-different-chains-d45533f0)

Sara Amar introduces MANGO Arena, an open-source testbed where four LLM agents run companies in a serial supply chain. It tracks the bullwhip effect—demand variability growing as orders move upstream—as well as rule-following and proposals to collude. In a pilot using three small reasoning models, demand variability grew substantially upstream, often more than under a textbook ordering policy. Rule violations were rare, and no agent proposed a money side deal. But Amar’s central warning is about evaluation: failed API calls were silently counted as “no order,” and even reruns with the same seeds and no failed calls produced large changes in upstream bullwhip. With only three demand seeds, conclusions about a disruption’s effect on a model did not survive rerunning the games. The paper reports these failures and proposes reporting practices for similar evaluations; its code, prompts and logged decisions are public. The pilot therefore illustrates how both system failures and run-to-run variation can shape apparent agent performance, while leaving the stability of findings beyond this small test unresolved.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Nov 17, 2026 (2026-11-17).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-19T07:31:54.748Z. First observed: 2026-11-19T05:04:18.159Z. Retrieved: 2026-11-19T07:30:59.037Z.

- [MANGO Arena: Same Seeds, Different Chains Evaluating LLM Agents in a Disrupted Supply Chain](https://arxiv.org/abs/2611.23613)

## Limits

The findings come from a pilot with three small reasoning models, two scripted reference policies, two conditions, and three demand seeds; they do not establish how results would generalize beyond those tests. The source reports that API failures affected metrics and that reruns varied, so the pilot's performance measurements require caution.


## Open question

How stable are the reported findings when the games are rerun with more demand seeds and without failed API calls?


## Source-stated

Sara Amar introduces MANGO Arena, an open-source testbed where four LLM agents each operate a company in a serial supply chain.

> We introduce MANGO Arena, an open-source testbed in which four LLM agents each run one company in a serial supply chain

- [Supporting source](https://arxiv.org/abs/2611.23613)

## Source-stated

MANGO Arena measures the bullwhip effect, rule adherence, and proposals to collude.

> This lets one environment measure three things at once: an economic outcome (the bullwhip effect), rule adherence under pressure, and proposals to collude.

- [Supporting source](https://arxiv.org/abs/2611.23613)

## Source-stated

In a pilot, all LLM agents amplified demand variability upstream, often more than a textbook order-up-to policy.

> all LLM agents amplified demand variability as it moved upstream, by 7 to 25 times at the Retailer and 13 to 179 times at the Supplier, often more than a textbook order-up-to policy.

- [Supporting source](https://arxiv.org/abs/2611.23613)

## Source-stated

The pilot recorded rare rule violations and no proposed money side deals.

> Real rule violations were rare (at most 7 of 288 decisions per model) and no agent proposed a money side deal.

- [Supporting source](https://arxiv.org/abs/2611.23613)

## Source-stated

The authors report that API failures silently counted as “no order” decisions, while upstream bullwhip varied substantially between reruns.

> API failures (rate limits and runaway hidden reasoning) entered the metrics silently as "no order" decisions. And when we reran games with the same seeds and no failed calls, upstream bullwhip changed by up to 17 times between runs.

- [Supporting source](https://arxiv.org/abs/2611.23613)

## Source-stated

The authors say that conclusions about disruption effects did not survive reruns with only three demand seeds.

> With three seeds, conclusions about how a disruption affects a given model did not survive a rerun.

- [Supporting source](https://arxiv.org/abs/2611.23613)

## Source and contributor context

Amar, Sara · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Reasoning model

A language model trained to work through a problem in written steps before it answers.

- [Definition reference](https://en.wikipedia.org/wiki/Reasoning_model)

## API

An application programming interface: a defined way for software to request data or actions from another system.

- [Definition reference](https://developer.mozilla.org/en-US/docs/Glossary/API)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: d634675f-1fbf-495f-85b0-6b3e2e387e41.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-19T07:31:54.748Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: b0360fabe99f6a6fc79cc864523f390be055f91c1040a7d0f22d74f238494da5. Source text hash: 052df621fa86eee29d87232eaf08e8e210208187a9281be3e6071840ae2b4950.


## Revision and source-check history

### Version 1 — 2026-11-19T07:31:54.748Z

Sara Amar introduces MANGO Arena, an open-source testbed where four LLM agents run companies in a serial supply chain. It tracks the bullwhip effect—demand variability growing as orders move upstream—as well as rule-following and proposals to collude. In a pilot using three small reasoning models, demand variability grew substantially upstream, often more than under a textbook ordering policy. Rule violations were rare, and no agent proposed a money side deal. But Amar’s central warning is about evaluation: failed API calls were silently counted as “no order,” and even reruns with the same seeds and no failed calls produced large changes in upstream bullwhip. With only three demand seeds, conclusions about a disruption’s effect on a model did not survive rerunning the games. The paper reports these failures and proposes reporting practices for similar evaluations; its code, prompts and logged decisions are public. The pilot therefore illustrates how both system failures and run-to-run variation can shape apparent agent performance, while leaving the stability of findings beyond this small test unresolved.

## Original source

arXiv cs.AI · Article

Original source date: Nov 17, 2026 (2026-11-17).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-19T07:31:54.748Z. First observed: 2026-11-19T05:04:18.159Z. Retrieved: 2026-11-19T07:30:59.037Z.

- [MANGO Arena: Same Seeds, Different Chains Evaluating LLM Agents in a Disrupted Supply Chain](https://arxiv.org/abs/2611.23613)

## Limits

The findings come from a pilot with three small reasoning models, two scripted reference policies, two conditions, and three demand seeds; they do not establish how results would generalize beyond those tests. The source reports that API failures affected metrics and that reruns varied, so the pilot's performance measurements require caution.


## Open question

How stable are the reported findings when the games are rerun with more demand seeds and without failed API calls?


## Source-stated

Sara Amar introduces MANGO Arena, an open-source testbed where four LLM agents each operate a company in a serial supply chain.

> We introduce MANGO Arena, an open-source testbed in which four LLM agents each run one company in a serial supply chain

- [Supporting source](https://arxiv.org/abs/2611.23613)

## Source-stated

MANGO Arena measures the bullwhip effect, rule adherence, and proposals to collude.

> This lets one environment measure three things at once: an economic outcome (the bullwhip effect), rule adherence under pressure, and proposals to collude.

- [Supporting source](https://arxiv.org/abs/2611.23613)

## Source-stated

In a pilot, all LLM agents amplified demand variability upstream, often more than a textbook order-up-to policy.

> all LLM agents amplified demand variability as it moved upstream, by 7 to 25 times at the Retailer and 13 to 179 times at the Supplier, often more than a textbook order-up-to policy.

- [Supporting source](https://arxiv.org/abs/2611.23613)

## Source-stated

The pilot recorded rare rule violations and no proposed money side deals.

> Real rule violations were rare (at most 7 of 288 decisions per model) and no agent proposed a money side deal.

- [Supporting source](https://arxiv.org/abs/2611.23613)

## Source-stated

The authors report that API failures silently counted as “no order” decisions, while upstream bullwhip varied substantially between reruns.

> API failures (rate limits and runaway hidden reasoning) entered the metrics silently as "no order" decisions. And when we reran games with the same seeds and no failed calls, upstream bullwhip changed by up to 17 times between runs.

- [Supporting source](https://arxiv.org/abs/2611.23613)

## Source-stated

The authors say that conclusions about disruption effects did not survive reruns with only three demand seeds.

> With three seeds, conclusions about how a disruption affects a given model did not survive a rerun.

- [Supporting source](https://arxiv.org/abs/2611.23613)

## Source and contributor context

Amar, Sara · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Reasoning model

A language model trained to work through a problem in written steps before it answers.

- [Definition reference](https://en.wikipedia.org/wiki/Reasoning_model)

## API

An application programming interface: a defined way for software to request data or actions from another system.

- [Definition reference](https://developer.mozilla.org/en-US/docs/Glossary/API)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: d634675f-1fbf-495f-85b0-6b3e2e387e41.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-19T07:31:54.748Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: b0360fabe99f6a6fc79cc864523f390be055f91c1040a7d0f22d74f238494da5. Source text hash: 052df621fa86eee29d87232eaf08e8e210208187a9281be3e6071840ae2b4950.


### uncertain — 2026-11-19T07:31:54.748Z — anthropic/claude-opus-5.5

Publication: d634675f-1fbf-495f-85b0-6b3e2e387e41

attribution — pass: The paper is correctly attributed to Sara Amar, its sole listed author, on arXiv. The claims say 'the authors,' while the summary correctly uses Amar. This is a minor wording inconsistency and does not misattribute any claim.

> Authors:Sara Amar

summary — uncertain: The summary is faithful and keeps the paper's central point about evaluation method. Two gaps limit understanding for non-specialists. First, 'seeds' and 'demand seeds' are never explained, yet the main lesson depends on them: the same fixed demand scenarios, rerun, gave very different results. Second, the 'disruption' is never identified as a supplier fire, so 'a disruption's effect' lacks context. Smallest repair: add a short gloss on seeds and name the supplier-fire condition. 'Order-up-to policy' could also use a term tooltip.

> two conditions (with and without a supplier fire) and three demand seeds

date — pass: The original publication date of 2026-11-17 matches the v1 submission date. The feed date of Nov 19 is correctly not used.

> \[Submitted on 17 Nov 2026\]

support — pass: Every claim is directly supported by the abstract. The closing interpretation, that system failures and run-to-run variation can shape apparent performance, follows from what the source reports. The limitations and the question are framed appropriately and do not turn the pilot into a general finding.

> With three seeds, conclusions about how a disruption affects a given model did not survive a rerun.

media — pass: The selected media is the verified arXiv abstract page itself, which is the direct original source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/d45533f08a97e3dff134741b71050f3dd1e7ae732439d6fccf7b5f71815dff48/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/sara-amar-on-mango-arena-same-seeds-different-chains-d45533f0.json)
