Alcobaça, Edesio, Rossell, Kevin, Gupta, Aman, Tang, Shao, Hong, Jiwoo, Carrillo-Mendoza, Pabel, Ferreira, Wanderson Conceição, Tedeschi, Alvaro, Simjee, Zayd, Rajpal, Shreya, Hime, Bruno Finardi, Sousa, Christian, Moneda, Luis, Fei, Herbert, Silva, Daniel, Ramanath, Rohan · Sep 24, 2026
Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale
Markdown · JSON · Original source
The paper’s authors argue that customer experience agents can be improved by testing them in simulation before exposing customers to failures in live experiments. Their workflow uses synthetic customers and simulated tool outputs to run multi-step without calling production backends. They tested it on Nubank’s Card Delivery agent and its expanded successor, Card Management, the company’s highest-volume chat-support agent in Brazil. Across four deployed versions, simulated and production version-level binary evaluator scores showed high correlation. In a live A/B test, simulation-guided iteration increased transactional net promoter score by 36.69 points. The authors also screened configurations in more than 16,000 simulated conversations. In a subsequent live A/B test, the selected model raised self-service rate by 8.82 percentage points to Nubank’s highest observed level, with no change in transactional net promoter score. The authors’ central case is that simulation allows broad exploration of models, reasoning settings, and prompts without customer exposure, making production improvements feasible that live experimentation alone would have made impractical. The evidence described here comes from one company’s agents and tests; how well the workflow transfers to other settings remains open.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Sep 24, 2026 (2026-09-24).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T01:08:29.521Z. First observed: 2026-09-25T08:18:34.892Z. Retrieved: 2026-11-09T01:07:41.525Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The evidence described in the abstract concerns Nubank agents and reported tests; it does not establish how well the workflow transfers to other organizations or settings.
Open question
How well do simulated version-level binary evaluator scores predict production scores beyond the four deployed versions tested?
Source-stated
The authors propose a simulation workflow that screens candidate customer experience agents before deployment, using synthetic customers and simulated tool outputs.
Synthetic customers react to agent responses and simulated tool outputs enable multi-step agentic workflows without invoking production backends.
Source-stated
Across four deployed versions, simulated and production version-level binary evaluator scores showed high correlation.
Across 4 deployed versions, simulated and production version-level binary evaluator scores show high correlation.
Source-stated
Simulation-guided iteration increased transactional net promoter score by 36.69 points in a live A/B test.
Simulation-guided iteration increased transactional net promoter score (tNPS) by 36.69 points in a live A/B test.
Source-stated
After screening open-weight configurations in over 16,000 simulated conversations, the selected model increased self-service rate by 8.82 percentage points in a live A/B test, with no statistically significant change in transactional net promoter score.
In a subsequent live A/B test, the selected model increased self-service rate (SSR) by 8.82 percentage points to the highest level observed at Nubank, with no statistically significant change in tNPS.
Source-stated
The reported tests concern Nubank’s Card Delivery agent and Card Management, its highest-volume chat-support agent in Brazil.
We use the Snowglobe simulator on Nubank's Card Delivery agent and its expanded successor, Card Management - Nubank's highest-volume chat-support agent in Brazil.
Source and contributor context
Alcobaça, Edesio, Rossell, Kevin, Gupta, Aman, Tang, Shao, Hong, Jiwoo, Carrillo-Mendoza, Pabel, Ferreira, Wanderson Conceição, Tedeschi, Alvaro, Simjee, Zayd, Rajpal, Shreya, Hime, Bruno Finardi, Sousa, Christian, Moneda, Luis, Fei, Herbert, Silva, Daniel, Ramanath, Rohan · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Agentic AI
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Open weights
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Statistical significance
A test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: b4c4b22d-e729-45d8-85fa-53124fe15f01.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T01:08:29.521Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 7f1121e5f0573417acb71755ba6c61bff8ebe1bc682db2ee93f5dc5bec86d85f. Source text hash: bbde15cb15f8c2b97130b9512aec35239e1084f2b3d94d0819a2265443fbf91f.
Revision and source-check history
Version 1 · 2026-11-09T01:08:29.521Z
Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale
The paper’s authors argue that customer experience agents can be improved by testing them in simulation before exposing customers to failures in live experiments. Their workflow uses synthetic customers and simulated tool outputs to run multi-step without calling production backends. They tested it on Nubank’s Card Delivery agent and its expanded successor, Card Management, the company’s highest-volume chat-support agent in Brazil. Across four deployed versions, simulated and production version-level binary evaluator scores showed high correlation. In a live A/B test, simulation-guided iteration increased transactional net promoter score by 36.69 points. The authors also screened configurations in more than 16,000 simulated conversations. In a subsequent live A/B test, the selected model raised self-service rate by 8.82 percentage points to Nubank’s highest observed level, with no change in transactional net promoter score. The authors’ central case is that simulation allows broad exploration of models, reasoning settings, and prompts without customer exposure, making production improvements feasible that live experimentation alone would have made impractical. The evidence described here comes from one company’s agents and tests; how well the workflow transfers to other settings remains open.
Original source
cs.AI · Article
Source details
Original source date: Sep 24, 2026 (2026-09-24).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-09T01:08:29.521Z. First observed: 2026-09-25T08:18:34.892Z. Retrieved: 2026-11-09T01:07:41.525Z.
Limits
The evidence described in the abstract concerns Nubank agents and reported tests; it does not establish how well the workflow transfers to other organizations or settings.
Open question
How well do simulated version-level binary evaluator scores predict production scores beyond the four deployed versions tested?
Source-stated
The authors propose a simulation workflow that screens candidate customer experience agents before deployment, using synthetic customers and simulated tool outputs.
Synthetic customers react to agent responses and simulated tool outputs enable multi-step agentic workflows without invoking production backends.
Source-stated
Across four deployed versions, simulated and production version-level binary evaluator scores showed high correlation.
Across 4 deployed versions, simulated and production version-level binary evaluator scores show high correlation.
Source-stated
Simulation-guided iteration increased transactional net promoter score by 36.69 points in a live A/B test.
Simulation-guided iteration increased transactional net promoter score (tNPS) by 36.69 points in a live A/B test.
Source-stated
After screening open-weight configurations in over 16,000 simulated conversations, the selected model increased self-service rate by 8.82 percentage points in a live A/B test, with no statistically significant change in transactional net promoter score.
In a subsequent live A/B test, the selected model increased self-service rate (SSR) by 8.82 percentage points to the highest level observed at Nubank, with no statistically significant change in tNPS.
Source-stated
The reported tests concern Nubank’s Card Delivery agent and Card Management, its highest-volume chat-support agent in Brazil.
We use the Snowglobe simulator on Nubank's Card Delivery agent and its expanded successor, Card Management - Nubank's highest-volume chat-support agent in Brazil.
Source and contributor context
Alcobaça, Edesio, Rossell, Kevin, Gupta, Aman, Tang, Shao, Hong, Jiwoo, Carrillo-Mendoza, Pabel, Ferreira, Wanderson Conceição, Tedeschi, Alvaro, Simjee, Zayd, Rajpal, Shreya, Hime, Bruno Finardi, Sousa, Christian, Moneda, Luis, Fei, Herbert, Silva, Daniel, Ramanath, Rohan · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Agentic AI
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Open weights
A model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Statistical significance
A test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: b4c4b22d-e729-45d8-85fa-53124fe15f01.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T01:08:29.521Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 7f1121e5f0573417acb71755ba6c61bff8ebe1bc682db2ee93f5dc5bec86d85f. Source text hash: bbde15cb15f8c2b97130b9512aec35239e1084f2b3d94d0819a2265443fbf91f.
passed · 2026-11-09T01:08:29.521Z · anthropic/claude-opus-5.5
Publication: b4c4b22d-e729-45d8-85fa-53124fe15f01
attribution — pass: The draft consistently attributes the workflow, results, and central case to the paper's authors. It presents the reported correlations and A/B-test gains as the authors' findings rather than as independently verified facts. The arXiv abstract page supports this framing.
We present a hypothesis-driven simulation workflow for screening candidate CX agents before deployment.
summary — pass: The draft leads with the authors' argument and names Nubank and its agents. It explains the mechanism (synthetic customers, simulated tool outputs, no production backends) and the stakes (avoiding customer-facing failures). The results and the open transfer question follow. Terms such as tNPS, open-weight, self-service rate and binary evaluator scores are kept exactly; tooltip coverage for them is a suggested addition. The question follows from the four-version correlation evidence.
live experiments expose customers to failures that can erode trust.
date — pass: The originalPublished date of 2026-09-24 matches the v1 submission date. It does not use the 5 Nov 2026 v2 revision or the 6 Nov feed date.
[Submitted on 24 Sep 2026 (v1), last revised 5 Nov 2026 (this version, v2)]
support — pass: Every claim is tied to a verbatim excerpt that matches the abstract, including the 36.69 tNPS gain and the 8.82 point SSR increase with no significant tNPS change. The scope limitation and the question on generalizing beyond four versions are reasonable, explicitly open framings. They do not overstate the evidence or invent costs or benefits.
Simulation made broad exploration of models, reasoning settings, and prompts feasible without customer exposure
media — pass: The selected media is the verified direct link to the arXiv abstract page, which is the original source. It is the most direct access point.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Definition referenceA model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Definition referenceA test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivAI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Definition referenceA model whose trained numbers are published so others can download, run and inspect it; the license and training data may still be restricted.
Definition referenceA test of whether a result is unlikely to be due to chance alone; it does not say how large or important an effect is.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXiv