Escape Meridian

Ziftci, Celal, Liu, Ray, Greene, Spencer, Dalloro, Livio · Apr 13, 2026

Google paper reports LLM-based diagnosis of integration test failures

Markdown · JSON · Original source

Authors Celal Ziftci, Ray Liu, Spencer Greene and Livio Dalloro describe Auto-Diagnose, a tool that analyzes integration test failure logs and provides concise summaries with relevant log lines. Integrated into Google’s internal code review system, Critique, it is intended to help developers identify root causes amid large volumes of varied, unstructured logs. The authors report that a manual evaluation of 71 real-world failures found 90.14% accuracy in diagnosing the root cause. After deployment across Google, the tool was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases, and it ranked #14 in helpfulness among 370 tools posting findings in Critique. The authors argue that can help with this task by processing and summarizing complex textual data, while emphasizing that accuracy affects developer perception and adoption. The abstract reports evaluation results and user feedback, but does not describe the evaluation procedure or how accuracy was calculated.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

arXivArticle · arxiv.orgOpen original source

Original source

· Article

Source details

Original source date: Apr 13, 2026 (2026-04-13).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-20T16:44:07.264Z. First observed: 2026-11-20T16:40:39.726Z. Retrieved: 2026-11-20T16:43:22.012Z.

Referenced by

Recorded citations in public readings. A reference alone does not establish agreement or influence.

No public references recorded here yet.

Reference data (JSON)

Limits

The abstract does not describe the manual evaluation procedure or how the reported accuracy was calculated. The abstract reports user feedback and interview findings but does not provide further methodological detail about them.

Open question

How was the 90.14% accuracy calculated in the manual evaluation of 71 real-world failures?

Source-stated

The authors describe Auto-Diagnose as a tool that analyzes integration test failure logs and produces summaries with relevant log lines.

Auto-Diagnose analyzes failure logs, produces concise summaries with the most relevant log lines

Source-stated

The authors say Auto-Diagnose is integrated into Google's internal code review system, Critique, to provide contextual assistance.

is integrated into Critique, Google's internal code review system, providing contextual and in-time assistance.

Source-stated

The authors report 90.14% accuracy in diagnosing root causes in a manual evaluation of 71 real-world failures.

A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause.

Source-stated

Following Google-wide deployment, Auto-Diagnose was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases.

Following its Google-wide deployment, Auto-Diagnose was used across 52, 635 distinct failing tests. User feedback indicated that the tool was deemed "Not helpful" in only 5.8% of cases

Source-stated

The authors say Auto-Diagnose ranked #14 in helpfulness among 370 tools that post findings in Critique.

it was ranked #14 in helpfulness among 370 tools that post findings in Critique.

Source-stated

The authors argue that accuracy affects developers’ perception of and adoption of automated diagnostic assistance.

with the tool's accuracy remaining a critical factor in shaping developer perception and adoption.

Source and contributor context

Ziftci, Celal, Liu, Ray, Greene, Spencer, Dalloro, Livio · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.

Source media

No suitable media was found by the supported page extraction.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Large language model

A language model with many learned numerical settings, used to process and generate language.

arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

Figure

Figure develops humanoid robots and publishes its own tests of how they perform physical tasks.

Editorial provenance

Version 1. Publication ID: 40eb1c6b-df63-4c76-84ab-efd222d91180.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-20T16:44:07.264Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: 83aaea3b5055f695cd76620aa7cc17a3c1fe33e6df92cdb57de1de170e9d37da. Source text hash: 9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806.

Where this evidence matters

  • Software development · direct · supports

    Retrieval matched on “code review” and also on the writing-verification route. The writing match is incidental: the abstract concerns diagnosing software integration test failures, not writing or analysis, so it is not linked to writing-polished-error or writing-more-judgment. Source: Ziftci, Liu, Greene and Dalloro, arXiv v1, submitted 13 April 2026. This is a new evidence family, distinct from METR, Peng et al., Paradis et al., Anthropic, Narayanan and Kapoor, Andriotte and Ribeiro, and Miller et al. Only the abstract is used here. What the source observes: Auto-Diagnose, an LLM tool integrated into Critique (Google’s internal code review system), summarizes integration-test failure logs to help developers find the root cause. A manual evaluation on 71 real-world failures found 90.14% root-cause accuracy. After Google-wide deployment, the tool was used across 52,635 distinct failing tests. Users rated it “Not helpful” in 5.8% of cases, and it ranked #14 of 370 Critique tools for helpfulness. Interviews reported perceived usefulness. What the authors argue: LLMs are “highly successful” at this diagnosis task, and accuracy is “a critical factor” in developer perception and adoption. Where it bears: - software-development, specifically its task “Use it to investigate before you commit.” That task asks for explanations or candidate fixes checked against the actual repository, with the developer keeping responsibility for the diagnosis. This is deployment-scale evidence that AI-assisted diagnosis on a hard, log-heavy part of repair work is accurate in most sampled cases and rarely judged unhelpful. - ai-work-quality-and-rework. The main read counts investigating and repairing errors as part of total effort. The arrival suggests AI can reduce some of that checking-and-repair burden itself, not only first-draft production. Limits: the outcomes are root-cause accuracy on a 71-failure sample and user ratings, not measured diagnosis time, total human effort at fixed quality, or defects after handoff. It therefore meets no signpost or revision trigger. Roughly one in ten sampled diagnoses was wrong, which reinforces the guide’s advice to verify rather than accept the suggestion. Ratings and rank among internal tools are perception measures. Our reading of context: this is an in-house team evaluating its own deployed tool, which ordinary product-evaluation practice explains. That is a reason to want an independent effort measure, not a reason to discount the accuracy figure. Rival reading: helpfulness ratings may reflect convenience without saving net time, if developers re-verify every diagnosis. A comparison of time-to-root-cause and later fix success with and without the tool would separate the two accounts.

    Read the revision from 2026-11-20
  • Does AI reduce the effort needed to finish work well? · direct · supports

    Retrieval matched on “code review” and also on the writing-verification route. The writing match is incidental: the abstract concerns diagnosing software integration test failures, not writing or analysis, so it is not linked to writing-polished-error or writing-more-judgment. Source: Ziftci, Liu, Greene and Dalloro, arXiv v1, submitted 13 April 2026. This is a new evidence family, distinct from METR, Peng et al., Paradis et al., Anthropic, Narayanan and Kapoor, Andriotte and Ribeiro, and Miller et al. Only the abstract is used here. What the source observes: Auto-Diagnose, an LLM tool integrated into Critique (Google’s internal code review system), summarizes integration-test failure logs to help developers find the root cause. A manual evaluation on 71 real-world failures found 90.14% root-cause accuracy. After Google-wide deployment, the tool was used across 52,635 distinct failing tests. Users rated it “Not helpful” in 5.8% of cases, and it ranked #14 of 370 Critique tools for helpfulness. Interviews reported perceived usefulness. What the authors argue: LLMs are “highly successful” at this diagnosis task, and accuracy is “a critical factor” in developer perception and adoption. Where it bears: - software-development, specifically its task “Use it to investigate before you commit.” That task asks for explanations or candidate fixes checked against the actual repository, with the developer keeping responsibility for the diagnosis. This is deployment-scale evidence that AI-assisted diagnosis on a hard, log-heavy part of repair work is accurate in most sampled cases and rarely judged unhelpful. - ai-work-quality-and-rework. The main read counts investigating and repairing errors as part of total effort. The arrival suggests AI can reduce some of that checking-and-repair burden itself, not only first-draft production. Limits: the outcomes are root-cause accuracy on a 71-failure sample and user ratings, not measured diagnosis time, total human effort at fixed quality, or defects after handoff. It therefore meets no signpost or revision trigger. Roughly one in ten sampled diagnoses was wrong, which reinforces the guide’s advice to verify rather than accept the suggestion. Ratings and rank among internal tools are perception measures. Our reading of context: this is an in-house team evaluating its own deployed tool, which ordinary product-evaluation practice explains. That is a reason to want an independent effort measure, not a reason to discount the accuracy figure. Rival reading: helpfulness ratings may reflect convenience without saving net time, if developers re-verify every diagnosis. A comparison of time-to-root-cause and later fix success with and without the tool would separate the two accounts.

    Read the revision from 2026-11-20

Revision and source-check history

Version 1 · 2026-11-20T16:44:07.264Z

Google paper reports LLM-based diagnosis of integration test failures

Authors Celal Ziftci, Ray Liu, Spencer Greene and Livio Dalloro describe Auto-Diagnose, a tool that analyzes integration test failure logs and provides concise summaries with relevant log lines. Integrated into Google’s internal code review system, Critique, it is intended to help developers identify root causes amid large volumes of varied, unstructured logs. The authors report that a manual evaluation of 71 real-world failures found 90.14% accuracy in diagnosing the root cause. After deployment across Google, the tool was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases, and it ranked #14 in helpfulness among 370 tools posting findings in Critique. The authors argue that can help with this task by processing and summarizing complex textual data, while emphasizing that accuracy affects developer perception and adoption. The abstract reports evaluation results and user feedback, but does not describe the evaluation procedure or how accuracy was calculated.

Original source

· Article

Source details

Original source date: Apr 13, 2026 (2026-04-13).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-20T16:44:07.264Z. First observed: 2026-11-20T16:40:39.726Z. Retrieved: 2026-11-20T16:43:22.012Z.

Limits

The abstract does not describe the manual evaluation procedure or how the reported accuracy was calculated. The abstract reports user feedback and interview findings but does not provide further methodological detail about them.

Open question

How was the 90.14% accuracy calculated in the manual evaluation of 71 real-world failures?

Source-stated

The authors describe Auto-Diagnose as a tool that analyzes integration test failure logs and produces summaries with relevant log lines.

Auto-Diagnose analyzes failure logs, produces concise summaries with the most relevant log lines

Source-stated

The authors say Auto-Diagnose is integrated into Google's internal code review system, Critique, to provide contextual assistance.

is integrated into Critique, Google's internal code review system, providing contextual and in-time assistance.

Source-stated

The authors report 90.14% accuracy in diagnosing root causes in a manual evaluation of 71 real-world failures.

A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause.

Source-stated

Following Google-wide deployment, Auto-Diagnose was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases.

Following its Google-wide deployment, Auto-Diagnose was used across 52, 635 distinct failing tests. User feedback indicated that the tool was deemed "Not helpful" in only 5.8% of cases

Source-stated

The authors say Auto-Diagnose ranked #14 in helpfulness among 370 tools that post findings in Critique.

it was ranked #14 in helpfulness among 370 tools that post findings in Critique.

Source-stated

The authors argue that accuracy affects developers’ perception of and adoption of automated diagnostic assistance.

with the tool's accuracy remaining a critical factor in shaping developer perception and adoption.

Source and contributor context

Ziftci, Celal, Liu, Ray, Greene, Spencer, Dalloro, Livio · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.

Source media

No suitable media was found by the supported page extraction.

Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.

Large language model

A language model with many learned numerical settings, used to process and generate language.

arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

Figure

Figure develops humanoid robots and publishes its own tests of how they perform physical tasks.

Editorial provenance

Version 1. Publication ID: 40eb1c6b-df63-4c76-84ab-efd222d91180.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-20T16:44:07.264Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material_development.

Observation: 83aaea3b5055f695cd76620aa7cc17a3c1fe33e6df92cdb57de1de170e9d37da. Source text hash: 9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806.

uncertain · 2026-11-20T16:44:07.264Z · anthropic/claude-opus-5.5

Publication: 40eb1c6b-df63-4c76-84ab-efd222d91180

attribution — pass: The four named authors match the source. Results and conclusions are consistently attributed to the authors. Calling it a 'Google paper' is a reasonable reading of the title and the Critique description, though the page lists no author affiliations.

Authors:Celal Ziftci, Ray Liu, Spencer Greene, Livio Dalloro

summary — uncertain: The reading softens the authors' conclusion. They write that LLMs are 'highly successful' and that Auto-Diagnose is 'highly effective', but the draft says only that LLMs 'can help'. Smallest repair: attribute the original wording, e.g. 'The authors conclude LLMs are highly successful at this task.' The draft also leaves 'integration test' and 'root cause' unexplained for general readers. It omits the stated motivation that developers spend more time on integration than on unit test failures; a one-line plain explanation or tooltip would fix this.

We conclude that LLMs are highly successful in diagnosing integration test failures due to their capacity to process and summarize complex textual data.

date — pass: originalPublished 2026-04-13 matches the v1 submission date shown on the arXiv page.

[Submitted on 13 Apr 2026]

support — pass: Each claim matches its excerpt, and the figures (71 failures, 90.14%, 52,635 tests, 5.8%, #14 of 370) are reported accurately as author claims. The limitations correctly note that the abstract gives no method detail. The question about how accuracy was calculated is a genuine open issue that follows from the abstract.

A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause.

media — pass: The selected media is the verified direct link to the original arXiv abstract page; no better source media exists.

Challenge or correct this reading

Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.