# Google paper reports LLM-based diagnosis of integration test failures

[Canonical reading](https://escapemeridian.com/readings/google-paper-reports-llm-based-diagnosis-of-integration-test-c5189029)

Authors Celal Ziftci, Ray Liu, Spencer Greene and Livio Dalloro describe Auto-Diagnose, a tool that analyzes integration test failure logs and provides concise summaries with relevant log lines. Integrated into Google’s internal code review system, Critique, it is intended to help developers identify root causes amid large volumes of varied, unstructured logs. The authors report that a manual evaluation of 71 real-world failures found 90.14% accuracy in diagnosing the root cause. After deployment across Google, the tool was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases, and it ranked \#14 in helpfulness among 370 tools posting findings in Critique. The authors argue that LLMs can help with this task by processing and summarizing complex textual data, while emphasizing that accuracy affects developer perception and adoption. The abstract reports evaluation results and user feedback, but does not describe the evaluation procedure or how accuracy was calculated.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv · Article

Original source date: Apr 13, 2026 (2026-04-13).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-20T16:44:07.264Z. First observed: 2026-11-20T16:40:39.726Z. Retrieved: 2026-11-20T16:43:22.012Z.

- [LLM-Based Automated Diagnosis Of Integration Test Failures At Google](https://arxiv.org/abs/2604.12108)

## Limits

The abstract does not describe the manual evaluation procedure or how the reported accuracy was calculated. The abstract reports user feedback and interview findings but does not provide further methodological detail about them.


## Open question

How was the 90.14% accuracy figure calculated in the manual evaluation of 71 real-world failures?


## Source-stated

The authors describe Auto-Diagnose as a tool that analyzes integration test failure logs and produces summaries with relevant log lines.

> Auto-Diagnose analyzes failure logs, produces concise summaries with the most relevant log lines

- [Supporting source](https://arxiv.org/abs/2604.12108)

## Source-stated

The authors say Auto-Diagnose is integrated into Google's internal code review system, Critique, to provide contextual assistance.

> is integrated into Critique, Google's internal code review system, providing contextual and in-time assistance.

- [Supporting source](https://arxiv.org/abs/2604.12108)

## Source-stated

The authors report 90.14% accuracy in diagnosing root causes in a manual evaluation of 71 real-world failures.

> A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause.

- [Supporting source](https://arxiv.org/abs/2604.12108)

## Source-stated

Following Google-wide deployment, Auto-Diagnose was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases.

> Following its Google-wide deployment, Auto-Diagnose was used across 52, 635 distinct failing tests. User feedback indicated that the tool was deemed "Not helpful" in only 5.8% of cases

- [Supporting source](https://arxiv.org/abs/2604.12108)

## Source-stated

The authors say Auto-Diagnose ranked \#14 in helpfulness among 370 tools that post findings in Critique.

> it was ranked \#14 in helpfulness among 370 tools that post findings in Critique.

- [Supporting source](https://arxiv.org/abs/2604.12108)

## Source-stated

The authors argue that accuracy affects developers’ perception of and adoption of automated diagnostic assistance.

> with the tool's accuracy remaining a critical factor in shaping developer perception and adoption.

- [Supporting source](https://arxiv.org/abs/2604.12108)

## Source and contributor context

Ziftci, Celal, Liu, Ray, Greene, Spencer, Dalloro, Livio · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Figure

Figure develops humanoid robots and publishes its own tests of how they perform physical tasks.

- [Explore this context](/entities/figure)
- [Reference](https://www.figure.ai/news)
- [Reference](https://www.figure.ai/news/helix-2-5-zero-shot-30-home-generalization)

## Editorial provenance

Version 1. Publication ID: 40eb1c6b-df63-4c76-84ab-efd222d91180.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-20T16:44:07.264Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 83aaea3b5055f695cd76620aa7cc17a3c1fe33e6df92cdb57de1de170e9d37da. Source text hash: 9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806.


## Where this evidence matters

- [Software development](/?work=software-development#possibilities) · direct · supports
  Retrieval matched on “code review” and also on the writing-verification route. The writing match is incidental: the abstract concerns diagnosing software integration test failures, not writing or analysis, so it is not linked to writing-polished-error or writing-more-judgment.

Source: Ziftci, Liu, Greene and Dalloro, arXiv v1, submitted 13 April 2026. This is a new evidence family, distinct from METR, Peng et al., Paradis et al., Anthropic, Narayanan and Kapoor, Andriotte and Ribeiro, and Miller et al. Only the abstract is used here.

What the source observes: Auto-Diagnose, an LLM tool integrated into Critique (Google’s internal code review system), summarizes integration-test failure logs to help developers find the root cause. A manual evaluation on 71 real-world failures found 90.14% root-cause accuracy. After Google-wide deployment, the tool was used across 52,635 distinct failing tests. Users rated it “Not helpful” in 5.8% of cases, and it ranked \#14 of 370 Critique tools for helpfulness. Interviews reported perceived usefulness.

What the authors argue: LLMs are “highly successful” at this diagnosis task, and accuracy is “a critical factor” in developer perception and adoption.

Where it bears:
- software-development, specifically its task “Use it to investigate before you commit.” That task asks for explanations or candidate fixes checked against the actual repository, with the developer keeping responsibility for the diagnosis. This is deployment-scale evidence that AI-assisted diagnosis on a hard, log-heavy part of repair work is accurate in most sampled cases and rarely judged unhelpful.
- ai-work-quality-and-rework. The main read counts investigating and repairing errors as part of total effort. The arrival suggests AI can reduce some of that checking-and-repair burden itself, not only first-draft production.

Limits: the outcomes are root-cause accuracy on a 71-failure sample and user ratings, not measured diagnosis time, total human effort at fixed quality, or defects after handoff. It therefore meets no signpost or revision trigger. Roughly one in ten sampled diagnoses was wrong, which reinforces the guide’s advice to verify rather than accept the suggestion. Ratings and rank among internal tools are perception measures.

Our reading of context: this is an in-house team evaluating its own deployed tool, which ordinary product-evaluation practice explains. That is a reason to want an independent effort measure, not a reason to discount the accuracy figure.

Rival reading: helpfulness ratings may reflect convenience without saving net time, if developers re-verify every diagnosis. A comparison of time-to-root-cause and later fix success with and without the tool would separate the two accounts.
  [Read the revision from 2026-11-20](/guide-maintenance/ai-work-quality-and-rework/history.json?revision=57cdcc2b517a602ecb395e140296d06308bd7c8a11e9e2a674ed538f781a14c6)

- [Does AI reduce the effort needed to finish work well?](/questions/ai-work-quality-and-rework#our-read) · direct · supports
  Retrieval matched on “code review” and also on the writing-verification route. The writing match is incidental: the abstract concerns diagnosing software integration test failures, not writing or analysis, so it is not linked to writing-polished-error or writing-more-judgment.

Source: Ziftci, Liu, Greene and Dalloro, arXiv v1, submitted 13 April 2026. This is a new evidence family, distinct from METR, Peng et al., Paradis et al., Anthropic, Narayanan and Kapoor, Andriotte and Ribeiro, and Miller et al. Only the abstract is used here.

What the source observes: Auto-Diagnose, an LLM tool integrated into Critique (Google’s internal code review system), summarizes integration-test failure logs to help developers find the root cause. A manual evaluation on 71 real-world failures found 90.14% root-cause accuracy. After Google-wide deployment, the tool was used across 52,635 distinct failing tests. Users rated it “Not helpful” in 5.8% of cases, and it ranked \#14 of 370 Critique tools for helpfulness. Interviews reported perceived usefulness.

What the authors argue: LLMs are “highly successful” at this diagnosis task, and accuracy is “a critical factor” in developer perception and adoption.

Where it bears:
- software-development, specifically its task “Use it to investigate before you commit.” That task asks for explanations or candidate fixes checked against the actual repository, with the developer keeping responsibility for the diagnosis. This is deployment-scale evidence that AI-assisted diagnosis on a hard, log-heavy part of repair work is accurate in most sampled cases and rarely judged unhelpful.
- ai-work-quality-and-rework. The main read counts investigating and repairing errors as part of total effort. The arrival suggests AI can reduce some of that checking-and-repair burden itself, not only first-draft production.

Limits: the outcomes are root-cause accuracy on a 71-failure sample and user ratings, not measured diagnosis time, total human effort at fixed quality, or defects after handoff. It therefore meets no signpost or revision trigger. Roughly one in ten sampled diagnoses was wrong, which reinforces the guide’s advice to verify rather than accept the suggestion. Ratings and rank among internal tools are perception measures.

Our reading of context: this is an in-house team evaluating its own deployed tool, which ordinary product-evaluation practice explains. That is a reason to want an independent effort measure, not a reason to discount the accuracy figure.

Rival reading: helpfulness ratings may reflect convenience without saving net time, if developers re-verify every diagnosis. A comparison of time-to-root-cause and later fix success with and without the tool would separate the two accounts.
  [Read the revision from 2026-11-20](/guide-maintenance/ai-work-quality-and-rework/history.json?revision=57cdcc2b517a602ecb395e140296d06308bd7c8a11e9e2a674ed538f781a14c6)

## Revision and source-check history

### Version 1 — 2026-11-20T16:44:07.264Z

Authors Celal Ziftci, Ray Liu, Spencer Greene and Livio Dalloro describe Auto-Diagnose, a tool that analyzes integration test failure logs and provides concise summaries with relevant log lines. Integrated into Google’s internal code review system, Critique, it is intended to help developers identify root causes amid large volumes of varied, unstructured logs. The authors report that a manual evaluation of 71 real-world failures found 90.14% accuracy in diagnosing the root cause. After deployment across Google, the tool was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases, and it ranked \#14 in helpfulness among 370 tools posting findings in Critique. The authors argue that LLMs can help with this task by processing and summarizing complex textual data, while emphasizing that accuracy affects developer perception and adoption. The abstract reports evaluation results and user feedback, but does not describe the evaluation procedure or how accuracy was calculated.

## Original source

arXiv · Article

Original source date: Apr 13, 2026 (2026-04-13).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-20T16:44:07.264Z. First observed: 2026-11-20T16:40:39.726Z. Retrieved: 2026-11-20T16:43:22.012Z.

- [LLM-Based Automated Diagnosis Of Integration Test Failures At Google](https://arxiv.org/abs/2604.12108)

## Limits

The abstract does not describe the manual evaluation procedure or how the reported accuracy was calculated. The abstract reports user feedback and interview findings but does not provide further methodological detail about them.


## Open question

How was the 90.14% accuracy figure calculated in the manual evaluation of 71 real-world failures?


## Source-stated

The authors describe Auto-Diagnose as a tool that analyzes integration test failure logs and produces summaries with relevant log lines.

> Auto-Diagnose analyzes failure logs, produces concise summaries with the most relevant log lines

- [Supporting source](https://arxiv.org/abs/2604.12108)

## Source-stated

The authors say Auto-Diagnose is integrated into Google's internal code review system, Critique, to provide contextual assistance.

> is integrated into Critique, Google's internal code review system, providing contextual and in-time assistance.

- [Supporting source](https://arxiv.org/abs/2604.12108)

## Source-stated

The authors report 90.14% accuracy in diagnosing root causes in a manual evaluation of 71 real-world failures.

> A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause.

- [Supporting source](https://arxiv.org/abs/2604.12108)

## Source-stated

Following Google-wide deployment, Auto-Diagnose was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases.

> Following its Google-wide deployment, Auto-Diagnose was used across 52, 635 distinct failing tests. User feedback indicated that the tool was deemed "Not helpful" in only 5.8% of cases

- [Supporting source](https://arxiv.org/abs/2604.12108)

## Source-stated

The authors say Auto-Diagnose ranked \#14 in helpfulness among 370 tools that post findings in Critique.

> it was ranked \#14 in helpfulness among 370 tools that post findings in Critique.

- [Supporting source](https://arxiv.org/abs/2604.12108)

## Source-stated

The authors argue that accuracy affects developers’ perception of and adoption of automated diagnostic assistance.

> with the tool's accuracy remaining a critical factor in shaping developer perception and adoption.

- [Supporting source](https://arxiv.org/abs/2604.12108)

## Source and contributor context

Ziftci, Celal, Liu, Ray, Greene, Spencer, Dalloro, Livio · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Figure

Figure develops humanoid robots and publishes its own tests of how they perform physical tasks.

- [Explore this context](/entities/figure)
- [Reference](https://www.figure.ai/news)
- [Reference](https://www.figure.ai/news/helix-2-5-zero-shot-30-home-generalization)

## Editorial provenance

Version 1. Publication ID: 40eb1c6b-df63-4c76-84ab-efd222d91180.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-20T16:44:07.264Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 83aaea3b5055f695cd76620aa7cc17a3c1fe33e6df92cdb57de1de170e9d37da. Source text hash: 9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806.


### uncertain — 2026-11-20T16:44:07.264Z — anthropic/claude-opus-5.5

Publication: 40eb1c6b-df63-4c76-84ab-efd222d91180

attribution — pass: The four named authors match the source. Results and conclusions are consistently attributed to the authors. Calling it a 'Google paper' is a reasonable reading of the title and the Critique description, though the page lists no author affiliations.

> Authors:Celal Ziftci, Ray Liu, Spencer Greene, Livio Dalloro

summary — uncertain: The reading softens the authors' conclusion. They write that LLMs are 'highly successful' and that Auto-Diagnose is 'highly effective', but the draft says only that LLMs 'can help'. Smallest repair: attribute the original wording, e.g. 'The authors conclude LLMs are highly successful at this task.' The draft also leaves 'integration test' and 'root cause' unexplained for general readers. It omits the stated motivation that developers spend more time on integration than on unit test failures; a one-line plain explanation or tooltip would fix this.

> We conclude that LLMs are highly successful in diagnosing integration test failures due to their capacity to process and summarize complex textual data.

date — pass: originalPublished 2026-04-13 matches the v1 submission date shown on the arXiv page.

> \[Submitted on 13 Apr 2026\]

support — pass: Each claim matches its excerpt, and the figures (71 failures, 90.14%, 52,635 tests, 5.8%, \#14 of 370) are reported accurately as author claims. The limitations correctly note that the abstract gives no method detail. The question about how accuracy was calculated is a genuine open issue that follows from the abstract.

> A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause.

media — pass: The selected media is the verified direct link to the original arXiv abstract page; no better source media exists.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/c5189029a128d8c1c18b47105b6e284f8c1a625d5deda9955d52832a7c093f76/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/google-paper-reports-llm-based-diagnosis-of-integration-test-c5189029.json)
