{"id":"c5189029a128d8c1c18b47105b6e284f8c1a625d5deda9955d52832a7c093f76","title":"Google paper reports LLM-based diagnosis of integration test failures","argument":"Authors Celal Ziftci, Ray Liu, Spencer Greene and Livio Dalloro describe Auto-Diagnose, a tool that analyzes integration test failure logs and provides concise summaries with relevant log lines. Integrated into Google’s internal code review system, Critique, it is intended to help developers identify root causes amid large volumes of varied, unstructured logs. The authors report that a manual evaluation of 71 real-world failures found 90.14% accuracy in diagnosing the root cause. After deployment across Google, the tool was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases, and it ranked #14 in helpfulness among 370 tools posting findings in Critique. The authors argue that LLMs can help with this task by processing and summarizing complex textual data, while emphasizing that accuracy affects developer perception and adoption. The abstract reports evaluation results and user feedback, but does not describe the evaluation procedure or how accuracy was calculated.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract does not describe the manual evaluation procedure or how the reported accuracy was calculated. The abstract reports user feedback and interview findings but does not provide further methodological detail about them.","speaker":{"name":"Ziftci, Celal, Liu, Ray, Greene, Spencer, Dalloro, Livio","roleAtSource":"Article author"},"source":{"title":"LLM-Based Automated Diagnosis Of Integration Test Failures At Google","publisher":"arXiv","format":"Article","url":"https://arxiv.org/abs/2604.12108","publishedAt":"2026-04-13","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors describe Auto-Diagnose as a tool that analyzes integration test failure logs and produces summaries with relevant log lines.","excerpt":"Auto-Diagnose analyzes failure logs, produces concise summaries with the most relevant log lines","url":"https://arxiv.org/abs/2604.12108","relationship":"Source-stated"},{"text":"The authors say Auto-Diagnose is integrated into Google's internal code review system, Critique, to provide contextual assistance.","excerpt":"is integrated into Critique, Google's internal code review system, providing contextual and in-time assistance.","url":"https://arxiv.org/abs/2604.12108","relationship":"Source-stated"},{"text":"The authors report 90.14% accuracy in diagnosing root causes in a manual evaluation of 71 real-world failures.","excerpt":"A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause.","url":"https://arxiv.org/abs/2604.12108","relationship":"Source-stated"},{"text":"Following Google-wide deployment, Auto-Diagnose was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases.","excerpt":"Following its Google-wide deployment, Auto-Diagnose was used across 52, 635 distinct failing tests. User feedback indicated that the tool was deemed \"Not helpful\" in only 5.8% of cases","url":"https://arxiv.org/abs/2604.12108","relationship":"Source-stated"},{"text":"The authors say Auto-Diagnose ranked #14 in helpfulness among 370 tools that post findings in Critique.","excerpt":"it was ranked #14 in helpfulness among 370 tools that post findings in Critique.","url":"https://arxiv.org/abs/2604.12108","relationship":"Source-stated"},{"text":"The authors argue that accuracy affects developers’ perception of and adoption of automated diagnostic assistance.","excerpt":"with the tool's accuracy remaining a critical factor in shaping developer perception and adoption.","url":"https://arxiv.org/abs/2604.12108","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2604.12108","checkedAt":"2026-11-21T01:16:09.806Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How was the 90.14% accuracy figure calculated in the manual evaluation of 71 real-world failures?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Google paper reports LLM-based diagnosis of integration test failures","summary":"Authors Celal Ziftci, Ray Liu, Spencer Greene and Livio Dalloro describe Auto-Diagnose, a tool that analyzes integration test failure logs and provides concise summaries with relevant log lines. Integrated into Google’s internal code review system, Critique, it is intended to help developers identify root causes amid large volumes of varied, unstructured logs. The authors report that a manual evaluation of 71 real-world failures found 90.14% accuracy in diagnosing the root cause. After deployment across Google, the tool was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases, and it ranked #14 in helpfulness among 370 tools posting findings in Critique. The authors argue that LLMs can help with this task by processing and summarizing complex textual data, while emphasizing that accuracy affects developer perception and adoption. The abstract reports evaluation results and user feedback, but does not describe the evaluation procedure or how accuracy was calculated.","question":"How was the 90.14% accuracy figure calculated in the manual evaluation of 71 real-world failures?","claims":[{"text":"The authors describe Auto-Diagnose as a tool that analyzes integration test failure logs and produces summaries with relevant log lines.","excerpt":"Auto-Diagnose analyzes failure logs, produces concise summaries with the most relevant log lines","support":"source-stated"},{"text":"The authors say Auto-Diagnose is integrated into Google's internal code review system, Critique, to provide contextual assistance.","excerpt":"is integrated into Critique, Google's internal code review system, providing contextual and in-time assistance.","support":"source-stated"},{"text":"The authors report 90.14% accuracy in diagnosing root causes in a manual evaluation of 71 real-world failures.","excerpt":"A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause.","support":"source-stated"},{"text":"Following Google-wide deployment, Auto-Diagnose was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases.","excerpt":"Following its Google-wide deployment, Auto-Diagnose was used across 52, 635 distinct failing tests. User feedback indicated that the tool was deemed \"Not helpful\" in only 5.8% of cases","support":"source-stated"},{"text":"The authors say Auto-Diagnose ranked #14 in helpfulness among 370 tools that post findings in Critique.","excerpt":"it was ranked #14 in helpfulness among 370 tools that post findings in Critique.","support":"source-stated"},{"text":"The authors argue that accuracy affects developers’ perception of and adoption of automated diagnostic assistance.","excerpt":"with the tool's accuracy remaining a critical factor in shaping developer perception and adoption.","support":"source-stated"}],"limitations":["The abstract does not describe the manual evaluation procedure or how the reported accuracy was calculated.","The abstract reports user feedback and interview findings but does not provide further methodological detail about them."],"mediaId":"39a30b296394debee8d9c22a8c1929fbff5b1f8781f53776d605e4fc0c3a9606","correctionReason":null,"id":"c5189029a128d8c1c18b47105b6e284f8c1a625d5deda9955d52832a7c093f76","publicationId":"40eb1c6b-df63-4c76-84ab-efd222d91180","version":1,"sourceUrl":"https://arxiv.org/abs/2604.12108","sourceTitle":"LLM-Based Automated Diagnosis Of Integration Test Failures At Google","publisher":"arXiv","publishedAt":null,"originalPublished":{"value":"2026-04-13","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-20T16:40:39.726Z","createdAt":"2026-11-20T16:44:07.264Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"39a30b296394debee8d9c22a8c1929fbff5b1f8781f53776d605e4fc0c3a9606","kind":"article","url":"https://arxiv.org/abs/2604.12108","sourceUrl":"https://arxiv.org/abs/2604.12108","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-20T16:43:22.098Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2604.12108","checkedAt":"2026-11-21T01:16:09.806Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"39a30b296394debee8d9c22a8c1929fbff5b1f8781f53776d605e4fc0c3a9606","confidence":0.8200000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"d0f7d9607e796a734c43231cedfb8f4452f63a5b9193773bf75bbe27f5a415a2","decidedAt":"2026-11-20T16:43:22.807Z"},"provenance":{"textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","observationId":"83aaea3b5055f695cd76620aa7cc17a3c1fe33e6df92cdb57de1de170e9d37da","retrievedAt":"2026-11-20T16:43:22.012Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"365232ea-ab68-4b2b-87b1-3185f2acd105"},"reviewer":{"model":"gpt-6-luna","runId":"8e124b64-8762-4e22-889b-fee74e3d9406"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-20T16:44:07.264Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Ziftci, Celal","url":null,"kind":"unknown"},{"name":"Liu, Ray","url":null,"kind":"unknown"},{"name":"Greene, Spencer","url":null,"kind":"unknown"},{"name":"Dalloro, Livio","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Ziftci, Celal","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ziftci, Celal","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2604.12108","locator":"head/meta[name=author|citation_author]"},"assertionId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T18:52:09.753Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124","sourceUrl":"https://arxiv.org/abs/2604.12108","contentSha256":"3c966c2087b187e5c19f25f9059eca51baee8e0f94985790ed08b29f504e9b8b","textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","observedAt":"2026-11-20T18:52:09.753Z"}},{"actor":{"name":"Liu, Ray","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Ray","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2604.12108","locator":"head/meta[name=author|citation_author]"},"assertionId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T18:52:09.753Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124","sourceUrl":"https://arxiv.org/abs/2604.12108","contentSha256":"3c966c2087b187e5c19f25f9059eca51baee8e0f94985790ed08b29f504e9b8b","textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","observedAt":"2026-11-20T18:52:09.753Z"}},{"actor":{"name":"Greene, Spencer","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Greene, Spencer","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2604.12108","locator":"head/meta[name=author|citation_author]"},"assertionId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T18:52:09.753Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124","sourceUrl":"https://arxiv.org/abs/2604.12108","contentSha256":"3c966c2087b187e5c19f25f9059eca51baee8e0f94985790ed08b29f504e9b8b","textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","observedAt":"2026-11-20T18:52:09.753Z"}},{"actor":{"name":"Dalloro, Livio","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Dalloro, Livio","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2604.12108","locator":"head/meta[name=author|citation_author]"},"assertionId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T18:52:09.753Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124","sourceUrl":"https://arxiv.org/abs/2604.12108","contentSha256":"3c966c2087b187e5c19f25f9059eca51baee8e0f94985790ed08b29f504e9b8b","textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","observedAt":"2026-11-20T18:52:09.753Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2604.12108","snapshotId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124","contentSha256":"3c966c2087b187e5c19f25f9059eca51baee8e0f94985790ed08b29f504e9b8b","textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","observedAt":"2026-11-20T18:52:09.753Z","basis":"publisher-page-metadata","sourceLinks":43,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/c5189029a128d8c1c18b47105b6e284f8c1a625d5deda9955d52832a7c093f76/source"}},"canonicalUrl":"https://escapemeridian.com/readings/google-paper-reports-llm-based-diagnosis-of-integration-test-c5189029","formats":{"html":"/readings/google-paper-reports-llm-based-diagnosis-of-integration-test-c5189029","markdown":"/readings/google-paper-reports-llm-based-diagnosis-of-integration-test-c5189029.md","json":"/readings/google-paper-reports-llm-based-diagnosis-of-integration-test-c5189029.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[{"questionId":"ai-work-quality-and-rework","dimensionId":"software-development","pathId":null,"targetId":"software-development","label":"Software development","url":"/?work=software-development#possibilities","relationship":"direct","role":"supports","explanation":"Retrieval matched on “code review” and also on the writing-verification route. The writing match is incidental: the abstract concerns diagnosing software integration test failures, not writing or analysis, so it is not linked to writing-polished-error or writing-more-judgment.\n\nSource: Ziftci, Liu, Greene and Dalloro, arXiv v1, submitted 13 April 2026. This is a new evidence family, distinct from METR, Peng et al., Paradis et al., Anthropic, Narayanan and Kapoor, Andriotte and Ribeiro, and Miller et al. Only the abstract is used here.\n\nWhat the source observes: Auto-Diagnose, an LLM tool integrated into Critique (Google’s internal code review system), summarizes integration-test failure logs to help developers find the root cause. A manual evaluation on 71 real-world failures found 90.14% root-cause accuracy. After Google-wide deployment, the tool was used across 52,635 distinct failing tests. Users rated it “Not helpful” in 5.8% of cases, and it ranked #14 of 370 Critique tools for helpfulness. Interviews reported perceived usefulness.\n\nWhat the authors argue: LLMs are “highly successful” at this diagnosis task, and accuracy is “a critical factor” in developer perception and adoption.\n\nWhere it bears:\n- software-development, specifically its task “Use it to investigate before you commit.” That task asks for explanations or candidate fixes checked against the actual repository, with the developer keeping responsibility for the diagnosis. This is deployment-scale evidence that AI-assisted diagnosis on a hard, log-heavy part of repair work is accurate in most sampled cases and rarely judged unhelpful.\n- ai-work-quality-and-rework. The main read counts investigating and repairing errors as part of total effort. The arrival suggests AI can reduce some of that checking-and-repair burden itself, not only first-draft production.\n\nLimits: the outcomes are root-cause accuracy on a 71-failure sample and user ratings, not measured diagnosis time, total human effort at fixed quality, or defects after handoff. It therefore meets no signpost or revision trigger. Roughly one in ten sampled diagnoses was wrong, which reinforces the guide’s advice to verify rather than accept the suggestion. Ratings and rank among internal tools are perception measures.\n\nOur reading of context: this is an in-house team evaluating its own deployed tool, which ordinary product-evaluation practice explains. That is a reason to want an independent effort measure, not a reason to discount the accuracy figure.\n\nRival reading: helpfulness ratings may reflect convenience without saving net time, if developers re-verify every diagnosis. A comparison of time-to-root-cause and later fix success with and without the tool would separate the two accounts.","mechanism":null,"recordedAt":"2026-11-20T17:24:45.824Z","assumptions":["Root-cause accuracy on 71 manually evaluated failures and user helpfulness ratings are partial proxies; they do not measure diagnosis time, total human effort at fixed quality or later defects.","The abstract does not describe how the 71 failures were sampled or how accuracy was judged; nothing is inferred from omitted text.","Google’s internal tooling, logs and review culture may not transfer to other employers or codebases."],"intermediateSourceIds":[],"intermediateReferences":[],"revisionId":"57cdcc2b517a602ecb395e140296d06308bd7c8a11e9e2a674ed538f781a14c6","history":"/guide-maintenance/ai-work-quality-and-rework/history.json?revision=57cdcc2b517a602ecb395e140296d06308bd7c8a11e9e2a674ed538f781a14c6"},{"questionId":"ai-work-quality-and-rework","dimensionId":null,"pathId":null,"targetId":"ai-work-quality-and-rework","label":"Does AI reduce the effort needed to finish work well?","url":"/questions/ai-work-quality-and-rework#our-read","relationship":"direct","role":"supports","explanation":"Retrieval matched on “code review” and also on the writing-verification route. The writing match is incidental: the abstract concerns diagnosing software integration test failures, not writing or analysis, so it is not linked to writing-polished-error or writing-more-judgment.\n\nSource: Ziftci, Liu, Greene and Dalloro, arXiv v1, submitted 13 April 2026. This is a new evidence family, distinct from METR, Peng et al., Paradis et al., Anthropic, Narayanan and Kapoor, Andriotte and Ribeiro, and Miller et al. Only the abstract is used here.\n\nWhat the source observes: Auto-Diagnose, an LLM tool integrated into Critique (Google’s internal code review system), summarizes integration-test failure logs to help developers find the root cause. A manual evaluation on 71 real-world failures found 90.14% root-cause accuracy. After Google-wide deployment, the tool was used across 52,635 distinct failing tests. Users rated it “Not helpful” in 5.8% of cases, and it ranked #14 of 370 Critique tools for helpfulness. Interviews reported perceived usefulness.\n\nWhat the authors argue: LLMs are “highly successful” at this diagnosis task, and accuracy is “a critical factor” in developer perception and adoption.\n\nWhere it bears:\n- software-development, specifically its task “Use it to investigate before you commit.” That task asks for explanations or candidate fixes checked against the actual repository, with the developer keeping responsibility for the diagnosis. This is deployment-scale evidence that AI-assisted diagnosis on a hard, log-heavy part of repair work is accurate in most sampled cases and rarely judged unhelpful.\n- ai-work-quality-and-rework. The main read counts investigating and repairing errors as part of total effort. The arrival suggests AI can reduce some of that checking-and-repair burden itself, not only first-draft production.\n\nLimits: the outcomes are root-cause accuracy on a 71-failure sample and user ratings, not measured diagnosis time, total human effort at fixed quality, or defects after handoff. It therefore meets no signpost or revision trigger. Roughly one in ten sampled diagnoses was wrong, which reinforces the guide’s advice to verify rather than accept the suggestion. Ratings and rank among internal tools are perception measures.\n\nOur reading of context: this is an in-house team evaluating its own deployed tool, which ordinary product-evaluation practice explains. That is a reason to want an independent effort measure, not a reason to discount the accuracy figure.\n\nRival reading: helpfulness ratings may reflect convenience without saving net time, if developers re-verify every diagnosis. A comparison of time-to-root-cause and later fix success with and without the tool would separate the two accounts.","mechanism":null,"recordedAt":"2026-11-20T17:24:45.824Z","assumptions":["Root-cause accuracy on 71 manually evaluated failures and user helpfulness ratings are partial proxies; they do not measure diagnosis time, total human effort at fixed quality or later defects.","The abstract does not describe how the 71 failures were sampled or how accuracy was judged; nothing is inferred from omitted text.","Google’s internal tooling, logs and review culture may not transfer to other employers or codebases."],"intermediateSourceIds":[],"intermediateReferences":[],"revisionId":"57cdcc2b517a602ecb395e140296d06308bd7c8a11e9e2a674ed538f781a14c6","history":"/guide-maintenance/ai-work-quality-and-rework/history.json?revision=57cdcc2b517a602ecb395e140296d06308bd7c8a11e9e2a674ed538f781a14c6"}],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"figure","name":"Figure","kind":"organization","context":"Figure develops humanoid robots and publishes its own tests of how they perform physical tasks.","referenceUrls":["https://www.figure.ai/news","https://www.figure.ai/news/helix-2-5-zero-shot-30-home-generalization"],"path":"/entities/figure","json":"/entities/figure.json","markdown":"/entities/figure.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"c5189029a128d8c1c18b47105b6e284f8c1a625d5deda9955d52832a7c093f76","revision":"40eb1c6b-df63-4c76-84ab-efd222d91180","revisionKind":"publication-id","title":"Google paper reports LLM-based diagnosis of integration test failures","url":"https://escapemeridian.com/readings/google-paper-reports-llm-based-diagnosis-of-integration-test-c5189029"},"source":{"type":"source","id":"https://arxiv.org/abs/2604.12108","revision":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","revisionKind":"sha256-source-text","title":"LLM-Based Automated Diagnosis Of Integration Test Failures At Google","url":"https://arxiv.org/abs/2604.12108"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"c5189029a128d8c1c18b47105b6e284f8c1a625d5deda9955d52832a7c093f76","title":"Google paper reports LLM-based diagnosis of integration test failures","argument":"Authors Celal Ziftci, Ray Liu, Spencer Greene and Livio Dalloro describe Auto-Diagnose, a tool that analyzes integration test failure logs and provides concise summaries with relevant log lines. Integrated into Google’s internal code review system, Critique, it is intended to help developers identify root causes amid large volumes of varied, unstructured logs. The authors report that a manual evaluation of 71 real-world failures found 90.14% accuracy in diagnosing the root cause. After deployment across Google, the tool was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases, and it ranked #14 in helpfulness among 370 tools posting findings in Critique. The authors argue that LLMs can help with this task by processing and summarizing complex textual data, while emphasizing that accuracy affects developer perception and adoption. The abstract reports evaluation results and user feedback, but does not describe the evaluation procedure or how accuracy was calculated.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract does not describe the manual evaluation procedure or how the reported accuracy was calculated. The abstract reports user feedback and interview findings but does not provide further methodological detail about them.","speaker":{"name":"Ziftci, Celal, Liu, Ray, Greene, Spencer, Dalloro, Livio","roleAtSource":"Article author"},"source":{"title":"LLM-Based Automated Diagnosis Of Integration Test Failures At Google","publisher":"arXiv","format":"Article","url":"https://arxiv.org/abs/2604.12108","publishedAt":"2026-04-13","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors describe Auto-Diagnose as a tool that analyzes integration test failure logs and produces summaries with relevant log lines.","excerpt":"Auto-Diagnose analyzes failure logs, produces concise summaries with the most relevant log lines","url":"https://arxiv.org/abs/2604.12108","relationship":"Source-stated"},{"text":"The authors say Auto-Diagnose is integrated into Google's internal code review system, Critique, to provide contextual assistance.","excerpt":"is integrated into Critique, Google's internal code review system, providing contextual and in-time assistance.","url":"https://arxiv.org/abs/2604.12108","relationship":"Source-stated"},{"text":"The authors report 90.14% accuracy in diagnosing root causes in a manual evaluation of 71 real-world failures.","excerpt":"A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause.","url":"https://arxiv.org/abs/2604.12108","relationship":"Source-stated"},{"text":"Following Google-wide deployment, Auto-Diagnose was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases.","excerpt":"Following its Google-wide deployment, Auto-Diagnose was used across 52, 635 distinct failing tests. User feedback indicated that the tool was deemed \"Not helpful\" in only 5.8% of cases","url":"https://arxiv.org/abs/2604.12108","relationship":"Source-stated"},{"text":"The authors say Auto-Diagnose ranked #14 in helpfulness among 370 tools that post findings in Critique.","excerpt":"it was ranked #14 in helpfulness among 370 tools that post findings in Critique.","url":"https://arxiv.org/abs/2604.12108","relationship":"Source-stated"},{"text":"The authors argue that accuracy affects developers’ perception of and adoption of automated diagnostic assistance.","excerpt":"with the tool's accuracy remaining a critical factor in shaping developer perception and adoption.","url":"https://arxiv.org/abs/2604.12108","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2604.12108","checkedAt":"2026-11-21T01:16:09.806Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How was the 90.14% accuracy figure calculated in the manual evaluation of 71 real-world failures?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Google paper reports LLM-based diagnosis of integration test failures","summary":"Authors Celal Ziftci, Ray Liu, Spencer Greene and Livio Dalloro describe Auto-Diagnose, a tool that analyzes integration test failure logs and provides concise summaries with relevant log lines. Integrated into Google’s internal code review system, Critique, it is intended to help developers identify root causes amid large volumes of varied, unstructured logs. The authors report that a manual evaluation of 71 real-world failures found 90.14% accuracy in diagnosing the root cause. After deployment across Google, the tool was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases, and it ranked #14 in helpfulness among 370 tools posting findings in Critique. The authors argue that LLMs can help with this task by processing and summarizing complex textual data, while emphasizing that accuracy affects developer perception and adoption. The abstract reports evaluation results and user feedback, but does not describe the evaluation procedure or how accuracy was calculated.","question":"How was the 90.14% accuracy figure calculated in the manual evaluation of 71 real-world failures?","claims":[{"text":"The authors describe Auto-Diagnose as a tool that analyzes integration test failure logs and produces summaries with relevant log lines.","excerpt":"Auto-Diagnose analyzes failure logs, produces concise summaries with the most relevant log lines","support":"source-stated"},{"text":"The authors say Auto-Diagnose is integrated into Google's internal code review system, Critique, to provide contextual assistance.","excerpt":"is integrated into Critique, Google's internal code review system, providing contextual and in-time assistance.","support":"source-stated"},{"text":"The authors report 90.14% accuracy in diagnosing root causes in a manual evaluation of 71 real-world failures.","excerpt":"A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause.","support":"source-stated"},{"text":"Following Google-wide deployment, Auto-Diagnose was used across 52,635 distinct failing tests; users rated it “Not helpful” in 5.8% of cases.","excerpt":"Following its Google-wide deployment, Auto-Diagnose was used across 52, 635 distinct failing tests. User feedback indicated that the tool was deemed \"Not helpful\" in only 5.8% of cases","support":"source-stated"},{"text":"The authors say Auto-Diagnose ranked #14 in helpfulness among 370 tools that post findings in Critique.","excerpt":"it was ranked #14 in helpfulness among 370 tools that post findings in Critique.","support":"source-stated"},{"text":"The authors argue that accuracy affects developers’ perception of and adoption of automated diagnostic assistance.","excerpt":"with the tool's accuracy remaining a critical factor in shaping developer perception and adoption.","support":"source-stated"}],"limitations":["The abstract does not describe the manual evaluation procedure or how the reported accuracy was calculated.","The abstract reports user feedback and interview findings but does not provide further methodological detail about them."],"mediaId":"39a30b296394debee8d9c22a8c1929fbff5b1f8781f53776d605e4fc0c3a9606","correctionReason":null,"id":"c5189029a128d8c1c18b47105b6e284f8c1a625d5deda9955d52832a7c093f76","publicationId":"40eb1c6b-df63-4c76-84ab-efd222d91180","version":1,"sourceUrl":"https://arxiv.org/abs/2604.12108","sourceTitle":"LLM-Based Automated Diagnosis Of Integration Test Failures At Google","publisher":"arXiv","publishedAt":null,"originalPublished":{"value":"2026-04-13","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-20T16:40:39.726Z","createdAt":"2026-11-20T16:44:07.264Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"39a30b296394debee8d9c22a8c1929fbff5b1f8781f53776d605e4fc0c3a9606","kind":"article","url":"https://arxiv.org/abs/2604.12108","sourceUrl":"https://arxiv.org/abs/2604.12108","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-20T16:43:22.098Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2604.12108","checkedAt":"2026-11-21T01:16:09.806Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"39a30b296394debee8d9c22a8c1929fbff5b1f8781f53776d605e4fc0c3a9606","confidence":0.8200000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"d0f7d9607e796a734c43231cedfb8f4452f63a5b9193773bf75bbe27f5a415a2","decidedAt":"2026-11-20T16:43:22.807Z"},"provenance":{"textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","observationId":"83aaea3b5055f695cd76620aa7cc17a3c1fe33e6df92cdb57de1de170e9d37da","retrievedAt":"2026-11-20T16:43:22.012Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"365232ea-ab68-4b2b-87b1-3185f2acd105"},"reviewer":{"model":"gpt-6-luna","runId":"8e124b64-8762-4e22-889b-fee74e3d9406"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-20T16:44:07.264Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Ziftci, Celal","url":null,"kind":"unknown"},{"name":"Liu, Ray","url":null,"kind":"unknown"},{"name":"Greene, Spencer","url":null,"kind":"unknown"},{"name":"Dalloro, Livio","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Ziftci, Celal","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ziftci, Celal","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2604.12108","locator":"head/meta[name=author|citation_author]"},"assertionId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T18:52:09.753Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124","sourceUrl":"https://arxiv.org/abs/2604.12108","contentSha256":"3c966c2087b187e5c19f25f9059eca51baee8e0f94985790ed08b29f504e9b8b","textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","observedAt":"2026-11-20T18:52:09.753Z"}},{"actor":{"name":"Liu, Ray","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Liu, Ray","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2604.12108","locator":"head/meta[name=author|citation_author]"},"assertionId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T18:52:09.753Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124","sourceUrl":"https://arxiv.org/abs/2604.12108","contentSha256":"3c966c2087b187e5c19f25f9059eca51baee8e0f94985790ed08b29f504e9b8b","textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","observedAt":"2026-11-20T18:52:09.753Z"}},{"actor":{"name":"Greene, Spencer","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Greene, Spencer","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2604.12108","locator":"head/meta[name=author|citation_author]"},"assertionId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T18:52:09.753Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124","sourceUrl":"https://arxiv.org/abs/2604.12108","contentSha256":"3c966c2087b187e5c19f25f9059eca51baee8e0f94985790ed08b29f504e9b8b","textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","observedAt":"2026-11-20T18:52:09.753Z"}},{"actor":{"name":"Dalloro, Livio","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Dalloro, Livio","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2604.12108","locator":"head/meta[name=author|citation_author]"},"assertionId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-20T18:52:09.753Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124","sourceUrl":"https://arxiv.org/abs/2604.12108","contentSha256":"3c966c2087b187e5c19f25f9059eca51baee8e0f94985790ed08b29f504e9b8b","textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","observedAt":"2026-11-20T18:52:09.753Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2604.12108","snapshotId":"34313cfd9e10b6d4432a9749c98ea8daf8be0d62c926f0128bc3c8790cbcf124","contentSha256":"3c966c2087b187e5c19f25f9059eca51baee8e0f94985790ed08b29f504e9b8b","textHash":"9d445b9b73bab7c912a9a9c29b6372bf3912c73a4af19a4a9a0c6424e586f806","observedAt":"2026-11-20T18:52:09.753Z","basis":"publisher-page-metadata","sourceLinks":43,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/c5189029a128d8c1c18b47105b6e284f8c1a625d5deda9955d52832a7c093f76/source"}}}],"audits":[{"publicationId":"40eb1c6b-df63-4c76-84ab-efd222d91180","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The four named authors match the source. Results and conclusions are consistently attributed to the authors. Calling it a 'Google paper' is a reasonable reading of the title and the Critique description, though the page lists no author affiliations.","excerpt":"Authors:Celal Ziftci, Ray Liu, Spencer Greene, Livio Dalloro"},{"dimension":"summary","status":"uncertain","explanation":"The reading softens the authors' conclusion. They write that LLMs are 'highly successful' and that Auto-Diagnose is 'highly effective', but the draft says only that LLMs 'can help'. Smallest repair: attribute the original wording, e.g. 'The authors conclude LLMs are highly successful at this task.' The draft also leaves 'integration test' and 'root cause' unexplained for general readers. It omits the stated motivation that developers spend more time on integration than on unit test failures; a one-line plain explanation or tooltip would fix this.","excerpt":"We conclude that LLMs are highly successful in diagnosing integration test failures due to their capacity to process and summarize complex textual data."},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-04-13 matches the v1 submission date shown on the arXiv page.","excerpt":"[Submitted on 13 Apr 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim matches its excerpt, and the figures (71 failures, 90.14%, 52,635 tests, 5.8%, #14 of 370) are reported accurately as author claims. The limitations correctly note that the abstract gives no method detail. The question about how accuracy was calculated is a genuine open issue that follows from the abstract.","excerpt":"A manual evaluation conducted on 71 real-world failures demonstrated 90.14% accuracy in diagnosing the root cause."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the original arXiv abstract page; no better source media exists.","excerpt":null}],"createdAt":"2026-11-20T16:44:07.264Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}