{"id":"2ec9733fca882f062bda6375238136092a1fa744a9dbd2fd532c25f81c91a95b","title":"arXiv cs.AI: Can AI Agents Synthesize Scientific Conclusions in Health?","argument":"The authors of “Can AI Agents Synthesize Scientific Conclusions in Health?” introduce SciConBench, a benchmark built from questions and expert-written conclusions in systematic reviews, to test how well AI agents synthesize scientific findings in health and medicine. They also introduce SciConHarness, a clean-room evaluation harness that gives agents controlled web interaction to mitigate data leakage. In their evaluation of frontier models and deep research agents, the best agent reaches a factual F1 of 0.337 under clean-room settings. The authors report that clean-room results are consistently lower than unconstrained results, which they say suggests leakage inflates estimates of agents’ synthesis capabilities. They also report that consumer-facing agents often produce incomplete, and sometimes contradictory, conclusions even when the ground-truth answer is available. Their argument is that measuring synthesis requires controlling for leakage, and that reliable synthesis of scientific conclusions in health remains an open challenge. The abstract does not identify the evaluated models or give the benchmark’s question breakdown.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract does not name the 8 frontier models and deep research agents evaluated. The abstract does not give a breakdown of SciConBench questions or report results by question type.","speaker":{"name":"Jung, Hayoung, Diniz, Pedro Viana, Roveda, José Reinaldo Corrêa, da Silva, Abner Fernandes, Jung, Haeun, Tsai, Enoch, Korolova, Aleksandra, Ribeiro, Manoel Horta","roleAtSource":"Article author"},"source":{"title":"Can AI Agents Synthesize Scientific Conclusions in Health?","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2606.11337","publishedAt":"2026-06-09","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce SciConBench, a benchmark of questions and expert-written conclusions from systematic reviews, to evaluate scientific conclusion synthesis in health and medical domains.","excerpt":"We introduce SciConBench, a large-scale live benchmark of 9.21K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis in health and medical domains.","url":"https://arxiv.org/abs/2606.11337","relationship":"Source-stated"},{"text":"The authors introduce SciConHarness, a clean-room evaluation harness with controlled web interaction intended to support valid measurement and mitigate data leakage.","excerpt":"To mitigate data leakage, we further introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement.","url":"https://arxiv.org/abs/2606.11337","relationship":"Source-stated"},{"text":"In the authors’ clean-room evaluation, the best agent achieves a factual F1 of 0.337.","excerpt":"under clean-room settings, the best agent achieves only a factual F1 of 0.337.","url":"https://arxiv.org/abs/2606.11337","relationship":"Source-stated"},{"text":"The authors report that clean-room evaluation consistently lowers performance relative to unconstrained evaluation, and suggest leakage inflates estimates of agents’ synthesis capabilities.","excerpt":"Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities.","url":"https://arxiv.org/abs/2606.11337","relationship":"Source-stated"},{"text":"The authors report that consumer-facing agents often generate incomplete and sometimes contradictory conclusions, including when the ground-truth answer is available.","excerpt":"we audit consumer-facing agents (e.g., Google AI Overview, OpenEvidence) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground-truth answer is available.","url":"https://arxiv.org/abs/2606.11337","relationship":"Source-stated"},{"text":"The authors conclude that reliable synthesis of scientific conclusions in health remains an open challenge and that clean-room evaluation is essential for assessing open-domain AI agents.","excerpt":"reliable synthesis of scientific conclusions in health remains an open challenge, and that clean-room evaluation is essential for assessing open-domain AI agents.","url":"https://arxiv.org/abs/2606.11337","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2606.11337","checkedAt":"2026-11-12T04:48:22.816Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How do the clean-room results vary across the evaluated models and questions in SciConBench?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Can AI Agents Synthesize Scientific Conclusions in Health?","summary":"The authors of “Can AI Agents Synthesize Scientific Conclusions in Health?” introduce SciConBench, a benchmark built from questions and expert-written conclusions in systematic reviews, to test how well AI agents synthesize scientific findings in health and medicine. They also introduce SciConHarness, a clean-room evaluation harness that gives agents controlled web interaction to mitigate data leakage. In their evaluation of frontier models and deep research agents, the best agent reaches a factual F1 of 0.337 under clean-room settings. The authors report that clean-room results are consistently lower than unconstrained results, which they say suggests leakage inflates estimates of agents’ synthesis capabilities. They also report that consumer-facing agents often produce incomplete, and sometimes contradictory, conclusions even when the ground-truth answer is available. Their argument is that measuring synthesis requires controlling for leakage, and that reliable synthesis of scientific conclusions in health remains an open challenge. The abstract does not identify the evaluated models or give the benchmark’s question breakdown.","question":"How do the clean-room results vary across the evaluated models and questions in SciConBench?","claims":[{"text":"The authors introduce SciConBench, a benchmark of questions and expert-written conclusions from systematic reviews, to evaluate scientific conclusion synthesis in health and medical domains.","excerpt":"We introduce SciConBench, a large-scale live benchmark of 9.21K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis in health and medical domains.","support":"source-stated"},{"text":"The authors introduce SciConHarness, a clean-room evaluation harness with controlled web interaction intended to support valid measurement and mitigate data leakage.","excerpt":"To mitigate data leakage, we further introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement.","support":"source-stated"},{"text":"In the authors’ clean-room evaluation, the best agent achieves a factual F1 of 0.337.","excerpt":"under clean-room settings, the best agent achieves only a factual F1 of 0.337.","support":"source-stated"},{"text":"The authors report that clean-room evaluation consistently lowers performance relative to unconstrained evaluation, and suggest leakage inflates estimates of agents’ synthesis capabilities.","excerpt":"Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities.","support":"source-stated"},{"text":"The authors report that consumer-facing agents often generate incomplete and sometimes contradictory conclusions, including when the ground-truth answer is available.","excerpt":"we audit consumer-facing agents (e.g., Google AI Overview, OpenEvidence) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground-truth answer is available.","support":"source-stated"},{"text":"The authors conclude that reliable synthesis of scientific conclusions in health remains an open challenge and that clean-room evaluation is essential for assessing open-domain AI agents.","excerpt":"reliable synthesis of scientific conclusions in health remains an open challenge, and that clean-room evaluation is essential for assessing open-domain AI agents.","support":"source-stated"}],"limitations":["The abstract does not name the 8 frontier models and deep research agents evaluated.","The abstract does not give a breakdown of SciConBench questions or report results by question type."],"mediaId":"c628d42837a460c11c3af8642815cbe1d2622d4f0bf3142df8c0dbc4b1fb4c8b","correctionReason":null,"id":"2ec9733fca882f062bda6375238136092a1fa744a9dbd2fd532c25f81c91a95b","publicationId":"077fbd20-df59-4bc2-8552-2315edc4ef1b","version":1,"sourceUrl":"https://arxiv.org/abs/2606.11337","sourceTitle":"Can AI Agents Synthesize Scientific Conclusions in Health?","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-06-09","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-11T05:00:43.541Z","createdAt":"2026-11-12T04:33:25.101Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"c628d42837a460c11c3af8642815cbe1d2622d4f0bf3142df8c0dbc4b1fb4c8b","kind":"article","url":"https://arxiv.org/abs/2606.11337","sourceUrl":"https://arxiv.org/abs/2606.11337","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T04:32:31.593Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2606.11337","checkedAt":"2026-11-12T04:48:22.816Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"c628d42837a460c11c3af8642815cbe1d2622d4f0bf3142df8c0dbc4b1fb4c8b","confidence":0.8700000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"bd875d6c8fcd22abcf8ca28621db731f6776a99dd4f215bf45e3c90c19213e9c","decidedAt":"2026-11-12T04:32:33.061Z"},"provenance":{"textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observationId":"009a05295246c67c1a8eb3e79ab079a75fe540da55682dc1991e8b5b9f6df107","retrievedAt":"2026-11-12T04:32:31.403Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"474a0ebb-b665-4fe3-8753-f47ebeca0622"},"reviewer":{"model":"gpt-6-luna","runId":"7af2fbba-5ee6-4efb-8ad8-dda89a7a1ee2"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T04:33:25.101Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Jung, Hayoung","url":null,"kind":"unknown"},{"name":"Diniz, Pedro Viana","url":null,"kind":"unknown"},{"name":"Roveda, José Reinaldo Corrêa","url":null,"kind":"unknown"},{"name":"da Silva, Abner Fernandes","url":null,"kind":"unknown"},{"name":"Jung, Haeun","url":null,"kind":"unknown"},{"name":"Tsai, Enoch","url":null,"kind":"unknown"},{"name":"Korolova, Aleksandra","url":null,"kind":"unknown"},{"name":"Ribeiro, Manoel Horta","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Jung, Hayoung","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jung, Hayoung","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"Diniz, Pedro Viana","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Diniz, Pedro Viana","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"Roveda, José Reinaldo Corrêa","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Roveda, José Reinaldo Corrêa","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"da Silva, Abner Fernandes","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"da Silva, Abner Fernandes","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"Jung, Haeun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jung, Haeun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"Tsai, Enoch","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tsai, Enoch","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"Korolova, Aleksandra","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Korolova, Aleksandra","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"Ribeiro, Manoel Horta","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ribeiro, Manoel Horta","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2606.11337","snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z","basis":"publisher-page-metadata","sourceLinks":50,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/2ec9733fca882f062bda6375238136092a1fa744a9dbd2fd532c25f81c91a95b/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-can-ai-agents-synthesize-scientific-conclusions-in-2ec9733f","formats":{"html":"/readings/arxiv-cs-ai-can-ai-agents-synthesize-scientific-conclusions-in-2ec9733f","markdown":"/readings/arxiv-cs-ai-can-ai-agents-synthesize-scientific-conclusions-in-2ec9733f.md","json":"/readings/arxiv-cs-ai-can-ai-agents-synthesize-scientific-conclusions-in-2ec9733f.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-ai-agent","name":"AI agent","explanation":"An AI system that chooses and carries out actions toward a goal, often using software tools.","referenceUrl":"https://www.ibm.com/think/topics/ai-agents"},{"id":"term-agent-harness","name":"Agent harness","explanation":"An agent harness is the system that lets an AI model act as an agent: it processes inputs, orchestrates tool calls and returns results. An evaluation harness runs evals end to end, including grading outputs.","referenceUrl":"https://www.anthropic.com/research/demystifying-evals-for-ai-agents"},{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"2ec9733fca882f062bda6375238136092a1fa744a9dbd2fd532c25f81c91a95b","revision":"077fbd20-df59-4bc2-8552-2315edc4ef1b","revisionKind":"publication-id","title":"arXiv cs.AI: Can AI Agents Synthesize Scientific Conclusions in Health?","url":"https://escapemeridian.com/readings/arxiv-cs-ai-can-ai-agents-synthesize-scientific-conclusions-in-2ec9733f"},"source":{"type":"source","id":"https://arxiv.org/abs/2606.11337","revision":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","revisionKind":"sha256-source-text","title":"Can AI Agents Synthesize Scientific Conclusions in Health?","url":"https://arxiv.org/abs/2606.11337"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"2ec9733fca882f062bda6375238136092a1fa744a9dbd2fd532c25f81c91a95b","title":"arXiv cs.AI: Can AI Agents Synthesize Scientific Conclusions in Health?","argument":"The authors of “Can AI Agents Synthesize Scientific Conclusions in Health?” introduce SciConBench, a benchmark built from questions and expert-written conclusions in systematic reviews, to test how well AI agents synthesize scientific findings in health and medicine. They also introduce SciConHarness, a clean-room evaluation harness that gives agents controlled web interaction to mitigate data leakage. In their evaluation of frontier models and deep research agents, the best agent reaches a factual F1 of 0.337 under clean-room settings. The authors report that clean-room results are consistently lower than unconstrained results, which they say suggests leakage inflates estimates of agents’ synthesis capabilities. They also report that consumer-facing agents often produce incomplete, and sometimes contradictory, conclusions even when the ground-truth answer is available. Their argument is that measuring synthesis requires controlling for leakage, and that reliable synthesis of scientific conclusions in health remains an open challenge. The abstract does not identify the evaluated models or give the benchmark’s question breakdown.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract does not name the 8 frontier models and deep research agents evaluated. The abstract does not give a breakdown of SciConBench questions or report results by question type.","speaker":{"name":"Jung, Hayoung, Diniz, Pedro Viana, Roveda, José Reinaldo Corrêa, da Silva, Abner Fernandes, Jung, Haeun, Tsai, Enoch, Korolova, Aleksandra, Ribeiro, Manoel Horta","roleAtSource":"Article author"},"source":{"title":"Can AI Agents Synthesize Scientific Conclusions in Health?","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2606.11337","publishedAt":"2026-06-09","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce SciConBench, a benchmark of questions and expert-written conclusions from systematic reviews, to evaluate scientific conclusion synthesis in health and medical domains.","excerpt":"We introduce SciConBench, a large-scale live benchmark of 9.21K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis in health and medical domains.","url":"https://arxiv.org/abs/2606.11337","relationship":"Source-stated"},{"text":"The authors introduce SciConHarness, a clean-room evaluation harness with controlled web interaction intended to support valid measurement and mitigate data leakage.","excerpt":"To mitigate data leakage, we further introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement.","url":"https://arxiv.org/abs/2606.11337","relationship":"Source-stated"},{"text":"In the authors’ clean-room evaluation, the best agent achieves a factual F1 of 0.337.","excerpt":"under clean-room settings, the best agent achieves only a factual F1 of 0.337.","url":"https://arxiv.org/abs/2606.11337","relationship":"Source-stated"},{"text":"The authors report that clean-room evaluation consistently lowers performance relative to unconstrained evaluation, and suggest leakage inflates estimates of agents’ synthesis capabilities.","excerpt":"Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities.","url":"https://arxiv.org/abs/2606.11337","relationship":"Source-stated"},{"text":"The authors report that consumer-facing agents often generate incomplete and sometimes contradictory conclusions, including when the ground-truth answer is available.","excerpt":"we audit consumer-facing agents (e.g., Google AI Overview, OpenEvidence) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground-truth answer is available.","url":"https://arxiv.org/abs/2606.11337","relationship":"Source-stated"},{"text":"The authors conclude that reliable synthesis of scientific conclusions in health remains an open challenge and that clean-room evaluation is essential for assessing open-domain AI agents.","excerpt":"reliable synthesis of scientific conclusions in health remains an open challenge, and that clean-room evaluation is essential for assessing open-domain AI agents.","url":"https://arxiv.org/abs/2606.11337","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2606.11337","checkedAt":"2026-11-12T04:48:22.816Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How do the clean-room results vary across the evaluated models and questions in SciConBench?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Can AI Agents Synthesize Scientific Conclusions in Health?","summary":"The authors of “Can AI Agents Synthesize Scientific Conclusions in Health?” introduce SciConBench, a benchmark built from questions and expert-written conclusions in systematic reviews, to test how well AI agents synthesize scientific findings in health and medicine. They also introduce SciConHarness, a clean-room evaluation harness that gives agents controlled web interaction to mitigate data leakage. In their evaluation of frontier models and deep research agents, the best agent reaches a factual F1 of 0.337 under clean-room settings. The authors report that clean-room results are consistently lower than unconstrained results, which they say suggests leakage inflates estimates of agents’ synthesis capabilities. They also report that consumer-facing agents often produce incomplete, and sometimes contradictory, conclusions even when the ground-truth answer is available. Their argument is that measuring synthesis requires controlling for leakage, and that reliable synthesis of scientific conclusions in health remains an open challenge. The abstract does not identify the evaluated models or give the benchmark’s question breakdown.","question":"How do the clean-room results vary across the evaluated models and questions in SciConBench?","claims":[{"text":"The authors introduce SciConBench, a benchmark of questions and expert-written conclusions from systematic reviews, to evaluate scientific conclusion synthesis in health and medical domains.","excerpt":"We introduce SciConBench, a large-scale live benchmark of 9.21K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis in health and medical domains.","support":"source-stated"},{"text":"The authors introduce SciConHarness, a clean-room evaluation harness with controlled web interaction intended to support valid measurement and mitigate data leakage.","excerpt":"To mitigate data leakage, we further introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement.","support":"source-stated"},{"text":"In the authors’ clean-room evaluation, the best agent achieves a factual F1 of 0.337.","excerpt":"under clean-room settings, the best agent achieves only a factual F1 of 0.337.","support":"source-stated"},{"text":"The authors report that clean-room evaluation consistently lowers performance relative to unconstrained evaluation, and suggest leakage inflates estimates of agents’ synthesis capabilities.","excerpt":"Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities.","support":"source-stated"},{"text":"The authors report that consumer-facing agents often generate incomplete and sometimes contradictory conclusions, including when the ground-truth answer is available.","excerpt":"we audit consumer-facing agents (e.g., Google AI Overview, OpenEvidence) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground-truth answer is available.","support":"source-stated"},{"text":"The authors conclude that reliable synthesis of scientific conclusions in health remains an open challenge and that clean-room evaluation is essential for assessing open-domain AI agents.","excerpt":"reliable synthesis of scientific conclusions in health remains an open challenge, and that clean-room evaluation is essential for assessing open-domain AI agents.","support":"source-stated"}],"limitations":["The abstract does not name the 8 frontier models and deep research agents evaluated.","The abstract does not give a breakdown of SciConBench questions or report results by question type."],"mediaId":"c628d42837a460c11c3af8642815cbe1d2622d4f0bf3142df8c0dbc4b1fb4c8b","correctionReason":null,"id":"2ec9733fca882f062bda6375238136092a1fa744a9dbd2fd532c25f81c91a95b","publicationId":"077fbd20-df59-4bc2-8552-2315edc4ef1b","version":1,"sourceUrl":"https://arxiv.org/abs/2606.11337","sourceTitle":"Can AI Agents Synthesize Scientific Conclusions in Health?","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-06-09","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-11T05:00:43.541Z","createdAt":"2026-11-12T04:33:25.101Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"c628d42837a460c11c3af8642815cbe1d2622d4f0bf3142df8c0dbc4b1fb4c8b","kind":"article","url":"https://arxiv.org/abs/2606.11337","sourceUrl":"https://arxiv.org/abs/2606.11337","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T04:32:31.593Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2606.11337","checkedAt":"2026-11-12T04:48:22.816Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"c628d42837a460c11c3af8642815cbe1d2622d4f0bf3142df8c0dbc4b1fb4c8b","confidence":0.8700000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"bd875d6c8fcd22abcf8ca28621db731f6776a99dd4f215bf45e3c90c19213e9c","decidedAt":"2026-11-12T04:32:33.061Z"},"provenance":{"textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observationId":"009a05295246c67c1a8eb3e79ab079a75fe540da55682dc1991e8b5b9f6df107","retrievedAt":"2026-11-12T04:32:31.403Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"474a0ebb-b665-4fe3-8753-f47ebeca0622"},"reviewer":{"model":"gpt-6-luna","runId":"7af2fbba-5ee6-4efb-8ad8-dda89a7a1ee2"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T04:33:25.101Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Jung, Hayoung","url":null,"kind":"unknown"},{"name":"Diniz, Pedro Viana","url":null,"kind":"unknown"},{"name":"Roveda, José Reinaldo Corrêa","url":null,"kind":"unknown"},{"name":"da Silva, Abner Fernandes","url":null,"kind":"unknown"},{"name":"Jung, Haeun","url":null,"kind":"unknown"},{"name":"Tsai, Enoch","url":null,"kind":"unknown"},{"name":"Korolova, Aleksandra","url":null,"kind":"unknown"},{"name":"Ribeiro, Manoel Horta","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Jung, Hayoung","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jung, Hayoung","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"Diniz, Pedro Viana","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Diniz, Pedro Viana","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"Roveda, José Reinaldo Corrêa","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Roveda, José Reinaldo Corrêa","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"da Silva, Abner Fernandes","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"da Silva, Abner Fernandes","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"Jung, Haeun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Jung, Haeun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"Tsai, Enoch","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tsai, Enoch","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"Korolova, Aleksandra","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Korolova, Aleksandra","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}},{"actor":{"name":"Ribeiro, Manoel Horta","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ribeiro, Manoel Horta","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2606.11337","locator":"head/meta[name=author|citation_author]"},"assertionId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T04:48:22.816Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","sourceUrl":"https://arxiv.org/abs/2606.11337","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z"}}],"participationCoverage":{"total":8,"returned":8,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2606.11337","snapshotId":"dc3ae0ae592dddefb47123ff192aab0dd30705064d05121a241f5ea51d899391","contentSha256":"1d0149143c0cb4965d99972909cffb96a9978c9a1ddce10117f026033f9d7d45","textHash":"44287b845bf017e8db28e8592b5e1953e0f4886561690a1018bb4091cbb1219d","observedAt":"2026-11-12T04:48:22.816Z","basis":"publisher-page-metadata","sourceLinks":50,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/2ec9733fca882f062bda6375238136092a1fa744a9dbd2fd532c25f81c91a95b/source"}}}],"audits":[{"publicationId":"077fbd20-df59-4bc2-8552-2315edc4ef1b","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"Findings and conclusions are consistently attributed to the paper's authors. The consumer-agent audit, the leakage interpretation and the open-challenge conclusion remain the authors' claims rather than established fact.","excerpt":"Authors:Hayoung Jung, Pedro Viana Diniz, José Reinaldo Corrêa Roveda, Abner Fernandes da Silva, Haeun Jung, Enoch Tsai, Aleksandra Korolova, Manoel Horta Ribeiro"},{"dimension":"summary","status":"uncertain","explanation":"Central terms go unexplained, so a non-specialist cannot read the headline result. The draft never says that factual F1 combines precision (correctness) and recall (comprehensiveness) of atomic facts on a 0-1 scale, so 0.337 has no meaning. It also does not explain data leakage or clean-room evaluation. It drops the authors' framing that factual quality remains low (their 'only'), as well as the 9.21K scale and the named consumer agents. Smallest repair: add plain glosses for F1 and leakage and restore the 'remains low' framing. The title merely repeats the source title.","excerpt":"measures correctness and comprehensiveness via factual precision and recall"},{"dimension":"date","status":"pass","explanation":"The originalPublished value of 2026-06-09 matches the v1 submission date. It is not the 10 Nov 2026 revision or the feed date.","excerpt":"[Submitted on 9 Jun 2026 (v1), last revised 10 Nov 2026 (this version, v2)]"},{"dimension":"support","status":"pass","explanation":"Each claim is directly supported by the abstract. The draft rephrases 'models' true synthesis capabilities' as 'agents' synthesis capabilities', which is a minor wording shift but not a contradiction. The draft makes no unsupported benefits or forecasts. The question about variation across models and questions follows legitimately from gaps in the abstract.","excerpt":"Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified arXiv abstract page for this paper. It is the direct original source.","excerpt":null}],"createdAt":"2026-11-12T04:33:25.101Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}