# Radhika Shah’s study finds fact-checking LLMs vulnerable to corrupted multilingual evidence

[Canonical reading](https://escapemeridian.com/readings/radhika-shah-s-study-finds-fact-checking-llms-vulnerable-to-5bb4fc4c)

Radhika Shah’s paper evaluates whether fact-checking LLMs remain accurate when evidence is corrupted or written in languages where models are weaker. The study builds gold evidence sets for 190 claims in each of English, German, Greek, and Romanian, then applies four corruption strategies at two severity levels. It reports that verdict accuracy is highly sensitive to both corruption and language: misleading edits cut Llama 3.1 8B’s English accuracy from 80.00% to 16.84%, and losses widen as language resource level decreases. Machine-translating Greek and Romanian claims into English recovers much, but not all, of the low-resource gap. Rubric-based LLM-as-a-judge scores decline more slowly than verdict accuracy, which the authors say suggests automated reasoning evaluation underestimates degradation. The paper cautions against deploying LLM fact-checkers on noisy multilingual evidence and says it releases evidence datasets and evaluation code for future work.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Sep 17, 2026 (2026-09-17).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-17T11:31:09.358Z. First observed: 2026-11-17T08:33:34.582Z. Retrieved: 2026-11-17T11:18:13.779Z.

- [Evaluating the Robustness of Fact-Checking LLMs on Multilingual Evidence](https://arxiv.org/abs/2611.18623)

## Limits

The abstract does not identify the four corruption strategies or give results for each strategy and severity level. The abstract does not name the other tested LLMs or provide their individual results. The evidence provided is the paper’s abstract, so it does not include details of the rubric-based scoring method or the released datasets and evaluation code.


## Open question

Which of the four corruption strategies, and which severity levels, account for the largest accuracy losses across the tested languages?


## Source-stated

The study evaluates fact-checking LLMs using evidence in English, German, Greek, and Romanian, and applies four corruption strategies at two severity levels.

> we construct gold evidence sets for 190 claims in each of English, German, Greek, and Romanian, then apply four corruption strategies at two severity levels.

- [Supporting source](https://arxiv.org/abs/2611.18623)

## Source-stated

The paper reports that misleading edits reduced Llama 3.1 8B’s English verdict accuracy from 80.00% to 16.84%, with losses widening as language resource level decreases.

> Misleading edit cut Llama 3.1 8B's English accuracy from 80.00% to 16.84%, and losses widen as language resource level decreases.

- [Supporting source](https://arxiv.org/abs/2611.18623)

## Source-stated

The paper reports that translating Greek and Romanian claims into English recovers much, but not all, of the low-resource gap.

> Machine-translating Greek and Romanian claims into English recovers much, but not all, of the low-resource gap.

- [Supporting source](https://arxiv.org/abs/2611.18623)

## Source-stated

The authors say rubric-based LLM-as-a-judge scores decline more slowly than accuracy, suggesting automated reasoning evaluation underestimates degradation.

> Judge scores decline more slowly than accuracy, suggesting that automated reasoning evaluation underestimates degradation.

- [Supporting source](https://arxiv.org/abs/2611.18623)

## Source-stated

The authors caution against deploying LLM fact-checkers on noisy multilingual evidence and say they release evidence datasets and evaluation code for future work.

> These results caution against deploying LLM fact-checkers on noisy multilingual evidence, and we release our evidence datasets and evaluation code to support future work.

- [Supporting source](https://arxiv.org/abs/2611.18623)

## Source and contributor context

Shah, Radhika · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: de6870f3-d38e-4163-8bff-1e2907b8d3be.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-17T11:31:09.358Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: fe3d7a4c3e23f77989c7b0ffdbab4f0e0c37bc5a53699aa5d4cad4905edf3c18. Source text hash: 9895dfb6dc9e30684bbec2a91f9addafdee3e268667804f073a3d5cc7d589303.


## Revision and source-check history

### Version 1 — 2026-11-17T11:31:09.358Z

Radhika Shah’s paper evaluates whether fact-checking LLMs remain accurate when evidence is corrupted or written in languages where models are weaker. The study builds gold evidence sets for 190 claims in each of English, German, Greek, and Romanian, then applies four corruption strategies at two severity levels. It reports that verdict accuracy is highly sensitive to both corruption and language: misleading edits cut Llama 3.1 8B’s English accuracy from 80.00% to 16.84%, and losses widen as language resource level decreases. Machine-translating Greek and Romanian claims into English recovers much, but not all, of the low-resource gap. Rubric-based LLM-as-a-judge scores decline more slowly than verdict accuracy, which the authors say suggests automated reasoning evaluation underestimates degradation. The paper cautions against deploying LLM fact-checkers on noisy multilingual evidence and says it releases evidence datasets and evaluation code for future work.

## Original source

arXiv cs.AI · Article

Original source date: Sep 17, 2026 (2026-09-17).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-17T11:31:09.358Z. First observed: 2026-11-17T08:33:34.582Z. Retrieved: 2026-11-17T11:18:13.779Z.

- [Evaluating the Robustness of Fact-Checking LLMs on Multilingual Evidence](https://arxiv.org/abs/2611.18623)

## Limits

The abstract does not identify the four corruption strategies or give results for each strategy and severity level. The abstract does not name the other tested LLMs or provide their individual results. The evidence provided is the paper’s abstract, so it does not include details of the rubric-based scoring method or the released datasets and evaluation code.


## Open question

Which of the four corruption strategies, and which severity levels, account for the largest accuracy losses across the tested languages?


## Source-stated

The study evaluates fact-checking LLMs using evidence in English, German, Greek, and Romanian, and applies four corruption strategies at two severity levels.

> we construct gold evidence sets for 190 claims in each of English, German, Greek, and Romanian, then apply four corruption strategies at two severity levels.

- [Supporting source](https://arxiv.org/abs/2611.18623)

## Source-stated

The paper reports that misleading edits reduced Llama 3.1 8B’s English verdict accuracy from 80.00% to 16.84%, with losses widening as language resource level decreases.

> Misleading edit cut Llama 3.1 8B's English accuracy from 80.00% to 16.84%, and losses widen as language resource level decreases.

- [Supporting source](https://arxiv.org/abs/2611.18623)

## Source-stated

The paper reports that translating Greek and Romanian claims into English recovers much, but not all, of the low-resource gap.

> Machine-translating Greek and Romanian claims into English recovers much, but not all, of the low-resource gap.

- [Supporting source](https://arxiv.org/abs/2611.18623)

## Source-stated

The authors say rubric-based LLM-as-a-judge scores decline more slowly than accuracy, suggesting automated reasoning evaluation underestimates degradation.

> Judge scores decline more slowly than accuracy, suggesting that automated reasoning evaluation underestimates degradation.

- [Supporting source](https://arxiv.org/abs/2611.18623)

## Source-stated

The authors caution against deploying LLM fact-checkers on noisy multilingual evidence and say they release evidence datasets and evaluation code for future work.

> These results caution against deploying LLM fact-checkers on noisy multilingual evidence, and we release our evidence datasets and evaluation code to support future work.

- [Supporting source](https://arxiv.org/abs/2611.18623)

## Source and contributor context

Shah, Radhika · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: de6870f3-d38e-4163-8bff-1e2907b8d3be.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-17T11:31:09.358Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: fe3d7a4c3e23f77989c7b0ffdbab4f0e0c37bc5a53699aa5d4cad4905edf3c18. Source text hash: 9895dfb6dc9e30684bbec2a91f9addafdee3e268667804f073a3d5cc7d589303.


### passed — 2026-11-17T11:31:09.358Z — anthropic/claude-opus-5.5

Publication: de6870f3-d38e-4163-8bff-1e2907b8d3be

attribution — pass: The paper is correctly attributed to Radhika Shah, its only listed author, and published on arXiv. Findings are framed as what the paper reports or argues. The phrase 'the authors say' is slightly off for a single-author paper, but it does not misattribute any claim.

> Authors:Radhika Shah

summary — pass: The title, summary and question match the abstract. They cover the four languages, the four corruption strategies at two severity levels, the Llama 3.1 8B drop, the partial recovery from translation, the judge-score lag and the deployment caution. The summary partly explains 'low-resource' as languages where models are weaker. Terms such as LLM-as-a-judge and gold evidence sets are kept exact and suit term-library tooltips. The question is an open issue the abstract leaves unresolved.

> real-world evidence is often incomplete, misleading, or written in a language where models are weaker

date — pass: The originalPublished date of 2026-09-17 matches the v1 submission date, not the later feed date of 2026-11-17.

> \[Submitted on 17 Sep 2026\]

support — pass: Every claim quotes the abstract verbatim and keeps the source's own framing, including 'suggesting' for the judge-score interpretation. The caution against deployment is attributed to the paper, not stated as fact. The draft invents no benefits, costs or forecasts. Its listed limitations correctly identify what the abstract omits.

> Judge scores decline more slowly than accuracy, suggesting that automated reasoning evaluation underestimates degradation.

media — pass: The selected media is the verified direct link to the original arXiv abstract page. It matches the source URL, which is the most direct access to the paper.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/5bb4fc4cd3a7a73d182bc2e43691a22d4245f13e45f9769059a806544b9ea62c/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/radhika-shah-s-study-finds-fact-checking-llms-vulnerable-to-5bb4fc4c.json)
