Beesetty, Madhu Sudhan Rao, Sharma, Surbhi, Maheshwari, Devesh, Meesala, Lokesh · Nov 6, 2026
arXiv cs.AI: The Invisible Gap in Romanized Indic Generation
Markdown · JSON · Original source
The paper’s authors argue that automatic metrics can miss serious failures in romanized Indic generation: producing Hindi, Telugu, or Kannada in Latin characters. They compare four on 142 OpenAssistant query-response pairs and find that metric scores can conceal differences apparent to native speakers. On a 72-response diagnostic subset, native speakers rated GPT-5.5 1.43 points higher than Opus 4.8 on a 12-point rubric, while the models’ full-dataset BGE-M3 means differed by only 0.004. The authors also report distinct output failures: Gemini sometimes leaks native script into Telugu responses, while Qwen sometimes produces English-only Kannada responses. Their proposed response is a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses. They argue evaluation should combine semantic similarity, script compliance, failure coverage, and native-speaker acceptability. The stakes are practical: a score that misses script or instruction failures may not reflect whether an output is usable for people communicating in romanized Indic languages. The paper’s abstract does not establish how well this evaluation approach generalizes beyond the tested models and query-response pairs.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 6, 2026 (2026-11-06).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T21:36:07.524Z. First observed: 2026-11-09T05:03:57.387Z. Retrieved: 2026-11-12T21:34:55.432Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported evaluation covers four and 142 OpenAssistant query-response pairs; the abstract does not establish how well the results generalize beyond them. The native-speaker comparison is reported for a 72-response diagnostic subset. The supplied evidence is the paper’s abstract, not its full methods or results.
Open question
How well would the authors’ evaluation policy identify these failures beyond the tested models and 142 OpenAssistant query-response pairs?
Source-stated
The authors argue that automatic metrics systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation.
We demonstrate that automatic metrics (BGE-M3, BERTScore) systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation.
Source-stated
The authors evaluated four LLMs on 142 OpenAssistant query-response pairs.
We evaluate four LLMs (Claude Opus 4.8, GPT-5.5, Gemini 2.5-flash, Qwen3.5-35B) on 142 OpenAssistant query-response pairs.
Source-stated
On a 72-response diagnostic subset, native speakers rated GPT-5.5 higher than Claude Opus 4.8, while their full-dataset BGE-M3 means were nearly the same.
On a 72-response diagnostic subset, native speakers rate GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, whereas their full-dataset BGE-M3 means differ by only 0.004.
Source-stated
The authors report that Gemini produces script spillage in Telugu responses and Qwen sometimes produces English-only Kannada responses.
Gemini produces script spillage (16.9% of Telugu responses leak native script), while Qwen collapses to English-only output (16.2% of Kannada responses).
Source-stated
The authors propose assigning zero to empty outputs, script-corrupted responses, and instruction collapses, and argue for combining several evaluation measures.
We introduce a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses
Source-stated
The authors argue that evaluation should combine semantic similarity, script compliance, failure coverage, and native-speaker acceptability.
Our results show that evaluating romanized Indic generation requires a coordinated bundle: semantic similarity, script compliance, failure coverage, and native-speaker acceptability.
Source and contributor context
Beesetty, Madhu Sudhan Rao, Sharma, Surbhi, Maheshwari, Devesh, Meesala, Lokesh · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: a6a256c4-be0a-47ff-839d-243e71854f31.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-12T21:36:07.524Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 69572fded9e817c4d80dc533f592c9b470149d8b7c18af697b10cbc96a2c2b6e. Source text hash: c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550.
Revision and source-check history
Version 1 · 2026-11-12T21:36:07.524Z
arXiv cs.AI: The Invisible Gap in Romanized Indic Generation
The paper’s authors argue that automatic metrics can miss serious failures in romanized Indic generation: producing Hindi, Telugu, or Kannada in Latin characters. They compare four on 142 OpenAssistant query-response pairs and find that metric scores can conceal differences apparent to native speakers. On a 72-response diagnostic subset, native speakers rated GPT-5.5 1.43 points higher than Opus 4.8 on a 12-point rubric, while the models’ full-dataset BGE-M3 means differed by only 0.004. The authors also report distinct output failures: Gemini sometimes leaks native script into Telugu responses, while Qwen sometimes produces English-only Kannada responses. Their proposed response is a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses. They argue evaluation should combine semantic similarity, script compliance, failure coverage, and native-speaker acceptability. The stakes are practical: a score that misses script or instruction failures may not reflect whether an output is usable for people communicating in romanized Indic languages. The paper’s abstract does not establish how well this evaluation approach generalizes beyond the tested models and query-response pairs.
Original source
cs.AI · Article
Source details
Original source date: Nov 6, 2026 (2026-11-06).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T21:36:07.524Z. First observed: 2026-11-09T05:03:57.387Z. Retrieved: 2026-11-12T21:34:55.432Z.
Limits
The reported evaluation covers four and 142 OpenAssistant query-response pairs; the abstract does not establish how well the results generalize beyond them. The native-speaker comparison is reported for a 72-response diagnostic subset. The supplied evidence is the paper’s abstract, not its full methods or results.
Open question
How well would the authors’ evaluation policy identify these failures beyond the tested models and 142 OpenAssistant query-response pairs?
Source-stated
The authors argue that automatic metrics systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation.
We demonstrate that automatic metrics (BGE-M3, BERTScore) systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation.
Source-stated
The authors evaluated four LLMs on 142 OpenAssistant query-response pairs.
We evaluate four LLMs (Claude Opus 4.8, GPT-5.5, Gemini 2.5-flash, Qwen3.5-35B) on 142 OpenAssistant query-response pairs.
Source-stated
On a 72-response diagnostic subset, native speakers rated GPT-5.5 higher than Claude Opus 4.8, while their full-dataset BGE-M3 means were nearly the same.
On a 72-response diagnostic subset, native speakers rate GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, whereas their full-dataset BGE-M3 means differ by only 0.004.
Source-stated
The authors report that Gemini produces script spillage in Telugu responses and Qwen sometimes produces English-only Kannada responses.
Gemini produces script spillage (16.9% of Telugu responses leak native script), while Qwen collapses to English-only output (16.2% of Kannada responses).
Source-stated
The authors propose assigning zero to empty outputs, script-corrupted responses, and instruction collapses, and argue for combining several evaluation measures.
We introduce a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses
Source-stated
The authors argue that evaluation should combine semantic similarity, script compliance, failure coverage, and native-speaker acceptability.
Our results show that evaluating romanized Indic generation requires a coordinated bundle: semantic similarity, script compliance, failure coverage, and native-speaker acceptability.
Source and contributor context
Beesetty, Madhu Sudhan Rao, Sharma, Surbhi, Maheshwari, Devesh, Meesala, Lokesh · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Editorial provenance
Version 1. Publication ID: a6a256c4-be0a-47ff-839d-243e71854f31.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-12T21:36:07.524Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 69572fded9e817c4d80dc533f592c9b470149d8b7c18af697b10cbc96a2c2b6e. Source text hash: c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550.
passed · 2026-11-12T21:36:07.524Z · anthropic/claude-opus-5.5
Publication: a6a256c4-be0a-47ff-839d-243e71854f31
attribution — pass: The reading attributes the claims to the paper's authors on arXiv. Results and recommendations stay framed as what the authors report, argue or propose, and none are presented as established fact. The 'arXiv cs.AI' label matches the publisher field. The paper's primary listing is cs.CL, cross-listed to cs.AI.
Authors:Madhu Sudhan Rao Beesetty, Surbhi Sharma, Devesh Maheshwari, Lokesh Meesala
summary — pass: The summary explains romanized Indic generation in plain terms and keeps the subset versus full-dataset distinction. It also accurately conveys the two failure modes and the proposed evaluation bundle. Minor issues: it softens 'systematically misrepresent' to 'can miss', and BGE-M3 and 'instruction collapses' are left unexplained. Tooltips could cover both terms. Context makes the main point understandable.
typing "aap kaise hain" rather than native script
date — pass: The originalPublished value of 2026-11-06 matches the v1 submission date on arXiv. It is not the later feed date of 2026-11-09.
[Submitted on 6 Nov 2026]
support — pass: Each claim matches a verbatim abstract excerpt. The usability stakes are stated conditionally and follow from the paper's argument about failures that metrics miss. The generalization question and the limitations are fair gaps given an abstract-only evaluation of four models on 142 pairs. No unsupported benefits or costs are introduced.
revealing failures metrics alone cannot detect
media — pass: The selected media is the verified direct link to the arXiv abstract page, which is the original source itself.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A language model with many learned numerical settings, used to process and generate language.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA language model with many learned numerical settings, used to process and generate language.
Definition reference