{"id":"6475199f040abccf5b58e9768f9b0cf82771471f86c03afc3abb1ce910e4da9d","title":"arXiv cs.AI: The Invisible Gap in Romanized Indic Generation","argument":"The paper’s authors argue that automatic metrics can miss serious failures in romanized Indic generation: producing Hindi, Telugu, or Kannada in Latin characters. They compare four LLMs on 142 OpenAssistant query-response pairs and find that metric scores can conceal differences apparent to native speakers. On a 72-response diagnostic subset, native speakers rated GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, while the models’ full-dataset BGE-M3 means differed by only 0.004. The authors also report distinct output failures: Gemini sometimes leaks native script into Telugu responses, while Qwen sometimes produces English-only Kannada responses. Their proposed response is a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses. They argue evaluation should combine semantic similarity, script compliance, failure coverage, and native-speaker acceptability. The stakes are practical: a score that misses script or instruction failures may not reflect whether an output is usable for people communicating in romanized Indic languages. The paper’s abstract does not establish how well this evaluation approach generalizes beyond the tested models and query-response pairs.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported evaluation covers four LLMs and 142 OpenAssistant query-response pairs; the abstract does not establish how well the results generalize beyond them. The native-speaker comparison is reported for a 72-response diagnostic subset. The supplied evidence is the paper’s abstract, not its full methods or results.","speaker":{"name":"Beesetty, Madhu Sudhan Rao, Sharma, Surbhi, Maheshwari, Devesh, Meesala, Lokesh","roleAtSource":"Article author"},"source":{"title":"The Invisible Gap: Romanized Indic Generation Fails Where Automatic Metrics Succeed","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.09085","publishedAt":"2026-11-06","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that automatic metrics systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation.","excerpt":"We demonstrate that automatic metrics (BGE-M3, BERTScore) systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation.","url":"https://arxiv.org/abs/2611.09085","relationship":"Source-stated"},{"text":"The authors evaluated four LLMs on 142 OpenAssistant query-response pairs.","excerpt":"We evaluate four LLMs (Claude Opus 4.8, GPT-5.5, Gemini 2.5-flash, Qwen3.5-35B) on 142 OpenAssistant query-response pairs.","url":"https://arxiv.org/abs/2611.09085","relationship":"Source-stated"},{"text":"On a 72-response diagnostic subset, native speakers rated GPT-5.5 higher than Claude Opus 4.8, while their full-dataset BGE-M3 means were nearly the same.","excerpt":"On a 72-response diagnostic subset, native speakers rate GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, whereas their full-dataset BGE-M3 means differ by only 0.004.","url":"https://arxiv.org/abs/2611.09085","relationship":"Source-stated"},{"text":"The authors report that Gemini produces script spillage in Telugu responses and Qwen sometimes produces English-only Kannada responses.","excerpt":"Gemini produces script spillage (16.9% of Telugu responses leak native script), while Qwen collapses to English-only output (16.2% of Kannada responses).","url":"https://arxiv.org/abs/2611.09085","relationship":"Source-stated"},{"text":"The authors propose assigning zero to empty outputs, script-corrupted responses, and instruction collapses, and argue for combining several evaluation measures.","excerpt":"We introduce a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses","url":"https://arxiv.org/abs/2611.09085","relationship":"Source-stated"},{"text":"The authors argue that evaluation should combine semantic similarity, script compliance, failure coverage, and native-speaker acceptability.","excerpt":"Our results show that evaluating romanized Indic generation requires a coordinated bundle: semantic similarity, script compliance, failure coverage, and native-speaker acceptability.","url":"https://arxiv.org/abs/2611.09085","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09085","checkedAt":"2026-11-13T14:40:09.376Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would the authors’ evaluation policy identify these failures beyond the tested models and 142 OpenAssistant query-response pairs?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: The Invisible Gap in Romanized Indic Generation","summary":"The paper’s authors argue that automatic metrics can miss serious failures in romanized Indic generation: producing Hindi, Telugu, or Kannada in Latin characters. They compare four LLMs on 142 OpenAssistant query-response pairs and find that metric scores can conceal differences apparent to native speakers. On a 72-response diagnostic subset, native speakers rated GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, while the models’ full-dataset BGE-M3 means differed by only 0.004. The authors also report distinct output failures: Gemini sometimes leaks native script into Telugu responses, while Qwen sometimes produces English-only Kannada responses. Their proposed response is a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses. They argue evaluation should combine semantic similarity, script compliance, failure coverage, and native-speaker acceptability. The stakes are practical: a score that misses script or instruction failures may not reflect whether an output is usable for people communicating in romanized Indic languages. The paper’s abstract does not establish how well this evaluation approach generalizes beyond the tested models and query-response pairs.","question":"How well would the authors’ evaluation policy identify these failures beyond the tested models and 142 OpenAssistant query-response pairs?","claims":[{"text":"The authors argue that automatic metrics systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation.","excerpt":"We demonstrate that automatic metrics (BGE-M3, BERTScore) systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation.","support":"source-stated"},{"text":"The authors evaluated four LLMs on 142 OpenAssistant query-response pairs.","excerpt":"We evaluate four LLMs (Claude Opus 4.8, GPT-5.5, Gemini 2.5-flash, Qwen3.5-35B) on 142 OpenAssistant query-response pairs.","support":"source-stated"},{"text":"On a 72-response diagnostic subset, native speakers rated GPT-5.5 higher than Claude Opus 4.8, while their full-dataset BGE-M3 means were nearly the same.","excerpt":"On a 72-response diagnostic subset, native speakers rate GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, whereas their full-dataset BGE-M3 means differ by only 0.004.","support":"source-stated"},{"text":"The authors report that Gemini produces script spillage in Telugu responses and Qwen sometimes produces English-only Kannada responses.","excerpt":"Gemini produces script spillage (16.9% of Telugu responses leak native script), while Qwen collapses to English-only output (16.2% of Kannada responses).","support":"source-stated"},{"text":"The authors propose assigning zero to empty outputs, script-corrupted responses, and instruction collapses, and argue for combining several evaluation measures.","excerpt":"We introduce a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses","support":"source-stated"},{"text":"The authors argue that evaluation should combine semantic similarity, script compliance, failure coverage, and native-speaker acceptability.","excerpt":"Our results show that evaluating romanized Indic generation requires a coordinated bundle: semantic similarity, script compliance, failure coverage, and native-speaker acceptability.","support":"source-stated"}],"limitations":["The reported evaluation covers four LLMs and 142 OpenAssistant query-response pairs; the abstract does not establish how well the results generalize beyond them.","The native-speaker comparison is reported for a 72-response diagnostic subset.","The supplied evidence is the paper’s abstract, not its full methods or results."],"mediaId":"ce018c154b74217c558dc4c5c6b7aa0afad92df60462b77bd4d0f399ffdd315d","correctionReason":null,"id":"6475199f040abccf5b58e9768f9b0cf82771471f86c03afc3abb1ce910e4da9d","publicationId":"a6a256c4-be0a-47ff-839d-243e71854f31","version":1,"sourceUrl":"https://arxiv.org/abs/2611.09085","sourceTitle":"The Invisible Gap: Romanized Indic Generation Fails Where Automatic Metrics Succeed","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-06","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-09T05:03:57.387Z","createdAt":"2026-11-12T21:36:07.524Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"ce018c154b74217c558dc4c5c6b7aa0afad92df60462b77bd4d0f399ffdd315d","kind":"article","url":"https://arxiv.org/abs/2611.09085","sourceUrl":"https://arxiv.org/abs/2611.09085","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T21:34:55.730Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09085","checkedAt":"2026-11-13T14:40:09.376Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"ce018c154b74217c558dc4c5c6b7aa0afad92df60462b77bd4d0f399ffdd315d","confidence":0.7999999999999999,"provider":"typesafe","model":"jev-1.13.0","inputHash":"d4538b14dc8bc0467438282781733f4bbad129e31db14600336e3eb9698b0e7d","decidedAt":"2026-11-12T21:34:57.977Z"},"provenance":{"textHash":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","observationId":"69572fded9e817c4d80dc533f592c9b470149d8b7c18af697b10cbc96a2c2b6e","retrievedAt":"2026-11-12T21:34:55.432Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"3155cb1e-673a-45ae-8463-c44de2aa54a1"},"reviewer":{"model":"gpt-6-luna","runId":"629482a0-6036-4da6-8d2a-5419579995c1"}},"audit":{"status":"passed","reviewedAt":"2026-11-12T21:36:07.524Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Beesetty, Madhu Sudhan Rao","url":null,"kind":"unknown"},{"name":"Sharma, Surbhi","url":null,"kind":"unknown"},{"name":"Maheshwari, Devesh","url":null,"kind":"unknown"},{"name":"Meesala, Lokesh","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Beesetty, Madhu Sudhan Rao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Beesetty, Madhu Sudhan Rao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09085","locator":"head/meta[name=author|citation_author]"},"assertionId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T14:40:09.376Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093","sourceUrl":"https://arxiv.org/abs/2611.09085","contentSha256":"146302ef6a64bdbfc0d05b7a3ad10d446459a697132be2550a9c937a36f6cf13","textHash":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","observedAt":"2026-11-13T14:40:09.376Z"}},{"actor":{"name":"Sharma, Surbhi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sharma, Surbhi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09085","locator":"head/meta[name=author|citation_author]"},"assertionId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T14:40:09.376Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093","sourceUrl":"https://arxiv.org/abs/2611.09085","contentSha256":"146302ef6a64bdbfc0d05b7a3ad10d446459a697132be2550a9c937a36f6cf13","textHash":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","observedAt":"2026-11-13T14:40:09.376Z"}},{"actor":{"name":"Maheshwari, Devesh","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Maheshwari, Devesh","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09085","locator":"head/meta[name=author|citation_author]"},"assertionId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T14:40:09.376Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093","sourceUrl":"https://arxiv.org/abs/2611.09085","contentSha256":"146302ef6a64bdbfc0d05b7a3ad10d446459a697132be2550a9c937a36f6cf13","textHash":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","observedAt":"2026-11-13T14:40:09.376Z"}},{"actor":{"name":"Meesala, Lokesh","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Meesala, Lokesh","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09085","locator":"head/meta[name=author|citation_author]"},"assertionId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T14:40:09.376Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093","sourceUrl":"https://arxiv.org/abs/2611.09085","contentSha256":"146302ef6a64bdbfc0d05b7a3ad10d446459a697132be2550a9c937a36f6cf13","textHash":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","observedAt":"2026-11-13T14:40:09.376Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.09085","snapshotId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093","contentSha256":"146302ef6a64bdbfc0d05b7a3ad10d446459a697132be2550a9c937a36f6cf13","textHash":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","observedAt":"2026-11-13T14:40:09.376Z","basis":"publisher-page-metadata","sourceLinks":43,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/6475199f040abccf5b58e9768f9b0cf82771471f86c03afc3abb1ce910e4da9d/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-the-invisible-gap-in-romanized-indic-generation-6475199f","formats":{"html":"/readings/arxiv-cs-ai-the-invisible-gap-in-romanized-indic-generation-6475199f","markdown":"/readings/arxiv-cs-ai-the-invisible-gap-in-romanized-indic-generation-6475199f.md","json":"/readings/arxiv-cs-ai-the-invisible-gap-in-romanized-indic-generation-6475199f.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"6475199f040abccf5b58e9768f9b0cf82771471f86c03afc3abb1ce910e4da9d","revision":"a6a256c4-be0a-47ff-839d-243e71854f31","revisionKind":"publication-id","title":"arXiv cs.AI: The Invisible Gap in Romanized Indic Generation","url":"https://escapemeridian.com/readings/arxiv-cs-ai-the-invisible-gap-in-romanized-indic-generation-6475199f"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.09085","revision":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","revisionKind":"sha256-source-text","title":"The Invisible Gap: Romanized Indic Generation Fails Where Automatic Metrics Succeed","url":"https://arxiv.org/abs/2611.09085"},"status":"partial","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"6475199f040abccf5b58e9768f9b0cf82771471f86c03afc3abb1ce910e4da9d","title":"arXiv cs.AI: The Invisible Gap in Romanized Indic Generation","argument":"The paper’s authors argue that automatic metrics can miss serious failures in romanized Indic generation: producing Hindi, Telugu, or Kannada in Latin characters. They compare four LLMs on 142 OpenAssistant query-response pairs and find that metric scores can conceal differences apparent to native speakers. On a 72-response diagnostic subset, native speakers rated GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, while the models’ full-dataset BGE-M3 means differed by only 0.004. The authors also report distinct output failures: Gemini sometimes leaks native script into Telugu responses, while Qwen sometimes produces English-only Kannada responses. Their proposed response is a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses. They argue evaluation should combine semantic similarity, script compliance, failure coverage, and native-speaker acceptability. The stakes are practical: a score that misses script or instruction failures may not reflect whether an output is usable for people communicating in romanized Indic languages. The paper’s abstract does not establish how well this evaluation approach generalizes beyond the tested models and query-response pairs.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported evaluation covers four LLMs and 142 OpenAssistant query-response pairs; the abstract does not establish how well the results generalize beyond them. The native-speaker comparison is reported for a 72-response diagnostic subset. The supplied evidence is the paper’s abstract, not its full methods or results.","speaker":{"name":"Beesetty, Madhu Sudhan Rao, Sharma, Surbhi, Maheshwari, Devesh, Meesala, Lokesh","roleAtSource":"Article author"},"source":{"title":"The Invisible Gap: Romanized Indic Generation Fails Where Automatic Metrics Succeed","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.09085","publishedAt":"2026-11-06","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that automatic metrics systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation.","excerpt":"We demonstrate that automatic metrics (BGE-M3, BERTScore) systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation.","url":"https://arxiv.org/abs/2611.09085","relationship":"Source-stated"},{"text":"The authors evaluated four LLMs on 142 OpenAssistant query-response pairs.","excerpt":"We evaluate four LLMs (Claude Opus 4.8, GPT-5.5, Gemini 2.5-flash, Qwen3.5-35B) on 142 OpenAssistant query-response pairs.","url":"https://arxiv.org/abs/2611.09085","relationship":"Source-stated"},{"text":"On a 72-response diagnostic subset, native speakers rated GPT-5.5 higher than Claude Opus 4.8, while their full-dataset BGE-M3 means were nearly the same.","excerpt":"On a 72-response diagnostic subset, native speakers rate GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, whereas their full-dataset BGE-M3 means differ by only 0.004.","url":"https://arxiv.org/abs/2611.09085","relationship":"Source-stated"},{"text":"The authors report that Gemini produces script spillage in Telugu responses and Qwen sometimes produces English-only Kannada responses.","excerpt":"Gemini produces script spillage (16.9% of Telugu responses leak native script), while Qwen collapses to English-only output (16.2% of Kannada responses).","url":"https://arxiv.org/abs/2611.09085","relationship":"Source-stated"},{"text":"The authors propose assigning zero to empty outputs, script-corrupted responses, and instruction collapses, and argue for combining several evaluation measures.","excerpt":"We introduce a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses","url":"https://arxiv.org/abs/2611.09085","relationship":"Source-stated"},{"text":"The authors argue that evaluation should combine semantic similarity, script compliance, failure coverage, and native-speaker acceptability.","excerpt":"Our results show that evaluating romanized Indic generation requires a coordinated bundle: semantic similarity, script compliance, failure coverage, and native-speaker acceptability.","url":"https://arxiv.org/abs/2611.09085","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09085","checkedAt":"2026-11-13T14:40:09.376Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would the authors’ evaluation policy identify these failures beyond the tested models and 142 OpenAssistant query-response pairs?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: The Invisible Gap in Romanized Indic Generation","summary":"The paper’s authors argue that automatic metrics can miss serious failures in romanized Indic generation: producing Hindi, Telugu, or Kannada in Latin characters. They compare four LLMs on 142 OpenAssistant query-response pairs and find that metric scores can conceal differences apparent to native speakers. On a 72-response diagnostic subset, native speakers rated GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, while the models’ full-dataset BGE-M3 means differed by only 0.004. The authors also report distinct output failures: Gemini sometimes leaks native script into Telugu responses, while Qwen sometimes produces English-only Kannada responses. Their proposed response is a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses. They argue evaluation should combine semantic similarity, script compliance, failure coverage, and native-speaker acceptability. The stakes are practical: a score that misses script or instruction failures may not reflect whether an output is usable for people communicating in romanized Indic languages. The paper’s abstract does not establish how well this evaluation approach generalizes beyond the tested models and query-response pairs.","question":"How well would the authors’ evaluation policy identify these failures beyond the tested models and 142 OpenAssistant query-response pairs?","claims":[{"text":"The authors argue that automatic metrics systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation.","excerpt":"We demonstrate that automatic metrics (BGE-M3, BERTScore) systematically misrepresent the quality of romanized Hindi, Telugu, and Kannada generation.","support":"source-stated"},{"text":"The authors evaluated four LLMs on 142 OpenAssistant query-response pairs.","excerpt":"We evaluate four LLMs (Claude Opus 4.8, GPT-5.5, Gemini 2.5-flash, Qwen3.5-35B) on 142 OpenAssistant query-response pairs.","support":"source-stated"},{"text":"On a 72-response diagnostic subset, native speakers rated GPT-5.5 higher than Claude Opus 4.8, while their full-dataset BGE-M3 means were nearly the same.","excerpt":"On a 72-response diagnostic subset, native speakers rate GPT-5.5 1.43 points higher than Claude Opus 4.8 on a 12-point rubric, whereas their full-dataset BGE-M3 means differ by only 0.004.","support":"source-stated"},{"text":"The authors report that Gemini produces script spillage in Telugu responses and Qwen sometimes produces English-only Kannada responses.","excerpt":"Gemini produces script spillage (16.9% of Telugu responses leak native script), while Qwen collapses to English-only output (16.2% of Kannada responses).","support":"source-stated"},{"text":"The authors propose assigning zero to empty outputs, script-corrupted responses, and instruction collapses, and argue for combining several evaluation measures.","excerpt":"We introduce a failure-aware evaluation policy that assigns zero to empty outputs, script-corrupted responses, and instruction collapses","support":"source-stated"},{"text":"The authors argue that evaluation should combine semantic similarity, script compliance, failure coverage, and native-speaker acceptability.","excerpt":"Our results show that evaluating romanized Indic generation requires a coordinated bundle: semantic similarity, script compliance, failure coverage, and native-speaker acceptability.","support":"source-stated"}],"limitations":["The reported evaluation covers four LLMs and 142 OpenAssistant query-response pairs; the abstract does not establish how well the results generalize beyond them.","The native-speaker comparison is reported for a 72-response diagnostic subset.","The supplied evidence is the paper’s abstract, not its full methods or results."],"mediaId":"ce018c154b74217c558dc4c5c6b7aa0afad92df60462b77bd4d0f399ffdd315d","correctionReason":null,"id":"6475199f040abccf5b58e9768f9b0cf82771471f86c03afc3abb1ce910e4da9d","publicationId":"a6a256c4-be0a-47ff-839d-243e71854f31","version":1,"sourceUrl":"https://arxiv.org/abs/2611.09085","sourceTitle":"The Invisible Gap: Romanized Indic Generation Fails Where Automatic Metrics Succeed","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-06","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-09T05:03:57.387Z","createdAt":"2026-11-12T21:36:07.524Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"ce018c154b74217c558dc4c5c6b7aa0afad92df60462b77bd4d0f399ffdd315d","kind":"article","url":"https://arxiv.org/abs/2611.09085","sourceUrl":"https://arxiv.org/abs/2611.09085","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T21:34:55.730Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.09085","checkedAt":"2026-11-13T14:40:09.376Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"ce018c154b74217c558dc4c5c6b7aa0afad92df60462b77bd4d0f399ffdd315d","confidence":0.7999999999999999,"provider":"typesafe","model":"jev-1.13.0","inputHash":"d4538b14dc8bc0467438282781733f4bbad129e31db14600336e3eb9698b0e7d","decidedAt":"2026-11-12T21:34:57.977Z"},"provenance":{"textHash":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","observationId":"69572fded9e817c4d80dc533f592c9b470149d8b7c18af697b10cbc96a2c2b6e","retrievedAt":"2026-11-12T21:34:55.432Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"3155cb1e-673a-45ae-8463-c44de2aa54a1"},"reviewer":{"model":"gpt-6-luna","runId":"629482a0-6036-4da6-8d2a-5419579995c1"}},"audit":{"status":"passed","reviewedAt":"2026-11-12T21:36:07.524Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Beesetty, Madhu Sudhan Rao","url":null,"kind":"unknown"},{"name":"Sharma, Surbhi","url":null,"kind":"unknown"},{"name":"Maheshwari, Devesh","url":null,"kind":"unknown"},{"name":"Meesala, Lokesh","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Beesetty, Madhu Sudhan Rao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Beesetty, Madhu Sudhan Rao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09085","locator":"head/meta[name=author|citation_author]"},"assertionId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T14:40:09.376Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093","sourceUrl":"https://arxiv.org/abs/2611.09085","contentSha256":"146302ef6a64bdbfc0d05b7a3ad10d446459a697132be2550a9c937a36f6cf13","textHash":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","observedAt":"2026-11-13T14:40:09.376Z"}},{"actor":{"name":"Sharma, Surbhi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sharma, Surbhi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09085","locator":"head/meta[name=author|citation_author]"},"assertionId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T14:40:09.376Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093","sourceUrl":"https://arxiv.org/abs/2611.09085","contentSha256":"146302ef6a64bdbfc0d05b7a3ad10d446459a697132be2550a9c937a36f6cf13","textHash":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","observedAt":"2026-11-13T14:40:09.376Z"}},{"actor":{"name":"Maheshwari, Devesh","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Maheshwari, Devesh","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09085","locator":"head/meta[name=author|citation_author]"},"assertionId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T14:40:09.376Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093","sourceUrl":"https://arxiv.org/abs/2611.09085","contentSha256":"146302ef6a64bdbfc0d05b7a3ad10d446459a697132be2550a9c937a36f6cf13","textHash":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","observedAt":"2026-11-13T14:40:09.376Z"}},{"actor":{"name":"Meesala, Lokesh","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Meesala, Lokesh","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.09085","locator":"head/meta[name=author|citation_author]"},"assertionId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-13T14:40:09.376Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093","sourceUrl":"https://arxiv.org/abs/2611.09085","contentSha256":"146302ef6a64bdbfc0d05b7a3ad10d446459a697132be2550a9c937a36f6cf13","textHash":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","observedAt":"2026-11-13T14:40:09.376Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.09085","snapshotId":"060b67aab6e0e1cd16835c2bc45e763c745f06e454d8304d2ea04efad57a7093","contentSha256":"146302ef6a64bdbfc0d05b7a3ad10d446459a697132be2550a9c937a36f6cf13","textHash":"c99785667cbc28ad5e5ccc3152f38f765f7babdb2bce7db3cd72a73ecf998550","observedAt":"2026-11-13T14:40:09.376Z","basis":"publisher-page-metadata","sourceLinks":43,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/6475199f040abccf5b58e9768f9b0cf82771471f86c03afc3abb1ce910e4da9d/source"}}}],"audits":[{"publicationId":"a6a256c4-be0a-47ff-839d-243e71854f31","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the claims to the paper's authors on arXiv. Results and recommendations stay framed as what the authors report, argue or propose, and none are presented as established fact. The 'arXiv cs.AI' label matches the publisher field. The paper's primary listing is cs.CL, cross-listed to cs.AI.","excerpt":"Authors:Madhu Sudhan Rao Beesetty, Surbhi Sharma, Devesh Maheshwari, Lokesh Meesala"},{"dimension":"summary","status":"pass","explanation":"The summary explains romanized Indic generation in plain terms and keeps the subset versus full-dataset distinction. It also accurately conveys the two failure modes and the proposed evaluation bundle. Minor issues: it softens 'systematically misrepresent' to 'can miss', and BGE-M3 and 'instruction collapses' are left unexplained. Tooltips could cover both terms. Context makes the main point understandable.","excerpt":"typing \"aap kaise hain\" rather than native script"},{"dimension":"date","status":"pass","explanation":"The originalPublished value of 2026-11-06 matches the v1 submission date on arXiv. It is not the later feed date of 2026-11-09.","excerpt":"[Submitted on 6 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim matches a verbatim abstract excerpt. The usability stakes are stated conditionally and follow from the paper's argument about failures that metrics miss. The generalization question and the limitations are fair gaps given an abstract-only evaluation of four models on 142 pairs. No unsupported benefits or costs are introduced.","excerpt":"revealing failures metrics alone cannot detect"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the arXiv abstract page, which is the original source itself.","excerpt":null}],"createdAt":"2026-11-12T21:36:07.524Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}