{"id":"060857accc4b789726afbfa061d8f1faa1abf24abc1698349fe90db13335584e","title":"arXiv cs.AI paper measures the token tax across languages of India","argument":"The authors of “The Indic Token Tax in 2026” argue that tokenizers can make the same content cost more, run more slowly, and occupy more of a context window in some languages than in English. They compare nine public tokenizers across 25 languages of India and human-translated Hinglish-English social media pairs. On GPT-4o’s tokenizer, the same content takes between 1.57 times as many tokens for Hindi and 5.01 times for Odia as it does in English; for Santali in Ol Chiki script, it takes 13.7 times as many. The paper gives an illustrative support-bot estimate of INR 56,640 per month for English chats and INR 102,829 for Marathi, while stressing that these are estimates, not measured production costs. It also reports that no tested tokenizer places more than 35% of word-internal boundaries on morpheme boundaries in the Hindi and Marathi comparison. The authors say a companion paper presents a tokenizer intended to reduce this disparity.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The support-bot figures are illustrative estimates based on official list prices accessed on 5 November 2026 and stated assumptions, not measured production costs. The reported comparisons cover the languages and translated datasets specified in the abstract; the evidence here does not establish how the results generalize beyond them. The evidence is the paper’s abstract and does not provide further methodological detail about the measurements.","speaker":{"name":"Arimbur, Johin Johny, Ghosh, Ankur Ashit, Sam, Shervin","roleAtSource":"Article author"},"source":{"title":"The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.13058","publishedAt":"2026-11-09","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that token-count differences can make the same content more expensive, slower, and take up more of a fixed context window in some languages.","excerpt":"a tokenizer that needs more tokens for the same content in one language makes that language more expensive, slower, and able to fit less text in a fixed context window","url":"https://arxiv.org/abs/2611.13058","relationship":"Source-stated"},{"text":"The authors compare nine public tokenizers across the Indian languages available in FLORES-200 and a set of human-translated Hinglish-English social media pairs.","excerpt":"for nine public tokenizers, including those of GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, Mistral-Nemo and four Sarvam models, on all 25 languages of India available in FLORES-200","url":"https://arxiv.org/abs/2611.13058","relationship":"Source-stated"},{"text":"On GPT-4o’s tokenizer, the same content uses 1.57 times as many tokens for Hindi, 5.01 times for Odia, and 13.7 times for Santali in Ol Chiki script as for English.","excerpt":"On GPT-4o's tokenizer the same content needs between 1.57 times (Hindi) and 5.01 times (Odia) the tokens of English, and 13.7 times for Santali in Ol Chiki script","url":"https://arxiv.org/abs/2611.13058","relationship":"Source-stated"},{"text":"The paper’s illustrative support-bot scenario estimates different monthly costs for English and Marathi chats, but the authors say these are not measured production costs.","excerpt":"these are estimates, not measured production costs","url":"https://arxiv.org/abs/2611.13058","relationship":"Source-stated"},{"text":"In the Hindi and Marathi morphological segmentation comparison, no tested tokenizer places more than 35% of word-internal boundaries on morpheme boundaries.","excerpt":"no tokenizer places more than 35% of its word-internal boundaries on morpheme boundaries (boundary F1 at most 0.48)","url":"https://arxiv.org/abs/2611.13058","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.13058","checkedAt":"2026-11-12T02:24:12.819Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How closely would the paper’s illustrative support-bot cost estimates match costs measured in production?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper measures the token tax across languages of India","summary":"The authors of “The Indic Token Tax in 2026” argue that tokenizers can make the same content cost more, run more slowly, and occupy more of a context window in some languages than in English. They compare nine public tokenizers across 25 languages of India and human-translated Hinglish-English social media pairs. On GPT-4o’s tokenizer, the same content takes between 1.57 times as many tokens for Hindi and 5.01 times for Odia as it does in English; for Santali in Ol Chiki script, it takes 13.7 times as many. The paper gives an illustrative support-bot estimate of INR 56,640 per month for English chats and INR 102,829 for Marathi, while stressing that these are estimates, not measured production costs. It also reports that no tested tokenizer places more than 35% of word-internal boundaries on morpheme boundaries in the Hindi and Marathi comparison. The authors say a companion paper presents a tokenizer intended to reduce this disparity.","question":"How closely would the paper’s illustrative support-bot cost estimates match costs measured in production?","claims":[{"text":"The authors argue that token-count differences can make the same content more expensive, slower, and take up more of a fixed context window in some languages.","excerpt":"a tokenizer that needs more tokens for the same content in one language makes that language more expensive, slower, and able to fit less text in a fixed context window","support":"source-stated"},{"text":"The authors compare nine public tokenizers across the Indian languages available in FLORES-200 and a set of human-translated Hinglish-English social media pairs.","excerpt":"for nine public tokenizers, including those of GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, Mistral-Nemo and four Sarvam models, on all 25 languages of India available in FLORES-200","support":"source-stated"},{"text":"On GPT-4o’s tokenizer, the same content uses 1.57 times as many tokens for Hindi, 5.01 times for Odia, and 13.7 times for Santali in Ol Chiki script as for English.","excerpt":"On GPT-4o's tokenizer the same content needs between 1.57 times (Hindi) and 5.01 times (Odia) the tokens of English, and 13.7 times for Santali in Ol Chiki script","support":"source-stated"},{"text":"The paper’s illustrative support-bot scenario estimates different monthly costs for English and Marathi chats, but the authors say these are not measured production costs.","excerpt":"these are estimates, not measured production costs","support":"source-stated"},{"text":"In the Hindi and Marathi morphological segmentation comparison, no tested tokenizer places more than 35% of word-internal boundaries on morpheme boundaries.","excerpt":"no tokenizer places more than 35% of its word-internal boundaries on morpheme boundaries (boundary F1 at most 0.48)","support":"source-stated"}],"limitations":["The support-bot figures are illustrative estimates based on official list prices accessed on 5 November 2026 and stated assumptions, not measured production costs.","The reported comparisons cover the languages and translated datasets specified in the abstract; the evidence here does not establish how the results generalize beyond them.","The evidence is the paper’s abstract and does not provide further methodological detail about the measurements."],"mediaId":"e226fc6281a69b5de998212a7865ae3b4864ff309f6d398a6a50e8f3ba69c2ec","correctionReason":null,"id":"060857accc4b789726afbfa061d8f1faa1abf24abc1698349fe90db13335584e","publicationId":"efbd4fae-ea35-4204-8064-1a6ec7f0a4df","version":1,"sourceUrl":"https://arxiv.org/abs/2611.13058","sourceTitle":"The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-09","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-11T05:06:30.177Z","createdAt":"2026-11-12T00:16:39.611Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"e226fc6281a69b5de998212a7865ae3b4864ff309f6d398a6a50e8f3ba69c2ec","kind":"article","url":"https://arxiv.org/abs/2611.13058","sourceUrl":"https://arxiv.org/abs/2611.13058","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T00:02:09.934Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.13058","checkedAt":"2026-11-12T02:24:12.819Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"e226fc6281a69b5de998212a7865ae3b4864ff309f6d398a6a50e8f3ba69c2ec","confidence":0.94,"provider":"typesafe","model":"jev-1.13.0","inputHash":"b2474d8e3aa5e2e9c4721b45035df1c6cdd871e58fc89ff5443dfd99d8e9e12f","decidedAt":"2026-11-12T00:02:10.698Z"},"provenance":{"textHash":"22ab36abdb3bf5ca637d15dcc3e985808dd3180ae9454f888c2c58e95e7e2f73","observationId":"4dde062eb75b78d1cf8cbe9f15883a256c6ee8450bd6f1360e0e352ece514084","retrievedAt":"2026-11-12T00:02:09.838Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"ba7b98c9-d148-41b8-8e53-a87fe193676b"},"reviewer":{"model":"gpt-6-luna","runId":"0bad1a23-53cd-4a00-8913-f66167d1ce0c"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T00:16:39.611Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Arimbur, Johin Johny","url":null,"kind":"unknown"},{"name":"Ghosh, Ankur Ashit","url":null,"kind":"unknown"},{"name":"Sam, Shervin","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Arimbur, Johin Johny","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Arimbur, Johin Johny","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13058","locator":"head/meta[name=author|citation_author]"},"assertionId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T02:24:12.819Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8","sourceUrl":"https://arxiv.org/abs/2611.13058","contentSha256":"162e863cb5105520a25815d294d684f4248b6016bcff4dabe944ff2137386ae1","textHash":"22ab36abdb3bf5ca637d15dcc3e985808dd3180ae9454f888c2c58e95e7e2f73","observedAt":"2026-11-12T02:24:12.819Z"}},{"actor":{"name":"Ghosh, Ankur Ashit","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ghosh, Ankur Ashit","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13058","locator":"head/meta[name=author|citation_author]"},"assertionId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T02:24:12.819Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8","sourceUrl":"https://arxiv.org/abs/2611.13058","contentSha256":"162e863cb5105520a25815d294d684f4248b6016bcff4dabe944ff2137386ae1","textHash":"22ab36abdb3bf5ca637d15dcc3e985808dd3180ae9454f888c2c58e95e7e2f73","observedAt":"2026-11-12T02:24:12.819Z"}},{"actor":{"name":"Sam, Shervin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sam, Shervin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13058","locator":"head/meta[name=author|citation_author]"},"assertionId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T02:24:12.819Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8","sourceUrl":"https://arxiv.org/abs/2611.13058","contentSha256":"162e863cb5105520a25815d294d684f4248b6016bcff4dabe944ff2137386ae1","textHash":"22ab36abdb3bf5ca637d15dcc3e985808dd3180ae9454f888c2c58e95e7e2f73","observedAt":"2026-11-12T02:24:12.819Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.13058","snapshotId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8","contentSha256":"162e863cb5105520a25815d294d684f4248b6016bcff4dabe944ff2137386ae1","textHash":"22ab36abdb3bf5ca637d15dcc3e985808dd3180ae9454f888c2c58e95e7e2f73","observedAt":"2026-11-12T02:24:12.819Z","basis":"publisher-page-metadata","sourceLinks":43,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/060857accc4b789726afbfa061d8f1faa1abf24abc1698349fe90db13335584e/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-measures-the-token-tax-across-languages-of-060857ac","formats":{"html":"/readings/arxiv-cs-ai-paper-measures-the-token-tax-across-languages-of-060857ac","markdown":"/readings/arxiv-cs-ai-paper-measures-the-token-tax-across-languages-of-060857ac.md","json":"/readings/arxiv-cs-ai-paper-measures-the-token-tax-across-languages-of-060857ac.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-tokens","name":"Model tokens","explanation":"Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.","referenceUrl":"https://developers.google.com/machine-learning/glossary#token"},{"id":"term-context-window","name":"Context window","explanation":"How much text a model can take in and consider at one time.","referenceUrl":"https://developers.google.com/machine-learning/glossary#context-window"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"060857accc4b789726afbfa061d8f1faa1abf24abc1698349fe90db13335584e","revision":"efbd4fae-ea35-4204-8064-1a6ec7f0a4df","revisionKind":"publication-id","title":"arXiv cs.AI paper measures the token tax across languages of India","url":"https://escapemeridian.com/readings/arxiv-cs-ai-paper-measures-the-token-tax-across-languages-of-060857ac"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.13058","revision":"22ab36abdb3bf5ca637d15dcc3e985808dd3180ae9454f888c2c58e95e7e2f73","revisionKind":"sha256-source-text","title":"The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India","url":"https://arxiv.org/abs/2611.13058"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"060857accc4b789726afbfa061d8f1faa1abf24abc1698349fe90db13335584e","title":"arXiv cs.AI paper measures the token tax across languages of India","argument":"The authors of “The Indic Token Tax in 2026” argue that tokenizers can make the same content cost more, run more slowly, and occupy more of a context window in some languages than in English. They compare nine public tokenizers across 25 languages of India and human-translated Hinglish-English social media pairs. On GPT-4o’s tokenizer, the same content takes between 1.57 times as many tokens for Hindi and 5.01 times for Odia as it does in English; for Santali in Ol Chiki script, it takes 13.7 times as many. The paper gives an illustrative support-bot estimate of INR 56,640 per month for English chats and INR 102,829 for Marathi, while stressing that these are estimates, not measured production costs. It also reports that no tested tokenizer places more than 35% of word-internal boundaries on morpheme boundaries in the Hindi and Marathi comparison. The authors say a companion paper presents a tokenizer intended to reduce this disparity.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The support-bot figures are illustrative estimates based on official list prices accessed on 5 November 2026 and stated assumptions, not measured production costs. The reported comparisons cover the languages and translated datasets specified in the abstract; the evidence here does not establish how the results generalize beyond them. The evidence is the paper’s abstract and does not provide further methodological detail about the measurements.","speaker":{"name":"Arimbur, Johin Johny, Ghosh, Ankur Ashit, Sam, Shervin","roleAtSource":"Article author"},"source":{"title":"The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.13058","publishedAt":"2026-11-09","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that token-count differences can make the same content more expensive, slower, and take up more of a fixed context window in some languages.","excerpt":"a tokenizer that needs more tokens for the same content in one language makes that language more expensive, slower, and able to fit less text in a fixed context window","url":"https://arxiv.org/abs/2611.13058","relationship":"Source-stated"},{"text":"The authors compare nine public tokenizers across the Indian languages available in FLORES-200 and a set of human-translated Hinglish-English social media pairs.","excerpt":"for nine public tokenizers, including those of GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, Mistral-Nemo and four Sarvam models, on all 25 languages of India available in FLORES-200","url":"https://arxiv.org/abs/2611.13058","relationship":"Source-stated"},{"text":"On GPT-4o’s tokenizer, the same content uses 1.57 times as many tokens for Hindi, 5.01 times for Odia, and 13.7 times for Santali in Ol Chiki script as for English.","excerpt":"On GPT-4o's tokenizer the same content needs between 1.57 times (Hindi) and 5.01 times (Odia) the tokens of English, and 13.7 times for Santali in Ol Chiki script","url":"https://arxiv.org/abs/2611.13058","relationship":"Source-stated"},{"text":"The paper’s illustrative support-bot scenario estimates different monthly costs for English and Marathi chats, but the authors say these are not measured production costs.","excerpt":"these are estimates, not measured production costs","url":"https://arxiv.org/abs/2611.13058","relationship":"Source-stated"},{"text":"In the Hindi and Marathi morphological segmentation comparison, no tested tokenizer places more than 35% of word-internal boundaries on morpheme boundaries.","excerpt":"no tokenizer places more than 35% of its word-internal boundaries on morpheme boundaries (boundary F1 at most 0.48)","url":"https://arxiv.org/abs/2611.13058","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.13058","checkedAt":"2026-11-12T02:24:12.819Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How closely would the paper’s illustrative support-bot cost estimates match costs measured in production?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI paper measures the token tax across languages of India","summary":"The authors of “The Indic Token Tax in 2026” argue that tokenizers can make the same content cost more, run more slowly, and occupy more of a context window in some languages than in English. They compare nine public tokenizers across 25 languages of India and human-translated Hinglish-English social media pairs. On GPT-4o’s tokenizer, the same content takes between 1.57 times as many tokens for Hindi and 5.01 times for Odia as it does in English; for Santali in Ol Chiki script, it takes 13.7 times as many. The paper gives an illustrative support-bot estimate of INR 56,640 per month for English chats and INR 102,829 for Marathi, while stressing that these are estimates, not measured production costs. It also reports that no tested tokenizer places more than 35% of word-internal boundaries on morpheme boundaries in the Hindi and Marathi comparison. The authors say a companion paper presents a tokenizer intended to reduce this disparity.","question":"How closely would the paper’s illustrative support-bot cost estimates match costs measured in production?","claims":[{"text":"The authors argue that token-count differences can make the same content more expensive, slower, and take up more of a fixed context window in some languages.","excerpt":"a tokenizer that needs more tokens for the same content in one language makes that language more expensive, slower, and able to fit less text in a fixed context window","support":"source-stated"},{"text":"The authors compare nine public tokenizers across the Indian languages available in FLORES-200 and a set of human-translated Hinglish-English social media pairs.","excerpt":"for nine public tokenizers, including those of GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, Mistral-Nemo and four Sarvam models, on all 25 languages of India available in FLORES-200","support":"source-stated"},{"text":"On GPT-4o’s tokenizer, the same content uses 1.57 times as many tokens for Hindi, 5.01 times for Odia, and 13.7 times for Santali in Ol Chiki script as for English.","excerpt":"On GPT-4o's tokenizer the same content needs between 1.57 times (Hindi) and 5.01 times (Odia) the tokens of English, and 13.7 times for Santali in Ol Chiki script","support":"source-stated"},{"text":"The paper’s illustrative support-bot scenario estimates different monthly costs for English and Marathi chats, but the authors say these are not measured production costs.","excerpt":"these are estimates, not measured production costs","support":"source-stated"},{"text":"In the Hindi and Marathi morphological segmentation comparison, no tested tokenizer places more than 35% of word-internal boundaries on morpheme boundaries.","excerpt":"no tokenizer places more than 35% of its word-internal boundaries on morpheme boundaries (boundary F1 at most 0.48)","support":"source-stated"}],"limitations":["The support-bot figures are illustrative estimates based on official list prices accessed on 5 November 2026 and stated assumptions, not measured production costs.","The reported comparisons cover the languages and translated datasets specified in the abstract; the evidence here does not establish how the results generalize beyond them.","The evidence is the paper’s abstract and does not provide further methodological detail about the measurements."],"mediaId":"e226fc6281a69b5de998212a7865ae3b4864ff309f6d398a6a50e8f3ba69c2ec","correctionReason":null,"id":"060857accc4b789726afbfa061d8f1faa1abf24abc1698349fe90db13335584e","publicationId":"efbd4fae-ea35-4204-8064-1a6ec7f0a4df","version":1,"sourceUrl":"https://arxiv.org/abs/2611.13058","sourceTitle":"The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-09","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-11T05:06:30.177Z","createdAt":"2026-11-12T00:16:39.611Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"e226fc6281a69b5de998212a7865ae3b4864ff309f6d398a6a50e8f3ba69c2ec","kind":"article","url":"https://arxiv.org/abs/2611.13058","sourceUrl":"https://arxiv.org/abs/2611.13058","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T00:02:09.934Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.13058","checkedAt":"2026-11-12T02:24:12.819Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"e226fc6281a69b5de998212a7865ae3b4864ff309f6d398a6a50e8f3ba69c2ec","confidence":0.94,"provider":"typesafe","model":"jev-1.13.0","inputHash":"b2474d8e3aa5e2e9c4721b45035df1c6cdd871e58fc89ff5443dfd99d8e9e12f","decidedAt":"2026-11-12T00:02:10.698Z"},"provenance":{"textHash":"22ab36abdb3bf5ca637d15dcc3e985808dd3180ae9454f888c2c58e95e7e2f73","observationId":"4dde062eb75b78d1cf8cbe9f15883a256c6ee8450bd6f1360e0e352ece514084","retrievedAt":"2026-11-12T00:02:09.838Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"ba7b98c9-d148-41b8-8e53-a87fe193676b"},"reviewer":{"model":"gpt-6-luna","runId":"0bad1a23-53cd-4a00-8913-f66167d1ce0c"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T00:16:39.611Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Arimbur, Johin Johny","url":null,"kind":"unknown"},{"name":"Ghosh, Ankur Ashit","url":null,"kind":"unknown"},{"name":"Sam, Shervin","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Arimbur, Johin Johny","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Arimbur, Johin Johny","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13058","locator":"head/meta[name=author|citation_author]"},"assertionId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T02:24:12.819Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8","sourceUrl":"https://arxiv.org/abs/2611.13058","contentSha256":"162e863cb5105520a25815d294d684f4248b6016bcff4dabe944ff2137386ae1","textHash":"22ab36abdb3bf5ca637d15dcc3e985808dd3180ae9454f888c2c58e95e7e2f73","observedAt":"2026-11-12T02:24:12.819Z"}},{"actor":{"name":"Ghosh, Ankur Ashit","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ghosh, Ankur Ashit","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13058","locator":"head/meta[name=author|citation_author]"},"assertionId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T02:24:12.819Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8","sourceUrl":"https://arxiv.org/abs/2611.13058","contentSha256":"162e863cb5105520a25815d294d684f4248b6016bcff4dabe944ff2137386ae1","textHash":"22ab36abdb3bf5ca637d15dcc3e985808dd3180ae9454f888c2c58e95e7e2f73","observedAt":"2026-11-12T02:24:12.819Z"}},{"actor":{"name":"Sam, Shervin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sam, Shervin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.13058","locator":"head/meta[name=author|citation_author]"},"assertionId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T02:24:12.819Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8","sourceUrl":"https://arxiv.org/abs/2611.13058","contentSha256":"162e863cb5105520a25815d294d684f4248b6016bcff4dabe944ff2137386ae1","textHash":"22ab36abdb3bf5ca637d15dcc3e985808dd3180ae9454f888c2c58e95e7e2f73","observedAt":"2026-11-12T02:24:12.819Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.13058","snapshotId":"5eb2f3de978dd9e1496e99b312c657cd1df0b626cbb048917fd36fcac56b2eb8","contentSha256":"162e863cb5105520a25815d294d684f4248b6016bcff4dabe944ff2137386ae1","textHash":"22ab36abdb3bf5ca637d15dcc3e985808dd3180ae9454f888c2c58e95e7e2f73","observedAt":"2026-11-12T02:24:12.819Z","basis":"publisher-page-metadata","sourceLinks":43,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/060857accc4b789726afbfa061d8f1faa1abf24abc1698349fe90db13335584e/source"}}}],"audits":[{"publicationId":"efbd4fae-ea35-4204-8064-1a6ec7f0a4df","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the findings and argument to the authors of the arXiv paper (Arimbur, Ghosh, Sam). It keeps measurements and the companion-paper claim as the authors' own statements, not established facts.","excerpt":"Authors:Johin Johny Arimbur, Ankur Ashit Ghosh, Shervin Sam"},{"dimension":"summary","status":"uncertain","explanation":"The core findings are faithful, but the support-bot sentence drops its scale and model. The figures cover 100,000 chats a month on GPT-4o, yet the reading says only 'per month,' so readers cannot interpret the INR amounts. Smallest repair: add 'for 100,000 chats a month on GPT-4o.' The reading also omits the key contrast that Sarvam-30B/105B is the only third-party tokenizer staying below 2.6 times on every language. That erases part of the paper's argument. Central terms (tokenizer, token, context window, morpheme boundary) rely on tooltips; that is acceptable if they exist in the term library.","excerpt":"a support bot handling 100,000 chats a month on GPT-4o would cost about INR 56,640 in English and INR 102,829 in Marathi"},{"dimension":"date","status":"pass","explanation":"The displayed originalPublished date, 2026-11-09, matches the arXiv v1 submission date. The later feed date of 11 November is correctly not used.","excerpt":"[Submitted on 9 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim matches the abstract verbatim in substance, including the token ratios, the 35% boundary figure and the authors' caveat that the costs are estimates. Calling the companion tokenizer 'intended to reduce' the disparity is slightly softer than the source but not a contradiction. The question follows directly from the stated caveat.","excerpt":"these are estimates, not measured production costs"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper, which is the most direct access to the source.","excerpt":null}],"createdAt":"2026-11-12T00:16:39.611Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}