# arXiv authors compare Jev on Arabic tasks and dialects [Canonical reading](https://escapemeridian.com/readings/arxiv-authors-compare-jev-on-arabic-tasks-and-dialects-3e2219c1) The paper’s authors compare Jev with decision models and large language models on Arabic tasks, using a shared zero-shot protocol. Their Arabic-focused frozen suite covers 43 tasks across 9 task families, and includes English controls in three parallel resources. Jev 1.14.0 scores below Gemma-4-31B on the aggregate measure, but the authors report that relative performance and gains between Jev versions vary by task family. They also examine content-matched comparisons between Modern Standard Arabic (MSA) and dialects: pooled dialect-minus-MSA accuracy gaps are negative across four datasets, while dialect-specific results vary more, and one interval includes zero. In a Palestinian banking comparison, Jev 1.14.0 has a larger penalty than either of two language models under simultaneous uncertainty; the authors say other gap contrasts remain inconclusive. Their point is that aggregate scores alone can obscure differences across tasks and language comparisons. The findings are descriptive: they do not establish a general ranking for dialect robustness or explain the observed gaps. The abstract does not identify what causes those differences. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Nov 7, 2026 (2026-11-07). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T14:32:18.368Z. First observed: 2026-11-10T08:42:30.027Z. Retrieved: 2026-11-10T14:31:34.135Z. - [Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects](https://arxiv.org/abs/2611.10802) ## Limits The evidence provided is the arXiv page and abstract, not the paper's full methods, results, or appendices. The authors state that the descriptive findings do not explain the causes of the observed gaps. ## Open question What factors account for the observed gaps between MSA and dialect results, which the authors say their descriptive findings do not explain? ## Source-stated The authors evaluate 67 systems on 43 tasks across 9 task families using a shared zero-shot protocol. > We evaluate 67 systems, including 47 decision models and 20 large language models (LLMs), on 43 tasks across 9 task families using a shared zero-shot protocol. - [Supporting source](https://arxiv.org/abs/2611.10802) ## Source-stated Jev 1.14.0's aggregate score is below Gemma-4-31B's, the highest-scoring system on that aggregate. > Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B, the highest-scoring system on this aggregate. - [Supporting source](https://arxiv.org/abs/2611.10802) ## Source-stated The authors report that relative performance and gains between Jev versions vary across task families. > Relative performance and gains between Jev versions vary across task families. - [Supporting source](https://arxiv.org/abs/2611.10802) ## Source-stated In content-matched comparisons, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results vary more, and one interval includes zero. > pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results show larger variation, and SIB-200's source-clustered interval includes zero. - [Supporting source](https://arxiv.org/abs/2611.10802) ## Source-stated The authors report a larger Palestinian banking penalty for Jev 1.14.0 than for either compared language model under simultaneous uncertainty; other gap contrasts remain inconclusive. > show a larger Palestinian banking penalty for Jev 1.14.0 than for either LLM under simultaneous uncertainty; the other gap contrasts remain inconclusive. - [Supporting source](https://arxiv.org/abs/2611.10802) ## Source-stated The authors say the descriptive findings neither establish a general model ranking for dialect robustness nor explain the causes of the observed gaps. > These descriptive findings show why aggregate scores should be read alongside task-family results, coverage and matched language comparisons. They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps. - [Supporting source](https://arxiv.org/abs/2611.10802) ## Source and contributor context Elsafty, Hossam, Deußer, Tobias, Sparrenberg, Lorenz, Ramamurthy, Rajkumar, Sifa, Rafet · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Zero-shot Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot). - [Definition reference](https://developers.google.com/machine-learning/glossary#zero-shot-prompting) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Jev Jev is TypeSafe AI’s model for fast evaluation and routing decisions inside AI systems. - [Explore this context](/entities/jev) - [Reference](https://typesafe.ai/blog/introducing-system-one-models-and-jev) - [Reference](https://typesafe.ai/) ## Editorial provenance Version 1. Publication ID: aaf13a7a-a3ce-4b26-8793-2f292d2e4a9c. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T14:32:18.368Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: f8d0befd59812829dab56d8d0a297cf6a8f81ff1086957db6dd7e28bbc5b816f. Source text hash: a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f. ## Revision and source-check history ### Version 1 — 2026-11-10T14:32:18.368Z The paper’s authors compare Jev with decision models and large language models on Arabic tasks, using a shared zero-shot protocol. Their Arabic-focused frozen suite covers 43 tasks across 9 task families, and includes English controls in three parallel resources. Jev 1.14.0 scores below Gemma-4-31B on the aggregate measure, but the authors report that relative performance and gains between Jev versions vary by task family. They also examine content-matched comparisons between Modern Standard Arabic (MSA) and dialects: pooled dialect-minus-MSA accuracy gaps are negative across four datasets, while dialect-specific results vary more, and one interval includes zero. In a Palestinian banking comparison, Jev 1.14.0 has a larger penalty than either of two language models under simultaneous uncertainty; the authors say other gap contrasts remain inconclusive. Their point is that aggregate scores alone can obscure differences across tasks and language comparisons. The findings are descriptive: they do not establish a general ranking for dialect robustness or explain the observed gaps. The abstract does not identify what causes those differences. ## Original source arXiv cs.AI · Article Original source date: Nov 7, 2026 (2026-11-07). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T14:32:18.368Z. First observed: 2026-11-10T08:42:30.027Z. Retrieved: 2026-11-10T14:31:34.135Z. - [Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects](https://arxiv.org/abs/2611.10802) ## Limits The evidence provided is the arXiv page and abstract, not the paper's full methods, results, or appendices. The authors state that the descriptive findings do not explain the causes of the observed gaps. ## Open question What factors account for the observed gaps between MSA and dialect results, which the authors say their descriptive findings do not explain? ## Source-stated The authors evaluate 67 systems on 43 tasks across 9 task families using a shared zero-shot protocol. > We evaluate 67 systems, including 47 decision models and 20 large language models (LLMs), on 43 tasks across 9 task families using a shared zero-shot protocol. - [Supporting source](https://arxiv.org/abs/2611.10802) ## Source-stated Jev 1.14.0's aggregate score is below Gemma-4-31B's, the highest-scoring system on that aggregate. > Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B, the highest-scoring system on this aggregate. - [Supporting source](https://arxiv.org/abs/2611.10802) ## Source-stated The authors report that relative performance and gains between Jev versions vary across task families. > Relative performance and gains between Jev versions vary across task families. - [Supporting source](https://arxiv.org/abs/2611.10802) ## Source-stated In content-matched comparisons, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results vary more, and one interval includes zero. > pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results show larger variation, and SIB-200's source-clustered interval includes zero. - [Supporting source](https://arxiv.org/abs/2611.10802) ## Source-stated The authors report a larger Palestinian banking penalty for Jev 1.14.0 than for either compared language model under simultaneous uncertainty; other gap contrasts remain inconclusive. > show a larger Palestinian banking penalty for Jev 1.14.0 than for either LLM under simultaneous uncertainty; the other gap contrasts remain inconclusive. - [Supporting source](https://arxiv.org/abs/2611.10802) ## Source-stated The authors say the descriptive findings neither establish a general model ranking for dialect robustness nor explain the causes of the observed gaps. > These descriptive findings show why aggregate scores should be read alongside task-family results, coverage and matched language comparisons. They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps. - [Supporting source](https://arxiv.org/abs/2611.10802) ## Source and contributor context Elsafty, Hossam, Deußer, Tobias, Sparrenberg, Lorenz, Ramamurthy, Rajkumar, Sifa, Rafet · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Zero-shot Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot). - [Definition reference](https://developers.google.com/machine-learning/glossary#zero-shot-prompting) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Jev Jev is TypeSafe AI’s model for fast evaluation and routing decisions inside AI systems. - [Explore this context](/entities/jev) - [Reference](https://typesafe.ai/blog/introducing-system-one-models-and-jev) - [Reference](https://typesafe.ai/) ## Editorial provenance Version 1. Publication ID: aaf13a7a-a3ce-4b26-8793-2f292d2e4a9c. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T14:32:18.368Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: f8d0befd59812829dab56d8d0a297cf6a8f81ff1086957db6dd7e28bbc5b816f. Source text hash: a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f. ### uncertain — 2026-11-10T14:32:18.368Z — anthropic/claude-opus-5.5 Publication: aaf13a7a-a3ce-4b26-8793-2f292d2e4a9c attribution — pass: The reading attributes the evaluation, scores and caveats to the paper's authors on arXiv. The source lists the authors and presents the findings as their own descriptive results. No source claim is presented as an established fact or as EM's opinion. > Authors:Hossam Elsafty, Tobias Deußer, Lorenz Sparrenberg, Rajkumar Ramamurthy, Rafet Sifa summary — uncertain: The central actor, Jev, is never identified. The abstract also does not say what kind of system it is, and the draft should state that gap. Key terms are left without tooltips or plain-language explanation: decision models, zero-shot protocol, frozen suite, simultaneous uncertainty and the penalty. The draft also drops a scope condition: the MSA-dialect gaps are measured within Jev 1.14.0 only, not across all systems. Smallest repair: add within Jev 1.14.0 and note that the abstract does not describe Jev. > In content-matched comparisons within Jev 1.14.0, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets date — pass: The originalPublished date of 2026-11-07 matches the v1 submission date shown on the arXiv page. The 2026-11-10 date is only a later feed date. > \[Submitted on 7 Nov 2026\] support — pass: Each claim matches the abstract verbatim in substance. The aggregate scores, task-family variation, the negative pooled gaps and the Palestinian banking contrast are all supported. The inconclusive contrasts and the stated limits are also preserved. The question follows from the authors' own stated limitation and does not invent benefits or costs. > They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps. media — pass: The selected media is the verified direct link to the original arXiv abstract page. It is the most direct way to access the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/3e2219c1f72555f4a9c8d87ff3b2576e9c6a1497aa2d2ca06b6ea76e3342cd0f/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-authors-compare-jev-on-arabic-tasks-and-dialects-3e2219c1.json)