{"id":"3e2219c1f72555f4a9c8d87ff3b2576e9c6a1497aa2d2ca06b6ea76e3342cd0f","title":"arXiv authors compare Jev on Arabic tasks and dialects","argument":"The paper’s authors compare Jev with decision models and large language models on Arabic tasks, using a shared zero-shot protocol. Their Arabic-focused frozen suite covers 43 tasks across 9 task families, and includes English controls in three parallel resources. Jev 1.14.0 scores below Gemma-4-31B on the aggregate measure, but the authors report that relative performance and gains between Jev versions vary by task family. They also examine content-matched comparisons between Modern Standard Arabic (MSA) and dialects: pooled dialect-minus-MSA accuracy gaps are negative across four datasets, while dialect-specific results vary more, and one interval includes zero. In a Palestinian banking comparison, Jev 1.14.0 has a larger penalty than either of two language models under simultaneous uncertainty; the authors say other gap contrasts remain inconclusive. Their point is that aggregate scores alone can obscure differences across tasks and language comparisons. The findings are descriptive: they do not establish a general ranking for dialect robustness or explain the observed gaps. The abstract does not identify what causes those differences.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence provided is the arXiv page and abstract, not the paper's full methods, results, or appendices. The authors state that the descriptive findings do not explain the causes of the observed gaps.","speaker":{"name":"Elsafty, Hossam, Deußer, Tobias, Sparrenberg, Lorenz, Ramamurthy, Rajkumar, Sifa, Rafet","roleAtSource":"Article author"},"source":{"title":"Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.10802","publishedAt":"2026-11-07","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors evaluate 67 systems on 43 tasks across 9 task families using a shared zero-shot protocol.","excerpt":"We evaluate 67 systems, including 47 decision models and 20 large language models (LLMs), on 43 tasks across 9 task families using a shared zero-shot protocol.","url":"https://arxiv.org/abs/2611.10802","relationship":"Source-stated"},{"text":"Jev 1.14.0's aggregate score is below Gemma-4-31B's, the highest-scoring system on that aggregate.","excerpt":"Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B, the highest-scoring system on this aggregate.","url":"https://arxiv.org/abs/2611.10802","relationship":"Source-stated"},{"text":"The authors report that relative performance and gains between Jev versions vary across task families.","excerpt":"Relative performance and gains between Jev versions vary across task families.","url":"https://arxiv.org/abs/2611.10802","relationship":"Source-stated"},{"text":"In content-matched comparisons, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results vary more, and one interval includes zero.","excerpt":"pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results show larger variation, and SIB-200's source-clustered interval includes zero.","url":"https://arxiv.org/abs/2611.10802","relationship":"Source-stated"},{"text":"The authors report a larger Palestinian banking penalty for Jev 1.14.0 than for either compared language model under simultaneous uncertainty; other gap contrasts remain inconclusive.","excerpt":"show a larger Palestinian banking penalty for Jev 1.14.0 than for either LLM under simultaneous uncertainty; the other gap contrasts remain inconclusive.","url":"https://arxiv.org/abs/2611.10802","relationship":"Source-stated"},{"text":"The authors say the descriptive findings neither establish a general model ranking for dialect robustness nor explain the causes of the observed gaps.","excerpt":"These descriptive findings show why aggregate scores should be read alongside task-family results, coverage and matched language comparisons. They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps.","url":"https://arxiv.org/abs/2611.10802","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10802","checkedAt":"2026-11-10T15:36:08.928Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What factors account for the observed gaps between MSA and dialect results, which the authors say their descriptive findings do not explain?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv authors compare Jev on Arabic tasks and dialects","summary":"The paper’s authors compare Jev with decision models and large language models on Arabic tasks, using a shared zero-shot protocol. Their Arabic-focused frozen suite covers 43 tasks across 9 task families, and includes English controls in three parallel resources. Jev 1.14.0 scores below Gemma-4-31B on the aggregate measure, but the authors report that relative performance and gains between Jev versions vary by task family. They also examine content-matched comparisons between Modern Standard Arabic (MSA) and dialects: pooled dialect-minus-MSA accuracy gaps are negative across four datasets, while dialect-specific results vary more, and one interval includes zero. In a Palestinian banking comparison, Jev 1.14.0 has a larger penalty than either of two language models under simultaneous uncertainty; the authors say other gap contrasts remain inconclusive. Their point is that aggregate scores alone can obscure differences across tasks and language comparisons. The findings are descriptive: they do not establish a general ranking for dialect robustness or explain the observed gaps. The abstract does not identify what causes those differences.","question":"What factors account for the observed gaps between MSA and dialect results, which the authors say their descriptive findings do not explain?","claims":[{"text":"The authors evaluate 67 systems on 43 tasks across 9 task families using a shared zero-shot protocol.","excerpt":"We evaluate 67 systems, including 47 decision models and 20 large language models (LLMs), on 43 tasks across 9 task families using a shared zero-shot protocol.","support":"source-stated"},{"text":"Jev 1.14.0's aggregate score is below Gemma-4-31B's, the highest-scoring system on that aggregate.","excerpt":"Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B, the highest-scoring system on this aggregate.","support":"source-stated"},{"text":"The authors report that relative performance and gains between Jev versions vary across task families.","excerpt":"Relative performance and gains between Jev versions vary across task families.","support":"source-stated"},{"text":"In content-matched comparisons, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results vary more, and one interval includes zero.","excerpt":"pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results show larger variation, and SIB-200's source-clustered interval includes zero.","support":"source-stated"},{"text":"The authors report a larger Palestinian banking penalty for Jev 1.14.0 than for either compared language model under simultaneous uncertainty; other gap contrasts remain inconclusive.","excerpt":"show a larger Palestinian banking penalty for Jev 1.14.0 than for either LLM under simultaneous uncertainty; the other gap contrasts remain inconclusive.","support":"source-stated"},{"text":"The authors say the descriptive findings neither establish a general model ranking for dialect robustness nor explain the causes of the observed gaps.","excerpt":"These descriptive findings show why aggregate scores should be read alongside task-family results, coverage and matched language comparisons. They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps.","support":"source-stated"}],"limitations":["The evidence provided is the arXiv page and abstract, not the paper's full methods, results, or appendices.","The authors state that the descriptive findings do not explain the causes of the observed gaps."],"mediaId":"c2244da7aab23043dc58943b145f5e30080199ce15f549bba09d71d7b5c89de9","correctionReason":null,"id":"3e2219c1f72555f4a9c8d87ff3b2576e9c6a1497aa2d2ca06b6ea76e3342cd0f","publicationId":"aaf13a7a-a3ce-4b26-8793-2f292d2e4a9c","version":1,"sourceUrl":"https://arxiv.org/abs/2611.10802","sourceTitle":"Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-07","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:42:30.027Z","createdAt":"2026-11-10T14:32:18.368Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"c2244da7aab23043dc58943b145f5e30080199ce15f549bba09d71d7b5c89de9","kind":"article","url":"https://arxiv.org/abs/2611.10802","sourceUrl":"https://arxiv.org/abs/2611.10802","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-10T14:31:34.241Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10802","checkedAt":"2026-11-10T15:36:08.928Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"c2244da7aab23043dc58943b145f5e30080199ce15f549bba09d71d7b5c89de9","confidence":0.8300000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"d0a80ae14e1e8fa68ebe1d3384d6ae9d413409ac4f623fdd33aa9646123040a7","decidedAt":"2026-11-10T14:31:34.871Z"},"provenance":{"textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observationId":"f8d0befd59812829dab56d8d0a297cf6a8f81ff1086957db6dd7e28bbc5b816f","retrievedAt":"2026-11-10T14:31:34.135Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"86f4601c-953c-4e4f-8f64-cae36a2f8050"},"reviewer":{"model":"gpt-6-luna","runId":"d549d026-ab2c-4c8c-8c31-e4b342f9ee4f"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-10T14:32:18.368Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Elsafty, Hossam","url":null,"kind":"unknown"},{"name":"Deußer, Tobias","url":null,"kind":"unknown"},{"name":"Sparrenberg, Lorenz","url":null,"kind":"unknown"},{"name":"Ramamurthy, Rajkumar","url":null,"kind":"unknown"},{"name":"Sifa, Rafet","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Elsafty, Hossam","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Elsafty, Hossam","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10802","locator":"head/meta[name=author|citation_author]"},"assertionId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:36:08.928Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443","sourceUrl":"https://arxiv.org/abs/2611.10802","contentSha256":"17c8688317b4d683e2aa6980dc3045e82323e015c503f1f35b206b7102b03541","textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observedAt":"2026-11-10T15:36:08.928Z"}},{"actor":{"name":"Deußer, Tobias","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Deußer, Tobias","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10802","locator":"head/meta[name=author|citation_author]"},"assertionId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:36:08.928Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443","sourceUrl":"https://arxiv.org/abs/2611.10802","contentSha256":"17c8688317b4d683e2aa6980dc3045e82323e015c503f1f35b206b7102b03541","textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observedAt":"2026-11-10T15:36:08.928Z"}},{"actor":{"name":"Sparrenberg, Lorenz","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sparrenberg, Lorenz","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10802","locator":"head/meta[name=author|citation_author]"},"assertionId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:36:08.928Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443","sourceUrl":"https://arxiv.org/abs/2611.10802","contentSha256":"17c8688317b4d683e2aa6980dc3045e82323e015c503f1f35b206b7102b03541","textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observedAt":"2026-11-10T15:36:08.928Z"}},{"actor":{"name":"Ramamurthy, Rajkumar","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ramamurthy, Rajkumar","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10802","locator":"head/meta[name=author|citation_author]"},"assertionId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:36:08.928Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443","sourceUrl":"https://arxiv.org/abs/2611.10802","contentSha256":"17c8688317b4d683e2aa6980dc3045e82323e015c503f1f35b206b7102b03541","textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observedAt":"2026-11-10T15:36:08.928Z"}},{"actor":{"name":"Sifa, Rafet","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sifa, Rafet","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10802","locator":"head/meta[name=author|citation_author]"},"assertionId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:36:08.928Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443","sourceUrl":"https://arxiv.org/abs/2611.10802","contentSha256":"17c8688317b4d683e2aa6980dc3045e82323e015c503f1f35b206b7102b03541","textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observedAt":"2026-11-10T15:36:08.928Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.10802","snapshotId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443","contentSha256":"17c8688317b4d683e2aa6980dc3045e82323e015c503f1f35b206b7102b03541","textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observedAt":"2026-11-10T15:36:08.928Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/3e2219c1f72555f4a9c8d87ff3b2576e9c6a1497aa2d2ca06b6ea76e3342cd0f/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-authors-compare-jev-on-arabic-tasks-and-dialects-3e2219c1","formats":{"html":"/readings/arxiv-authors-compare-jev-on-arabic-tasks-and-dialects-3e2219c1","markdown":"/readings/arxiv-authors-compare-jev-on-arabic-tasks-and-dialects-3e2219c1.md","json":"/readings/arxiv-authors-compare-jev-on-arabic-tasks-and-dialects-3e2219c1.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-zero-shot","name":"Zero-shot","explanation":"Doing a task without worked examples (zero-shot) or with only a few examples given in the prompt (few-shot).","referenceUrl":"https://developers.google.com/machine-learning/glossary#zero-shot-prompting"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"jev","name":"Jev","kind":"model","context":"Jev is TypeSafe AI’s model for fast evaluation and routing decisions inside AI systems.","referenceUrls":["https://typesafe.ai/blog/introducing-system-one-models-and-jev","https://typesafe.ai/"],"path":"/entities/jev","json":"/entities/jev.json","markdown":"/entities/jev.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"3e2219c1f72555f4a9c8d87ff3b2576e9c6a1497aa2d2ca06b6ea76e3342cd0f","revision":"aaf13a7a-a3ce-4b26-8793-2f292d2e4a9c","revisionKind":"publication-id","title":"arXiv authors compare Jev on Arabic tasks and dialects","url":"https://escapemeridian.com/readings/arxiv-authors-compare-jev-on-arabic-tasks-and-dialects-3e2219c1"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.10802","revision":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","revisionKind":"sha256-source-text","title":"Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects","url":"https://arxiv.org/abs/2611.10802"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"3e2219c1f72555f4a9c8d87ff3b2576e9c6a1497aa2d2ca06b6ea76e3342cd0f","title":"arXiv authors compare Jev on Arabic tasks and dialects","argument":"The paper’s authors compare Jev with decision models and large language models on Arabic tasks, using a shared zero-shot protocol. Their Arabic-focused frozen suite covers 43 tasks across 9 task families, and includes English controls in three parallel resources. Jev 1.14.0 scores below Gemma-4-31B on the aggregate measure, but the authors report that relative performance and gains between Jev versions vary by task family. They also examine content-matched comparisons between Modern Standard Arabic (MSA) and dialects: pooled dialect-minus-MSA accuracy gaps are negative across four datasets, while dialect-specific results vary more, and one interval includes zero. In a Palestinian banking comparison, Jev 1.14.0 has a larger penalty than either of two language models under simultaneous uncertainty; the authors say other gap contrasts remain inconclusive. Their point is that aggregate scores alone can obscure differences across tasks and language comparisons. The findings are descriptive: they do not establish a general ranking for dialect robustness or explain the observed gaps. The abstract does not identify what causes those differences.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence provided is the arXiv page and abstract, not the paper's full methods, results, or appendices. The authors state that the descriptive findings do not explain the causes of the observed gaps.","speaker":{"name":"Elsafty, Hossam, Deußer, Tobias, Sparrenberg, Lorenz, Ramamurthy, Rajkumar, Sifa, Rafet","roleAtSource":"Article author"},"source":{"title":"Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.10802","publishedAt":"2026-11-07","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors evaluate 67 systems on 43 tasks across 9 task families using a shared zero-shot protocol.","excerpt":"We evaluate 67 systems, including 47 decision models and 20 large language models (LLMs), on 43 tasks across 9 task families using a shared zero-shot protocol.","url":"https://arxiv.org/abs/2611.10802","relationship":"Source-stated"},{"text":"Jev 1.14.0's aggregate score is below Gemma-4-31B's, the highest-scoring system on that aggregate.","excerpt":"Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B, the highest-scoring system on this aggregate.","url":"https://arxiv.org/abs/2611.10802","relationship":"Source-stated"},{"text":"The authors report that relative performance and gains between Jev versions vary across task families.","excerpt":"Relative performance and gains between Jev versions vary across task families.","url":"https://arxiv.org/abs/2611.10802","relationship":"Source-stated"},{"text":"In content-matched comparisons, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results vary more, and one interval includes zero.","excerpt":"pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results show larger variation, and SIB-200's source-clustered interval includes zero.","url":"https://arxiv.org/abs/2611.10802","relationship":"Source-stated"},{"text":"The authors report a larger Palestinian banking penalty for Jev 1.14.0 than for either compared language model under simultaneous uncertainty; other gap contrasts remain inconclusive.","excerpt":"show a larger Palestinian banking penalty for Jev 1.14.0 than for either LLM under simultaneous uncertainty; the other gap contrasts remain inconclusive.","url":"https://arxiv.org/abs/2611.10802","relationship":"Source-stated"},{"text":"The authors say the descriptive findings neither establish a general model ranking for dialect robustness nor explain the causes of the observed gaps.","excerpt":"These descriptive findings show why aggregate scores should be read alongside task-family results, coverage and matched language comparisons. They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps.","url":"https://arxiv.org/abs/2611.10802","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10802","checkedAt":"2026-11-10T15:36:08.928Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"What factors account for the observed gaps between MSA and dialect results, which the authors say their descriptive findings do not explain?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv authors compare Jev on Arabic tasks and dialects","summary":"The paper’s authors compare Jev with decision models and large language models on Arabic tasks, using a shared zero-shot protocol. Their Arabic-focused frozen suite covers 43 tasks across 9 task families, and includes English controls in three parallel resources. Jev 1.14.0 scores below Gemma-4-31B on the aggregate measure, but the authors report that relative performance and gains between Jev versions vary by task family. They also examine content-matched comparisons between Modern Standard Arabic (MSA) and dialects: pooled dialect-minus-MSA accuracy gaps are negative across four datasets, while dialect-specific results vary more, and one interval includes zero. In a Palestinian banking comparison, Jev 1.14.0 has a larger penalty than either of two language models under simultaneous uncertainty; the authors say other gap contrasts remain inconclusive. Their point is that aggregate scores alone can obscure differences across tasks and language comparisons. The findings are descriptive: they do not establish a general ranking for dialect robustness or explain the observed gaps. The abstract does not identify what causes those differences.","question":"What factors account for the observed gaps between MSA and dialect results, which the authors say their descriptive findings do not explain?","claims":[{"text":"The authors evaluate 67 systems on 43 tasks across 9 task families using a shared zero-shot protocol.","excerpt":"We evaluate 67 systems, including 47 decision models and 20 large language models (LLMs), on 43 tasks across 9 task families using a shared zero-shot protocol.","support":"source-stated"},{"text":"Jev 1.14.0's aggregate score is below Gemma-4-31B's, the highest-scoring system on that aggregate.","excerpt":"Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B, the highest-scoring system on this aggregate.","support":"source-stated"},{"text":"The authors report that relative performance and gains between Jev versions vary across task families.","excerpt":"Relative performance and gains between Jev versions vary across task families.","support":"source-stated"},{"text":"In content-matched comparisons, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results vary more, and one interval includes zero.","excerpt":"pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results show larger variation, and SIB-200's source-clustered interval includes zero.","support":"source-stated"},{"text":"The authors report a larger Palestinian banking penalty for Jev 1.14.0 than for either compared language model under simultaneous uncertainty; other gap contrasts remain inconclusive.","excerpt":"show a larger Palestinian banking penalty for Jev 1.14.0 than for either LLM under simultaneous uncertainty; the other gap contrasts remain inconclusive.","support":"source-stated"},{"text":"The authors say the descriptive findings neither establish a general model ranking for dialect robustness nor explain the causes of the observed gaps.","excerpt":"These descriptive findings show why aggregate scores should be read alongside task-family results, coverage and matched language comparisons. They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps.","support":"source-stated"}],"limitations":["The evidence provided is the arXiv page and abstract, not the paper's full methods, results, or appendices.","The authors state that the descriptive findings do not explain the causes of the observed gaps."],"mediaId":"c2244da7aab23043dc58943b145f5e30080199ce15f549bba09d71d7b5c89de9","correctionReason":null,"id":"3e2219c1f72555f4a9c8d87ff3b2576e9c6a1497aa2d2ca06b6ea76e3342cd0f","publicationId":"aaf13a7a-a3ce-4b26-8793-2f292d2e4a9c","version":1,"sourceUrl":"https://arxiv.org/abs/2611.10802","sourceTitle":"Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-07","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:42:30.027Z","createdAt":"2026-11-10T14:32:18.368Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"c2244da7aab23043dc58943b145f5e30080199ce15f549bba09d71d7b5c89de9","kind":"article","url":"https://arxiv.org/abs/2611.10802","sourceUrl":"https://arxiv.org/abs/2611.10802","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-10T14:31:34.241Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10802","checkedAt":"2026-11-10T15:36:08.928Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"c2244da7aab23043dc58943b145f5e30080199ce15f549bba09d71d7b5c89de9","confidence":0.8300000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"d0a80ae14e1e8fa68ebe1d3384d6ae9d413409ac4f623fdd33aa9646123040a7","decidedAt":"2026-11-10T14:31:34.871Z"},"provenance":{"textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observationId":"f8d0befd59812829dab56d8d0a297cf6a8f81ff1086957db6dd7e28bbc5b816f","retrievedAt":"2026-11-10T14:31:34.135Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"86f4601c-953c-4e4f-8f64-cae36a2f8050"},"reviewer":{"model":"gpt-6-luna","runId":"d549d026-ab2c-4c8c-8c31-e4b342f9ee4f"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-10T14:32:18.368Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Elsafty, Hossam","url":null,"kind":"unknown"},{"name":"Deußer, Tobias","url":null,"kind":"unknown"},{"name":"Sparrenberg, Lorenz","url":null,"kind":"unknown"},{"name":"Ramamurthy, Rajkumar","url":null,"kind":"unknown"},{"name":"Sifa, Rafet","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Elsafty, Hossam","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Elsafty, Hossam","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10802","locator":"head/meta[name=author|citation_author]"},"assertionId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:36:08.928Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443","sourceUrl":"https://arxiv.org/abs/2611.10802","contentSha256":"17c8688317b4d683e2aa6980dc3045e82323e015c503f1f35b206b7102b03541","textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observedAt":"2026-11-10T15:36:08.928Z"}},{"actor":{"name":"Deußer, Tobias","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Deußer, Tobias","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10802","locator":"head/meta[name=author|citation_author]"},"assertionId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:36:08.928Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443","sourceUrl":"https://arxiv.org/abs/2611.10802","contentSha256":"17c8688317b4d683e2aa6980dc3045e82323e015c503f1f35b206b7102b03541","textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observedAt":"2026-11-10T15:36:08.928Z"}},{"actor":{"name":"Sparrenberg, Lorenz","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sparrenberg, Lorenz","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10802","locator":"head/meta[name=author|citation_author]"},"assertionId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:36:08.928Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443","sourceUrl":"https://arxiv.org/abs/2611.10802","contentSha256":"17c8688317b4d683e2aa6980dc3045e82323e015c503f1f35b206b7102b03541","textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observedAt":"2026-11-10T15:36:08.928Z"}},{"actor":{"name":"Ramamurthy, Rajkumar","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ramamurthy, Rajkumar","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10802","locator":"head/meta[name=author|citation_author]"},"assertionId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:36:08.928Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443","sourceUrl":"https://arxiv.org/abs/2611.10802","contentSha256":"17c8688317b4d683e2aa6980dc3045e82323e015c503f1f35b206b7102b03541","textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observedAt":"2026-11-10T15:36:08.928Z"}},{"actor":{"name":"Sifa, Rafet","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Sifa, Rafet","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10802","locator":"head/meta[name=author|citation_author]"},"assertionId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T15:36:08.928Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443","sourceUrl":"https://arxiv.org/abs/2611.10802","contentSha256":"17c8688317b4d683e2aa6980dc3045e82323e015c503f1f35b206b7102b03541","textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observedAt":"2026-11-10T15:36:08.928Z"}}],"participationCoverage":{"total":5,"returned":5,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.10802","snapshotId":"03e6fb2d4a52da19464aa5873b3001c33157261e0f0c125817cfbe00693cd443","contentSha256":"17c8688317b4d683e2aa6980dc3045e82323e015c503f1f35b206b7102b03541","textHash":"a8f80fab5cf019b06c69c877c8d07115a1493783b2bd70870302fd14574d810f","observedAt":"2026-11-10T15:36:08.928Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/3e2219c1f72555f4a9c8d87ff3b2576e9c6a1497aa2d2ca06b6ea76e3342cd0f/source"}}}],"audits":[{"publicationId":"aaf13a7a-a3ce-4b26-8793-2f292d2e4a9c","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the evaluation, scores and caveats to the paper's authors on arXiv. The source lists the authors and presents the findings as their own descriptive results. No source claim is presented as an established fact or as EM's opinion.","excerpt":"Authors:Hossam Elsafty, Tobias Deußer, Lorenz Sparrenberg, Rajkumar Ramamurthy, Rafet Sifa"},{"dimension":"summary","status":"uncertain","explanation":"The central actor, Jev, is never identified. The abstract also does not say what kind of system it is, and the draft should state that gap. Key terms are left without tooltips or plain-language explanation: decision models, zero-shot protocol, frozen suite, simultaneous uncertainty and the penalty. The draft also drops a scope condition: the MSA-dialect gaps are measured within Jev 1.14.0 only, not across all systems. Smallest repair: add within Jev 1.14.0 and note that the abstract does not describe Jev.","excerpt":"In content-matched comparisons within Jev 1.14.0, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets"},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-11-07 matches the v1 submission date shown on the arXiv page. The 2026-11-10 date is only a later feed date.","excerpt":"[Submitted on 7 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim matches the abstract verbatim in substance. The aggregate scores, task-family variation, the negative pooled gaps and the Palestinian banking contrast are all supported. The inconclusive contrasts and the stated limits are also preserved. The question follows from the authors' own stated limitation and does not invent benefits or costs.","excerpt":"They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the original arXiv abstract page. It is the most direct way to access the source.","excerpt":null}],"createdAt":"2026-11-10T14:32:18.368Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}