{"id":"55dc9a5c854f45313fffec0b90b953d97987be20ccb613d24c46e26f416d73e4","title":"arXiv cs.AI: NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria","argument":"Authors Zainab Afolabi, Nikita Kozodoi, Jack Butler and Babs Khalidson introduce NaijaFacts, a benchmark of factual and statistical questions about Nigeria. They say native speakers wrote the questions, which are grounded in official statistics and available in English, Nigerian Pidgin, Yoruba, Igbo and Hausa. Similar questions about Nigeria, the UK and the US in different languages let the authors distinguish a language effect—when a model knows a fact but answers less well in a low-resource language—from a knowledge effect, when the fact is absent across languages.\nAcross eight frontier and Nigeria-adapted models, the authors report that the knowledge effect dominates: English-query accuracy is 20% for Nigeria, compared with 49–50% for the UK and US. They report a smaller language effect that tracks web representation; accuracy falls from English to Yoruba by 9 points on Nigeria questions and up to 24 points on UK and US questions. The authors also report that scale does not close the gap: the strongest models score 66–79% on UK and US questions, but none exceed 31% on Nigerian statistics. In an extension covering all 54 African countries with their best-performing frontier model, they report 96% accuracy on culture and 98% on politics, against 26% on statistics. The results frame missing country-specific knowledge—not only language—as a major gap in the tested models.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence is the arXiv page and abstract, which do not identify the eight models or provide question counts or detailed evaluation methods.","speaker":{"name":"Afolabi, Zainab, Kozodoi, Nikita, Butler, Jack, Khalidson, Babs","roleAtSource":"Article author"},"source":{"title":"NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.06504","publishedAt":"2026-11-04","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce a Nigeria-focused benchmark with questions in five languages, written by native speakers and grounded in official statistics.","excerpt":"a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authoritative official statistics, with every question available in English, Nigerian Pidgin, \\yoruba, Igbo, and Hausa","url":"https://arxiv.org/abs/2611.06504","relationship":"Source-stated"},{"text":"The authors use comparable questions about Nigeria, the UK and the US to distinguish a language effect from a knowledge effect.","excerpt":"yields a country by language matrix that disentangles a language effect (the model knows the fact but degrades in a low-resource language) from a knowledge effect (the fact is absent in every language).","url":"https://arxiv.org/abs/2611.06504","relationship":"Source-stated"},{"text":"Across eight models, the authors report that English-query accuracy is 20% for Nigeria and 49–50% for the UK and US, with the knowledge effect dominating.","excerpt":"Across eight frontier and Nigeria-adapted models, the knowledge effect dominates: English-query accuracy is 49--50\\% for the US/UK but 20\\% for Nigeria.","url":"https://arxiv.org/abs/2611.06504","relationship":"Source-stated"},{"text":"The authors report a smaller language effect that tracks web representation, with accuracy falling from English to Yoruba by 9 points on Nigeria questions and up to 24 points on UK and US questions.","excerpt":"The language effect is smaller and tracks web representation, degrading from English to \\yoruba{} (9 points on Nigeria, up to 24 on US/UK questions).","url":"https://arxiv.org/abs/2611.06504","relationship":"Source-stated"},{"text":"The authors report that none of the tested models exceed 31% on Nigerian statistics, while the strongest score 66–79% on UK and US questions.","excerpt":"the strongest models reach 66--79\\% on the UK and the US but none exceed 31\\% on Nigerian statistics.","url":"https://arxiv.org/abs/2611.06504","relationship":"Source-stated"},{"text":"In an extension covering 54 African countries, the authors report much higher accuracy on culture and politics than on statistics.","excerpt":"Extending to all 54 African countries with our best performing frontier model shows the same profile continent-wide, 96\\% on culture and 98\\% on politics against 26\\% on statistics","url":"https://arxiv.org/abs/2611.06504","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.06504","checkedAt":"2026-11-09T10:20:07.476Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much do the reported accuracy gaps vary across the questions and models used in NaijaFacts?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria","summary":"Authors Zainab Afolabi, Nikita Kozodoi, Jack Butler and Babs Khalidson introduce NaijaFacts, a benchmark of factual and statistical questions about Nigeria. They say native speakers wrote the questions, which are grounded in official statistics and available in English, Nigerian Pidgin, Yoruba, Igbo and Hausa. Similar questions about Nigeria, the UK and the US in different languages let the authors distinguish a language effect—when a model knows a fact but answers less well in a low-resource language—from a knowledge effect, when the fact is absent across languages.\nAcross eight frontier and Nigeria-adapted models, the authors report that the knowledge effect dominates: English-query accuracy is 20% for Nigeria, compared with 49–50% for the UK and US. They report a smaller language effect that tracks web representation; accuracy falls from English to Yoruba by 9 points on Nigeria questions and up to 24 points on UK and US questions. The authors also report that scale does not close the gap: the strongest models score 66–79% on UK and US questions, but none exceed 31% on Nigerian statistics. In an extension covering all 54 African countries with their best-performing frontier model, they report 96% accuracy on culture and 98% on politics, against 26% on statistics. The results frame missing country-specific knowledge—not only language—as a major gap in the tested models.","question":"How much do the reported accuracy gaps vary across the questions and models used in NaijaFacts?","claims":[{"text":"The authors introduce a Nigeria-focused benchmark with questions in five languages, written by native speakers and grounded in official statistics.","excerpt":"a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authoritative official statistics, with every question available in English, Nigerian Pidgin, \\yoruba, Igbo, and Hausa","support":"source-stated"},{"text":"The authors use comparable questions about Nigeria, the UK and the US to distinguish a language effect from a knowledge effect.","excerpt":"yields a country by language matrix that disentangles a language effect (the model knows the fact but degrades in a low-resource language) from a knowledge effect (the fact is absent in every language).","support":"source-stated"},{"text":"Across eight models, the authors report that English-query accuracy is 20% for Nigeria and 49–50% for the UK and US, with the knowledge effect dominating.","excerpt":"Across eight frontier and Nigeria-adapted models, the knowledge effect dominates: English-query accuracy is 49--50\\% for the US/UK but 20\\% for Nigeria.","support":"source-stated"},{"text":"The authors report a smaller language effect that tracks web representation, with accuracy falling from English to Yoruba by 9 points on Nigeria questions and up to 24 points on UK and US questions.","excerpt":"The language effect is smaller and tracks web representation, degrading from English to \\yoruba{} (9 points on Nigeria, up to 24 on US/UK questions).","support":"source-stated"},{"text":"The authors report that none of the tested models exceed 31% on Nigerian statistics, while the strongest score 66–79% on UK and US questions.","excerpt":"the strongest models reach 66--79\\% on the UK and the US but none exceed 31\\% on Nigerian statistics.","support":"source-stated"},{"text":"In an extension covering 54 African countries, the authors report much higher accuracy on culture and politics than on statistics.","excerpt":"Extending to all 54 African countries with our best performing frontier model shows the same profile continent-wide, 96\\% on culture and 98\\% on politics against 26\\% on statistics","support":"source-stated"}],"limitations":["The supplied evidence is the arXiv page and abstract, which do not identify the eight models or provide question counts or detailed evaluation methods."],"mediaId":"7074cf23f684a0aff68f477689a6f5d9e1f979fc2dba53d95ba61f3522383e27","correctionReason":null,"id":"55dc9a5c854f45313fffec0b90b953d97987be20ccb613d24c46e26f416d73e4","publicationId":"8ac43c37-eb16-4ead-8d45-5ce7d8e18c15","version":1,"sourceUrl":"https://arxiv.org/abs/2611.06504","sourceTitle":"NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-04","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:06:30.927Z","createdAt":"2026-11-09T09:17:16.643Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"7074cf23f684a0aff68f477689a6f5d9e1f979fc2dba53d95ba61f3522383e27","kind":"article","url":"https://arxiv.org/abs/2611.06504","sourceUrl":"https://arxiv.org/abs/2611.06504","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T09:10:59.804Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.06504","checkedAt":"2026-11-09T10:20:07.476Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"7074cf23f684a0aff68f477689a6f5d9e1f979fc2dba53d95ba61f3522383e27","confidence":0.85,"provider":"typesafe","model":"jev-1.13.0","inputHash":"55391215b471af3f12c3d482278e01f3df165c61ef6b1796d136b99c70dd6843","decidedAt":"2026-11-09T09:11:01.481Z"},"provenance":{"textHash":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","observationId":"974669458fd80c8a26f25d109deb2ac0647fdc28c7f04530f2c6e09784520ba1","retrievedAt":"2026-11-09T09:10:59.512Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"4306e2b5-e2d5-4baa-88b7-6fe8415d5809"},"reviewer":{"model":"gpt-6-luna","runId":"6a408cba-8f26-4eca-8e10-e679d21884cd"}},"audit":{"status":"passed","reviewedAt":"2026-11-09T09:17:16.643Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Afolabi, Zainab","url":null,"kind":"unknown"},{"name":"Kozodoi, Nikita","url":null,"kind":"unknown"},{"name":"Butler, Jack","url":null,"kind":"unknown"},{"name":"Khalidson, Babs","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Afolabi, Zainab","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Afolabi, Zainab","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06504","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T10:20:07.476Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010","sourceUrl":"https://arxiv.org/abs/2611.06504","contentSha256":"15e85f2ec18497a3ab8e14ce9d53aeab4744481d28a8da0ff463739fc9a47eaa","textHash":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","observedAt":"2026-11-09T10:20:07.476Z"}},{"actor":{"name":"Kozodoi, Nikita","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kozodoi, Nikita","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06504","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T10:20:07.476Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010","sourceUrl":"https://arxiv.org/abs/2611.06504","contentSha256":"15e85f2ec18497a3ab8e14ce9d53aeab4744481d28a8da0ff463739fc9a47eaa","textHash":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","observedAt":"2026-11-09T10:20:07.476Z"}},{"actor":{"name":"Butler, Jack","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Butler, Jack","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06504","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T10:20:07.476Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010","sourceUrl":"https://arxiv.org/abs/2611.06504","contentSha256":"15e85f2ec18497a3ab8e14ce9d53aeab4744481d28a8da0ff463739fc9a47eaa","textHash":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","observedAt":"2026-11-09T10:20:07.476Z"}},{"actor":{"name":"Khalidson, Babs","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Khalidson, Babs","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06504","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T10:20:07.476Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010","sourceUrl":"https://arxiv.org/abs/2611.06504","contentSha256":"15e85f2ec18497a3ab8e14ce9d53aeab4744481d28a8da0ff463739fc9a47eaa","textHash":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","observedAt":"2026-11-09T10:20:07.476Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.06504","snapshotId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010","contentSha256":"15e85f2ec18497a3ab8e14ce9d53aeab4744481d28a8da0ff463739fc9a47eaa","textHash":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","observedAt":"2026-11-09T10:20:07.476Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/55dc9a5c854f45313fffec0b90b953d97987be20ccb613d24c46e26f416d73e4/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-naijafacts-a-framework-for-disentangling-language-55dc9a5c","formats":{"html":"/readings/arxiv-cs-ai-naijafacts-a-framework-for-disentangling-language-55dc9a5c","markdown":"/readings/arxiv-cs-ai-naijafacts-a-framework-for-disentangling-language-55dc9a5c.md","json":"/readings/arxiv-cs-ai-naijafacts-a-framework-for-disentangling-language-55dc9a5c.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-frontier-model","name":"Frontier model","explanation":"One of the most capable AI models available at a given time; the set shifts as models improve.","referenceUrl":"https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"55dc9a5c854f45313fffec0b90b953d97987be20ccb613d24c46e26f416d73e4","revision":"8ac43c37-eb16-4ead-8d45-5ce7d8e18c15","revisionKind":"publication-id","title":"arXiv cs.AI: NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria","url":"https://escapemeridian.com/readings/arxiv-cs-ai-naijafacts-a-framework-for-disentangling-language-55dc9a5c"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.06504","revision":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","revisionKind":"sha256-source-text","title":"NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria","url":"https://arxiv.org/abs/2611.06504"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"55dc9a5c854f45313fffec0b90b953d97987be20ccb613d24c46e26f416d73e4","title":"arXiv cs.AI: NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria","argument":"Authors Zainab Afolabi, Nikita Kozodoi, Jack Butler and Babs Khalidson introduce NaijaFacts, a benchmark of factual and statistical questions about Nigeria. They say native speakers wrote the questions, which are grounded in official statistics and available in English, Nigerian Pidgin, Yoruba, Igbo and Hausa. Similar questions about Nigeria, the UK and the US in different languages let the authors distinguish a language effect—when a model knows a fact but answers less well in a low-resource language—from a knowledge effect, when the fact is absent across languages.\nAcross eight frontier and Nigeria-adapted models, the authors report that the knowledge effect dominates: English-query accuracy is 20% for Nigeria, compared with 49–50% for the UK and US. They report a smaller language effect that tracks web representation; accuracy falls from English to Yoruba by 9 points on Nigeria questions and up to 24 points on UK and US questions. The authors also report that scale does not close the gap: the strongest models score 66–79% on UK and US questions, but none exceed 31% on Nigerian statistics. In an extension covering all 54 African countries with their best-performing frontier model, they report 96% accuracy on culture and 98% on politics, against 26% on statistics. The results frame missing country-specific knowledge—not only language—as a major gap in the tested models.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence is the arXiv page and abstract, which do not identify the eight models or provide question counts or detailed evaluation methods.","speaker":{"name":"Afolabi, Zainab, Kozodoi, Nikita, Butler, Jack, Khalidson, Babs","roleAtSource":"Article author"},"source":{"title":"NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.06504","publishedAt":"2026-11-04","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors introduce a Nigeria-focused benchmark with questions in five languages, written by native speakers and grounded in official statistics.","excerpt":"a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authoritative official statistics, with every question available in English, Nigerian Pidgin, \\yoruba, Igbo, and Hausa","url":"https://arxiv.org/abs/2611.06504","relationship":"Source-stated"},{"text":"The authors use comparable questions about Nigeria, the UK and the US to distinguish a language effect from a knowledge effect.","excerpt":"yields a country by language matrix that disentangles a language effect (the model knows the fact but degrades in a low-resource language) from a knowledge effect (the fact is absent in every language).","url":"https://arxiv.org/abs/2611.06504","relationship":"Source-stated"},{"text":"Across eight models, the authors report that English-query accuracy is 20% for Nigeria and 49–50% for the UK and US, with the knowledge effect dominating.","excerpt":"Across eight frontier and Nigeria-adapted models, the knowledge effect dominates: English-query accuracy is 49--50\\% for the US/UK but 20\\% for Nigeria.","url":"https://arxiv.org/abs/2611.06504","relationship":"Source-stated"},{"text":"The authors report a smaller language effect that tracks web representation, with accuracy falling from English to Yoruba by 9 points on Nigeria questions and up to 24 points on UK and US questions.","excerpt":"The language effect is smaller and tracks web representation, degrading from English to \\yoruba{} (9 points on Nigeria, up to 24 on US/UK questions).","url":"https://arxiv.org/abs/2611.06504","relationship":"Source-stated"},{"text":"The authors report that none of the tested models exceed 31% on Nigerian statistics, while the strongest score 66–79% on UK and US questions.","excerpt":"the strongest models reach 66--79\\% on the UK and the US but none exceed 31\\% on Nigerian statistics.","url":"https://arxiv.org/abs/2611.06504","relationship":"Source-stated"},{"text":"In an extension covering 54 African countries, the authors report much higher accuracy on culture and politics than on statistics.","excerpt":"Extending to all 54 African countries with our best performing frontier model shows the same profile continent-wide, 96\\% on culture and 98\\% on politics against 26\\% on statistics","url":"https://arxiv.org/abs/2611.06504","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.06504","checkedAt":"2026-11-09T10:20:07.476Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How much do the reported accuracy gaps vary across the questions and models used in NaijaFacts?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria","summary":"Authors Zainab Afolabi, Nikita Kozodoi, Jack Butler and Babs Khalidson introduce NaijaFacts, a benchmark of factual and statistical questions about Nigeria. They say native speakers wrote the questions, which are grounded in official statistics and available in English, Nigerian Pidgin, Yoruba, Igbo and Hausa. Similar questions about Nigeria, the UK and the US in different languages let the authors distinguish a language effect—when a model knows a fact but answers less well in a low-resource language—from a knowledge effect, when the fact is absent across languages.\nAcross eight frontier and Nigeria-adapted models, the authors report that the knowledge effect dominates: English-query accuracy is 20% for Nigeria, compared with 49–50% for the UK and US. They report a smaller language effect that tracks web representation; accuracy falls from English to Yoruba by 9 points on Nigeria questions and up to 24 points on UK and US questions. The authors also report that scale does not close the gap: the strongest models score 66–79% on UK and US questions, but none exceed 31% on Nigerian statistics. In an extension covering all 54 African countries with their best-performing frontier model, they report 96% accuracy on culture and 98% on politics, against 26% on statistics. The results frame missing country-specific knowledge—not only language—as a major gap in the tested models.","question":"How much do the reported accuracy gaps vary across the questions and models used in NaijaFacts?","claims":[{"text":"The authors introduce a Nigeria-focused benchmark with questions in five languages, written by native speakers and grounded in official statistics.","excerpt":"a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authoritative official statistics, with every question available in English, Nigerian Pidgin, \\yoruba, Igbo, and Hausa","support":"source-stated"},{"text":"The authors use comparable questions about Nigeria, the UK and the US to distinguish a language effect from a knowledge effect.","excerpt":"yields a country by language matrix that disentangles a language effect (the model knows the fact but degrades in a low-resource language) from a knowledge effect (the fact is absent in every language).","support":"source-stated"},{"text":"Across eight models, the authors report that English-query accuracy is 20% for Nigeria and 49–50% for the UK and US, with the knowledge effect dominating.","excerpt":"Across eight frontier and Nigeria-adapted models, the knowledge effect dominates: English-query accuracy is 49--50\\% for the US/UK but 20\\% for Nigeria.","support":"source-stated"},{"text":"The authors report a smaller language effect that tracks web representation, with accuracy falling from English to Yoruba by 9 points on Nigeria questions and up to 24 points on UK and US questions.","excerpt":"The language effect is smaller and tracks web representation, degrading from English to \\yoruba{} (9 points on Nigeria, up to 24 on US/UK questions).","support":"source-stated"},{"text":"The authors report that none of the tested models exceed 31% on Nigerian statistics, while the strongest score 66–79% on UK and US questions.","excerpt":"the strongest models reach 66--79\\% on the UK and the US but none exceed 31\\% on Nigerian statistics.","support":"source-stated"},{"text":"In an extension covering 54 African countries, the authors report much higher accuracy on culture and politics than on statistics.","excerpt":"Extending to all 54 African countries with our best performing frontier model shows the same profile continent-wide, 96\\% on culture and 98\\% on politics against 26\\% on statistics","support":"source-stated"}],"limitations":["The supplied evidence is the arXiv page and abstract, which do not identify the eight models or provide question counts or detailed evaluation methods."],"mediaId":"7074cf23f684a0aff68f477689a6f5d9e1f979fc2dba53d95ba61f3522383e27","correctionReason":null,"id":"55dc9a5c854f45313fffec0b90b953d97987be20ccb613d24c46e26f416d73e4","publicationId":"8ac43c37-eb16-4ead-8d45-5ce7d8e18c15","version":1,"sourceUrl":"https://arxiv.org/abs/2611.06504","sourceTitle":"NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-04","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:06:30.927Z","createdAt":"2026-11-09T09:17:16.643Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"7074cf23f684a0aff68f477689a6f5d9e1f979fc2dba53d95ba61f3522383e27","kind":"article","url":"https://arxiv.org/abs/2611.06504","sourceUrl":"https://arxiv.org/abs/2611.06504","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T09:10:59.804Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.06504","checkedAt":"2026-11-09T10:20:07.476Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"7074cf23f684a0aff68f477689a6f5d9e1f979fc2dba53d95ba61f3522383e27","confidence":0.85,"provider":"typesafe","model":"jev-1.13.0","inputHash":"55391215b471af3f12c3d482278e01f3df165c61ef6b1796d136b99c70dd6843","decidedAt":"2026-11-09T09:11:01.481Z"},"provenance":{"textHash":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","observationId":"974669458fd80c8a26f25d109deb2ac0647fdc28c7f04530f2c6e09784520ba1","retrievedAt":"2026-11-09T09:10:59.512Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"4306e2b5-e2d5-4baa-88b7-6fe8415d5809"},"reviewer":{"model":"gpt-6-luna","runId":"6a408cba-8f26-4eca-8e10-e679d21884cd"}},"audit":{"status":"passed","reviewedAt":"2026-11-09T09:17:16.643Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Afolabi, Zainab","url":null,"kind":"unknown"},{"name":"Kozodoi, Nikita","url":null,"kind":"unknown"},{"name":"Butler, Jack","url":null,"kind":"unknown"},{"name":"Khalidson, Babs","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Afolabi, Zainab","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Afolabi, Zainab","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06504","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T10:20:07.476Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010","sourceUrl":"https://arxiv.org/abs/2611.06504","contentSha256":"15e85f2ec18497a3ab8e14ce9d53aeab4744481d28a8da0ff463739fc9a47eaa","textHash":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","observedAt":"2026-11-09T10:20:07.476Z"}},{"actor":{"name":"Kozodoi, Nikita","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Kozodoi, Nikita","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06504","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T10:20:07.476Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010","sourceUrl":"https://arxiv.org/abs/2611.06504","contentSha256":"15e85f2ec18497a3ab8e14ce9d53aeab4744481d28a8da0ff463739fc9a47eaa","textHash":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","observedAt":"2026-11-09T10:20:07.476Z"}},{"actor":{"name":"Butler, Jack","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Butler, Jack","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06504","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T10:20:07.476Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010","sourceUrl":"https://arxiv.org/abs/2611.06504","contentSha256":"15e85f2ec18497a3ab8e14ce9d53aeab4744481d28a8da0ff463739fc9a47eaa","textHash":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","observedAt":"2026-11-09T10:20:07.476Z"}},{"actor":{"name":"Khalidson, Babs","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Khalidson, Babs","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.06504","locator":"head/meta[name=author|citation_author]"},"assertionId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T10:20:07.476Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010","sourceUrl":"https://arxiv.org/abs/2611.06504","contentSha256":"15e85f2ec18497a3ab8e14ce9d53aeab4744481d28a8da0ff463739fc9a47eaa","textHash":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","observedAt":"2026-11-09T10:20:07.476Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.06504","snapshotId":"0ebb54845f794540f5266e426553140fdab556ba0623a5b4bbe31a9a4eddf010","contentSha256":"15e85f2ec18497a3ab8e14ce9d53aeab4744481d28a8da0ff463739fc9a47eaa","textHash":"34027fdf45aede90343f144e175ee79c79b03dd4fdb7ce9aa7b6f5334279d45e","observedAt":"2026-11-09T10:20:07.476Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/55dc9a5c854f45313fffec0b90b953d97987be20ccb613d24c46e26f416d73e4/source"}}}],"audits":[{"publicationId":"8ac43c37-eb16-4ead-8d45-5ce7d8e18c15","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading names the four listed authors and attributes the benchmark design, findings and interpretation to them throughout (introduce, say, report). It does not present the results as established facts or as Escape Meridian's view.","excerpt":"Authors:Zainab Afolabi, Nikita Kozodoi, Jack Butler, Babs Khalidson"},{"dimension":"summary","status":"pass","explanation":"The reading leads with the actors and the new benchmark. It explains the language-versus-knowledge distinction in plain terms and accurately conveys the main figures, the 'scale does not help' finding and the continent-wide extension. Terms such as frontier models and web representation are left for tooltips, but they do not block understanding. The closing framing matches the abstract's 'knowledge effect dominates.' The question about variance across questions and models is a fair open issue. A minor optional repair would add the authors' explicit conclusion that the gap is not specific to Nigeria.","excerpt":"Across eight frontier and Nigeria-adapted models, the knowledge effect dominates"},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-11-04. This matches the arXiv submission date for v1, not the later feed date of 2026-11-05.","excerpt":"[Submitted on 4 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Every claim's excerpt is a verbatim match to the abstract, and the numbers are reproduced correctly: 20% vs 49–50%, 9 and up to 24 points, 66–79% vs 31%, and 96%/98% vs 26%. The reading invents no benefits, costs or forecasts. The limitation correctly notes that the abstract does not identify the models, the question counts or the detailed methods.","excerpt":"the strongest models reach 66--79\\% on the UK and the US but none exceed 31\\% on Nigerian statistics."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified, direct arXiv abstract page for this paper. It matches the source URL and is the most direct way to reach the source.","excerpt":null}],"createdAt":"2026-11-09T09:17:16.643Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}