arXiv cs.AI: Who Can Evaluate AI Independently? A Framework for Analysing Evaluation Infrastructure

arXiv cs.AI: Who Can Evaluate AI Independently? A Framework for Analysing Evaluation Infrastructure

The authors of “Who Can Evaluate AI Independently?” argue that taking part in an AI evaluation does not itself give participants authority over its design, execution or supporting resources. They introduce PAAI (Power, Access, Adequacy, Independence), a framework for examining who controls evaluation infrastructure, the terms of access, whether it suits the evaluation’s purpose and what autonomy it supports. Applying the framework to African AI evaluation projects and joint testing exercises by an international network of AI Safety Institutes, the authors describe African actors contributing data, language expertise and task design while external institutions retain authority over infrastructure and evaluation frameworks. In the joint exercises, the institutes that led each exercise also ran the models; Kenya moved from annotation to task design, with no documented model execution. The authors’ central argument is that participation alone does not confer evaluative authority. They argue that credible governance requires access alongside control over priorities, methods, publication and continuity.

Publication: 9a9fe201-4d0a-4329-818b-4c4af520e91e

attribution — pass: The draft consistently attributes the arguments and findings to the paper's authors. The arXiv title and publisher label match the evidence. The paper's primary category is cs.CY, but it is cross-listed in cs.AI, so the label is acceptable.

Subjects: Computers and Society (cs.CY); Artificial Intelligence (cs.AI)

summary — uncertain: The draft omits the abstract's sharpest finding: design authority without infrastructure authority never yielded more than partial independence. It also leaves out the three named configurations. The question then relies on 'partial independence', which a reader cannot follow without that finding. Scope details are missing: twelve projects (2022-2025), ten institutes, mostly high-income members, and Kenya as the network's only African member, which links the two case studies. Evaluation infrastructure and AI Safety Institutes are not explained in plain terms. Smallest repair: add the partial-independence finding, the three configurations and Kenya's status.

design authority without infrastructure authority never yielded more than partial independence

date — pass: originalPublished is 2026-11-05. This matches the v1 submission date shown on the arXiv page. The feed date of Nov 6 is later and is correctly not used.

[Submitted on 5 Nov 2026]

support — pass: Each claim's excerpt is verbatim and supports it. The governance conclusion stays attributed to the authors as their argument. The limitation about undocumented execution properly avoids overclaiming. No unsupported benefits, costs or forecasts are added.

For evaluation to serve as a credible governance mechanism, access must come with control over priorities, methods, publication and continuity.

media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct access to the source.