{"id":"0f4fcc4647125e5e2d7aeefa7ef757eb7c0a83062cb1797019c0c303e713aae4","title":"Anthropic outlines funding for third-party model evaluations","argument":"Anthropic’s post, published July 1, 2024, announces funding for third-party evaluations of advanced AI capabilities. Anthropic says the supply of high-quality, safety-relevant evaluations does not meet demand, and argues that outside evaluation work can provide tools for the wider field of AI safety.\n\nThe proposed work ranges from assessments for AI Safety Levels to evaluations of advanced science, societal impacts, and model autonomy. Anthropic also seeks infrastructure to help subject-matter experts develop evaluations, and trials comparing task performance between groups with and without model access. The post’s central concern is whether evaluations measure the risks they are meant to capture: Anthropic says even strong performance on an evaluation may not be enough to worry experts. It calls for realistic, safety-relevant threat modeling and presents the initiative as a way to advance comprehensive AI evaluation as an industry standard. The post does not establish whether funded evaluations will track risks beyond the tests themselves.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published July 1, 2024. Anthropic says even strong evaluation results may not indicate the risks the evaluations are intended to measure. The post does not report outcomes from the funded evaluations.","speaker":{"name":"Anthropic","roleAtSource":"Publisher"},"source":{"title":"Funding third-party model evaluations","publisher":"Anthropic","format":"Article","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","publishedAt":"2024-07-01T00:00:00.000Z","dateBasis":"Date stated on the publisher’s page."},"evidence":[{"text":"Anthropic announces an initiative to fund evaluations developed by third-party organizations to measure advanced AI capabilities.","excerpt":"today we're introducing a new initiative to fund evaluations developed by third-party organizations that can effectively measure advanced capabilities in AI models.","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","relationship":"Source-stated"},{"text":"Anthropic says demand for high-quality, safety-relevant evaluations exceeds supply.","excerpt":"Developing high-quality, safety-relevant evaluations remains challenging, and the demand is outpacing the supply.","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","relationship":"Source-stated"},{"text":"Anthropic says evaluation performance may not adequately indicate the risks an evaluation is intended to measure.","excerpt":"even the best evaluations are not always indicative of risks they purport to measure.","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","relationship":"Source-stated"},{"text":"Anthropic presents the initiative as supporting progress toward making comprehensive AI evaluation an industry standard.","excerpt":"We hope this initiative serves as a catalyst for progress towards a future where comprehensive AI evaluation is an industry standard.","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","relationship":"Source-stated"},{"text":"Anthropic says that high performance on an evaluation has often not been sufficient to concern experts.","excerpt":"Most of the time, when models have performed highly, experts have realized that high performance on that version of the evaluation is not sufficient to worry them.","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How will funded evaluations show that their results track risks outside the test itself?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Anthropic outlines funding for third-party model evaluations","summary":"Anthropic’s post, published July 1, 2024, announces funding for third-party evaluations of advanced AI capabilities. Anthropic says the supply of high-quality, safety-relevant evaluations does not meet demand, and argues that outside evaluation work can provide tools for the wider field of AI safety.\n\nThe proposed work ranges from assessments for AI Safety Levels to evaluations of advanced science, societal impacts, and model autonomy. Anthropic also seeks infrastructure to help subject-matter experts develop evaluations, and trials comparing task performance between groups with and without model access. The post’s central concern is whether evaluations measure the risks they are meant to capture: Anthropic says even strong performance on an evaluation may not be enough to worry experts. It calls for realistic, safety-relevant threat modeling and presents the initiative as a way to advance comprehensive AI evaluation as an industry standard. The post does not establish whether funded evaluations will track risks beyond the tests themselves.","question":"How will funded evaluations show that their results track risks outside the test itself?","claims":[{"text":"Anthropic announces an initiative to fund evaluations developed by third-party organizations to measure advanced AI capabilities.","excerpt":"today we're introducing a new initiative to fund evaluations developed by third-party organizations that can effectively measure advanced capabilities in AI models.","support":"source-stated"},{"text":"Anthropic says demand for high-quality, safety-relevant evaluations exceeds supply.","excerpt":"Developing high-quality, safety-relevant evaluations remains challenging, and the demand is outpacing the supply.","support":"source-stated"},{"text":"Anthropic says evaluation performance may not adequately indicate the risks an evaluation is intended to measure.","excerpt":"even the best evaluations are not always indicative of risks they purport to measure.","support":"source-stated"},{"text":"Anthropic presents the initiative as supporting progress toward making comprehensive AI evaluation an industry standard.","excerpt":"We hope this initiative serves as a catalyst for progress towards a future where comprehensive AI evaluation is an industry standard.","support":"source-stated"},{"text":"Anthropic says that high performance on an evaluation has often not been sufficient to concern experts.","excerpt":"Most of the time, when models have performed highly, experts have realized that high performance on that version of the evaluation is not sufficient to worry them.","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published July 1, 2024.","Anthropic says even strong evaluation results may not indicate the risks the evaluations are intended to measure. The post does not report outcomes from the funded evaluations."],"mediaId":null,"correctionReason":null,"id":"0f4fcc4647125e5e2d7aeefa7ef757eb7c0a83062cb1797019c0c303e713aae4","publicationId":"411350ce-8bf1-2cac-a4e4-21cf3970780c","version":1,"sourceUrl":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","sourceTitle":"Funding third-party model evaluations","publisher":"Anthropic","publishedAt":"2024-07-01T00:00:00.000Z","originalPublished":{"value":"2024-07-01T00:00:00.000Z","precision":"instant","basis":"backfill:article_published"},"firstObservedAt":"2026-10-21T23:08:20.413Z","createdAt":"2026-11-18T12:08:16.089Z","historical":true,"entityIds":["anthropic"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"a798f9a095ed7dfd5fb6a60d0c99d46ba6b8816943b2db07744b93a3ad0ecc15","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-24T11:15:41.011Z"},"provenance":{"textHash":"b6be36431294a996764d88889f7faadd3651c1529cc64d01621f78cbf1c3aa1e","observationId":"a10c4ff88afcb6aeacdc0a451794d8579d7ea518d085e162793379599be38efb","retrievedAt":"2026-10-24T11:13:35.541Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"d57a6481-4a77-b93c-f54f-0b8def961b94"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"6b146a23-0fe7-6809-f7c7-da1e2ee854a6"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T12:03:46.933Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/anthropic-outlines-funding-for-third-party-model-evaluations-0f4fcc46","formats":{"html":"/readings/anthropic-outlines-funding-for-third-party-model-evaluations-0f4fcc46","markdown":"/readings/anthropic-outlines-funding-for-third-party-model-evaluations-0f4fcc46.md","json":"/readings/anthropic-outlines-funding-for-third-party-model-evaluations-0f4fcc46.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-evals","name":"Evals","explanation":"Structured tests of what an AI system can do or how it behaves, run before or after release.","referenceUrl":"https://developers.google.com/machine-learning/glossary#evals"}],"entities":[{"id":"anthropic","name":"Anthropic","kind":"organization","context":"Anthropic develops Claude and publishes research about AI capability, safety and how people use its systems.","referenceUrls":["https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/anthropic","json":"/entities/anthropic.json","markdown":"/entities/anthropic.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"0f4fcc4647125e5e2d7aeefa7ef757eb7c0a83062cb1797019c0c303e713aae4","revision":"411350ce-8bf1-2cac-a4e4-21cf3970780c","revisionKind":"publication-id","title":"Anthropic outlines funding for third-party model evaluations","url":"https://escapemeridian.com/readings/anthropic-outlines-funding-for-third-party-model-evaluations-0f4fcc46"},"source":{"type":"source","id":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","revision":"b6be36431294a996764d88889f7faadd3651c1529cc64d01621f78cbf1c3aa1e","revisionKind":"sha256-source-text","title":"Funding third-party model evaluations","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"0f4fcc4647125e5e2d7aeefa7ef757eb7c0a83062cb1797019c0c303e713aae4","title":"Anthropic outlines funding for third-party model evaluations","argument":"Anthropic’s post, published July 1, 2024, announces funding for third-party evaluations of advanced AI capabilities. Anthropic says the supply of high-quality, safety-relevant evaluations does not meet demand, and argues that outside evaluation work can provide tools for the wider field of AI safety.\n\nThe proposed work ranges from assessments for AI Safety Levels to evaluations of advanced science, societal impacts, and model autonomy. Anthropic also seeks infrastructure to help subject-matter experts develop evaluations, and trials comparing task performance between groups with and without model access. The post’s central concern is whether evaluations measure the risks they are meant to capture: Anthropic says even strong performance on an evaluation may not be enough to worry experts. It calls for realistic, safety-relevant threat modeling and presents the initiative as a way to advance comprehensive AI evaluation as an industry standard. The post does not establish whether funded evaluations will track risks beyond the tests themselves.","topicIds":["artificial-general-intelligence"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 24, 2026; originally published July 1, 2024. Anthropic says even strong evaluation results may not indicate the risks the evaluations are intended to measure. The post does not report outcomes from the funded evaluations.","speaker":{"name":"Anthropic","roleAtSource":"Publisher"},"source":{"title":"Funding third-party model evaluations","publisher":"Anthropic","format":"Article","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","publishedAt":"2024-07-01T00:00:00.000Z","dateBasis":"Date stated on the publisher’s page."},"evidence":[{"text":"Anthropic announces an initiative to fund evaluations developed by third-party organizations to measure advanced AI capabilities.","excerpt":"today we're introducing a new initiative to fund evaluations developed by third-party organizations that can effectively measure advanced capabilities in AI models.","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","relationship":"Source-stated"},{"text":"Anthropic says demand for high-quality, safety-relevant evaluations exceeds supply.","excerpt":"Developing high-quality, safety-relevant evaluations remains challenging, and the demand is outpacing the supply.","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","relationship":"Source-stated"},{"text":"Anthropic says evaluation performance may not adequately indicate the risks an evaluation is intended to measure.","excerpt":"even the best evaluations are not always indicative of risks they purport to measure.","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","relationship":"Source-stated"},{"text":"Anthropic presents the initiative as supporting progress toward making comprehensive AI evaluation an industry standard.","excerpt":"We hope this initiative serves as a catalyst for progress towards a future where comprehensive AI evaluation is an industry standard.","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","relationship":"Source-stated"},{"text":"Anthropic says that high performance on an evaluation has often not been sufficient to concern experts.","excerpt":"Most of the time, when models have performed highly, experts have realized that high performance on that version of the evaluation is not sufficient to worry them.","url":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How will funded evaluations show that their results track risks outside the test itself?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Anthropic outlines funding for third-party model evaluations","summary":"Anthropic’s post, published July 1, 2024, announces funding for third-party evaluations of advanced AI capabilities. Anthropic says the supply of high-quality, safety-relevant evaluations does not meet demand, and argues that outside evaluation work can provide tools for the wider field of AI safety.\n\nThe proposed work ranges from assessments for AI Safety Levels to evaluations of advanced science, societal impacts, and model autonomy. Anthropic also seeks infrastructure to help subject-matter experts develop evaluations, and trials comparing task performance between groups with and without model access. The post’s central concern is whether evaluations measure the risks they are meant to capture: Anthropic says even strong performance on an evaluation may not be enough to worry experts. It calls for realistic, safety-relevant threat modeling and presents the initiative as a way to advance comprehensive AI evaluation as an industry standard. The post does not establish whether funded evaluations will track risks beyond the tests themselves.","question":"How will funded evaluations show that their results track risks outside the test itself?","claims":[{"text":"Anthropic announces an initiative to fund evaluations developed by third-party organizations to measure advanced AI capabilities.","excerpt":"today we're introducing a new initiative to fund evaluations developed by third-party organizations that can effectively measure advanced capabilities in AI models.","support":"source-stated"},{"text":"Anthropic says demand for high-quality, safety-relevant evaluations exceeds supply.","excerpt":"Developing high-quality, safety-relevant evaluations remains challenging, and the demand is outpacing the supply.","support":"source-stated"},{"text":"Anthropic says evaluation performance may not adequately indicate the risks an evaluation is intended to measure.","excerpt":"even the best evaluations are not always indicative of risks they purport to measure.","support":"source-stated"},{"text":"Anthropic presents the initiative as supporting progress toward making comprehensive AI evaluation an industry standard.","excerpt":"We hope this initiative serves as a catalyst for progress towards a future where comprehensive AI evaluation is an industry standard.","support":"source-stated"},{"text":"Anthropic says that high performance on an evaluation has often not been sufficient to concern experts.","excerpt":"Most of the time, when models have performed highly, experts have realized that high performance on that version of the evaluation is not sufficient to worry them.","support":"source-stated"}],"limitations":["Text as retrieved on October 24, 2026; originally published July 1, 2024.","Anthropic says even strong evaluation results may not indicate the risks the evaluations are intended to measure. The post does not report outcomes from the funded evaluations."],"mediaId":null,"correctionReason":null,"id":"0f4fcc4647125e5e2d7aeefa7ef757eb7c0a83062cb1797019c0c303e713aae4","publicationId":"411350ce-8bf1-2cac-a4e4-21cf3970780c","version":1,"sourceUrl":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","sourceTitle":"Funding third-party model evaluations","publisher":"Anthropic","publishedAt":"2024-07-01T00:00:00.000Z","originalPublished":{"value":"2024-07-01T00:00:00.000Z","precision":"instant","basis":"backfill:article_published"},"firstObservedAt":"2026-10-21T23:08:20.413Z","createdAt":"2026-11-18T12:08:16.089Z","historical":true,"entityIds":["anthropic"],"topicIds":["artificial-general-intelligence"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.anthropic.com/news/a-new-initiative-for-developing-third-party-model-evaluations","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"a798f9a095ed7dfd5fb6a60d0c99d46ba6b8816943b2db07744b93a3ad0ecc15","rubricVersion":"unrecorded","provider":"unrecorded","model":"anthropic/claude-opus-5.5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-24T11:15:41.011Z"},"provenance":{"textHash":"b6be36431294a996764d88889f7faadd3651c1529cc64d01621f78cbf1c3aa1e","observationId":"a10c4ff88afcb6aeacdc0a451794d8579d7ea518d085e162793379599be38efb","retrievedAt":"2026-10-24T11:13:35.541Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"d57a6481-4a77-b93c-f54f-0b8def961b94"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"6b146a23-0fe7-6809-f7c7-da1e2ee854a6"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-18T12:03:46.933Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"411350ce-8bf1-2cac-a4e4-21cf3970780c","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading consistently attributes the announcement, the supply-demand claim and the evaluation principles to Anthropic, the publisher of this single post. It does not present Anthropic's assertions as established fact.","excerpt":"today we're introducing a new initiative to fund evaluations developed by third-party organizations"},{"dimension":"summary","status":"uncertain","explanation":"\"AI Safety Levels\" is a central term that is neither explained in the reading nor in its definitions. A general reader cannot tell that these are levels from Anthropic's Responsible Scaling Policy that set safety and security requirements. Smallest repair: add one clause saying so. The reading also calls risk-tracking \"the post's central concern\", but the source makes that point as one of ten principles within a funding call. Smallest repair: call it a key principle rather than the central concern. The title and question are otherwise fair.","excerpt":"These levels determine the safety and security requirements for models with specific capabilities."},{"dimension":"date","status":"pass","explanation":"The stated July 1, 2024 matches the UTC day of originalPublished, and there is no hindsight framing. The page was modified in October 2026. The cited passages are core body text consistent with the original announcement, and the reading relies on no apparent later addition. Evidence kind is publisher-page, so no archive disclosure is required.","excerpt":null},{"dimension":"support","status":"pass","explanation":"Each claim matches its excerpt verbatim. The summary's remark that outside work can provide tools for the field rests on the opening paragraph. The question and limitation stay within the source and add no unsupported benefits or costs. The later page edit does not appear to touch these core passages.","excerpt":"Our investment in these evaluations is intended to elevate the entire field of AI safety, providing valuable tools that benefit the whole ecosystem."},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified media candidates were supplied, so nothing is misidentified.","excerpt":null}],"createdAt":"2026-11-18T12:03:46.933Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}