{"id":"cdd4521abff1b84035fa2a73febfdbb79b8d693ed913a509789b3194ef25cb89","title":"Nathan Lambert on Artificial Analysis v5 and the benchmarker's dilemma","argument":"Nathan Lambert argues that independent AI evaluation is difficult not only because scoring models is technically demanding, but because benchmark creators must choose what to measure and how to combine results. In his view, a single index is easy to discuss but cannot represent every capability fairly, and its choices can influence both public perceptions and model development.\n\nDiscussing Artificial Analysis’s Intelligence Index v5, Lambert says the ranking looks reasonable overall but reflects trade-offs in what counts as frontier capability. He argues that the index may understate the economic importance of open models if it emphasizes capabilities where closed models lead, while noting that open models have narrower strengths and remain behind on some science evaluations.\n\nLambert also questions whether private benchmarks, whose prompts and test environments are not public, are worth the loss of public scrutiny. Hidden tests may make it harder to tailor training to an evaluation, he writes, but they also prevent the community from finding and reporting flaws. He describes benchmark evaluation as a matter of trust, noting that commercial testing and public-interest institutions each involve trade-offs. Whether more specialized indices can broaden the picture without adding new biases remains open.","topicIds":["institutions-and-trust","intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"This is Lambert’s analysis and assessment of Artificial Analysis v5, not an independent verification of the index or its rankings. The source does not establish whether private benchmarks will reduce benchmark-focused training or how much public scrutiny would be lost in practice. Lambert’s discussion of open models’ future economic role and progress includes expectations and uncertainties, not demonstrated outcomes.","speaker":{"name":"Nathan Lambert","roleAtSource":"Article author"},"source":{"title":"Artificial Analysis v5 and the benchmarker's dilemma","publisher":"Interconnects AI","format":"Article","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","publishedAt":"2026-10-28T12:58:39+00:00","dateBasis":"Date stated by the publisher."},"evidence":[{"text":"Lambert says independent evaluation must balance a simple, comparable score against the trade-offs introduced by any aggregation method.","excerpt":"You need your benchmark to reduce to one single number, so that it is easy to discuss, use in reports, and compare models, but every method of aggregating scores comes with meaningful trade-offs.","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","relationship":"Source-stated"},{"text":"Lambert says the v5 ranking appears reasonable overall, but argues that the index reflects a choice about which capabilities matter.","excerpt":"At first glance, in aggregate, their model ranking definitely looks reasonable. Still, it shows signs of strain and confusion, as any benchmark will","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","relationship":"Source-stated"},{"text":"Lambert argues that an index emphasizing frontier capabilities could miss the economic role of open models in existing workflows.","excerpt":"over time as existing workflows of open models become key factors of the economy, it could become a benchmark that misses where the core economic engine of the industry is.","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","relationship":"Source-stated"},{"text":"Lambert says private benchmarks may limit opportunities for the community to find flaws and improve evaluations.","excerpt":"you no longer get the benefit of the community finding issues in the benchmark and pushing the benchmark to improve itself – which happens a lot!","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","relationship":"Source-stated"},{"text":"Lambert describes benchmark evaluation as dependent on trust and subject to trade-offs between different kinds of institutions.","excerpt":"The entities I trust the most to be fair on evaluations are non-profits, academics, and government institutions like CAISI. Today, the institutions I trust to be accurate on the frontier evaluation are startups like Artificial Analysis and Arena.","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","relationship":"Source-stated"}],"sourceMedia":{"status":"found","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","checkedAt":"2026-10-29T15:33:05.605Z","items":[{"type":"image","url":"https://substackcdn.com/image/fetch/$s_!WPu_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25433598-9909-4aff-b43f-d380c3eb099b_1200x567.jpeg","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","attribution":"Interconnects AI","basis":"publisher-embed","availability":"verified","label":"Image","verifiedAt":"2026-10-29T15:33:05.919Z"}],"note":"Media declared by the original publisher; availability checked separately from the editorial reading."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Will private benchmarks reduce benchmark-focused training without sacrificing the public scrutiny that Lambert says can expose flaws?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Nathan Lambert on Artificial Analysis v5 and the benchmarker's dilemma","summary":"Nathan Lambert argues that independent AI evaluation is difficult not only because scoring models is technically demanding, but because benchmark creators must choose what to measure and how to combine results. In his view, a single index is easy to discuss but cannot represent every capability fairly, and its choices can influence both public perceptions and model development.\n\nDiscussing Artificial Analysis’s Intelligence Index v5, Lambert says the ranking looks reasonable overall but reflects trade-offs in what counts as frontier capability. He argues that the index may understate the economic importance of open models if it emphasizes capabilities where closed models lead, while noting that open models have narrower strengths and remain behind on some science evaluations.\n\nLambert also questions whether private benchmarks, whose prompts and test environments are not public, are worth the loss of public scrutiny. Hidden tests may make it harder to tailor training to an evaluation, he writes, but they also prevent the community from finding and reporting flaws. He describes benchmark evaluation as a matter of trust, noting that commercial testing and public-interest institutions each involve trade-offs. Whether more specialized indices can broaden the picture without adding new biases remains open.","question":"Will private benchmarks reduce benchmark-focused training without sacrificing the public scrutiny that Lambert says can expose flaws?","claims":[{"text":"Lambert says independent evaluation must balance a simple, comparable score against the trade-offs introduced by any aggregation method.","excerpt":"You need your benchmark to reduce to one single number, so that it is easy to discuss, use in reports, and compare models, but every method of aggregating scores comes with meaningful trade-offs.","support":"source-stated"},{"text":"Lambert says the v5 ranking appears reasonable overall, but argues that the index reflects a choice about which capabilities matter.","excerpt":"At first glance, in aggregate, their model ranking definitely looks reasonable. Still, it shows signs of strain and confusion, as any benchmark will","support":"source-stated"},{"text":"Lambert argues that an index emphasizing frontier capabilities could miss the economic role of open models in existing workflows.","excerpt":"over time as existing workflows of open models become key factors of the economy, it could become a benchmark that misses where the core economic engine of the industry is.","support":"source-stated"},{"text":"Lambert says private benchmarks may limit opportunities for the community to find flaws and improve evaluations.","excerpt":"you no longer get the benefit of the community finding issues in the benchmark and pushing the benchmark to improve itself – which happens a lot!","support":"source-stated"},{"text":"Lambert describes benchmark evaluation as dependent on trust and subject to trade-offs between different kinds of institutions.","excerpt":"The entities I trust the most to be fair on evaluations are non-profits, academics, and government institutions like CAISI. Today, the institutions I trust to be accurate on the frontier evaluation are startups like Artificial Analysis and Arena.","support":"source-stated"}],"limitations":["This is Lambert’s analysis and assessment of Artificial Analysis v5, not an independent verification of the index or its rankings.","The source does not establish whether private benchmarks will reduce benchmark-focused training or how much public scrutiny would be lost in practice.","Lambert’s discussion of open models’ future economic role and progress includes expectations and uncertainties, not demonstrated outcomes."],"mediaId":"887c59818f68e469bbe98a7ced922e847e223dd634ae5b949c58d6b02e99ea8b","correctionReason":null,"id":"cdd4521abff1b84035fa2a73febfdbb79b8d693ed913a509789b3194ef25cb89","publicationId":"8b6438b1-2c35-42ee-8e13-c05834717a42","version":1,"sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","sourceTitle":"Artificial Analysis v5 and the benchmarker's dilemma","publisher":"Nathan Lambert","publishedAt":"2026-10-28T12:58:39+00:00","originalPublished":{"value":"2026-10-28T12:58:39+00:00","precision":"instant","basis":"publisher_page"},"firstObservedAt":"2026-10-28T13:10:38.657Z","createdAt":"2026-10-29T15:33:49.440Z","historical":false,"entityIds":["nathan-lambert"],"topicIds":["institutions-and-trust","intelligence-and-discovery"],"media":[{"id":"887c59818f68e469bbe98a7ced922e847e223dd634ae5b949c58d6b02e99ea8b","kind":"article","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-29T15:33:05.919Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."},{"id":"6d836bda4cb472eafdc7186a01154dd344af019bd1a87315cdf49c80363695e2","kind":"image","url":"https://substackcdn.com/image/fetch/$s_!WPu_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25433598-9909-4aff-b43f-d380c3eb099b_1200x567.jpeg","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","label":"Image","provenance":"publisher_page","verification":{"status":"verified","checkedAt":"2026-10-29T15:33:05.919Z","method":"publisher_association_and_head","contentType":"image/jpeg","cacheUntil":1793288285919},"recommendationReason":"Observed on the acquired publisher source; availability checked."}],"sourceMedia":{"status":"found","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","checkedAt":"2026-10-29T15:33:05.605Z","items":[{"type":"image","url":"https://substackcdn.com/image/fetch/$s_!WPu_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25433598-9909-4aff-b43f-d380c3eb099b_1200x567.jpeg","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","attribution":"Interconnects AI","basis":"publisher-embed","availability":"verified","label":"Image","verifiedAt":"2026-10-29T15:33:05.919Z"}],"note":"Media declared by the original publisher; availability checked separately from the editorial reading."},"editorialDecision":{"outcome":"accept","reasonCodes":["useful_perspective"],"rubricVersion":"stream-admission-v5","mediaId":"887c59818f68e469bbe98a7ced922e847e223dd634ae5b949c58d6b02e99ea8b","confidence":0.9800000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"d692824ad15270ea5d011e720b505790cf8d86248f9f046c317c520c9a026ad9","decidedAt":"2026-10-29T15:33:07.925Z"},"provenance":{"textHash":"d4ad702b35b8d9ea7de32566fdc7c42e314090677bd0781ad90de887e63e28a0","observationId":"348f89c4e8140cc0b63986826c073e27036c49efff5f31c7020b56b713997e9f","retrievedAt":"2026-10-29T15:33:05.605Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"79c268c2-cc6d-4f1d-8c1f-d38c8705962c"},"reviewer":{"model":"gpt-6-luna","runId":"f088b903-cb98-43e3-8a21-06e8e4efb21c"}},"audit":{"status":"passed","reviewedAt":"2026-10-29T15:33:49.440Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Nathan Lambert","url":"https://substack.com/@natolambert","kind":"Person"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Nathan Lambert","url":"https://substack.com/@natolambert","kind":"Person","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Nathan Lambert","status":"stated","provenance":{"basis":"article-jsonld","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","locator":"document/html[1]/body[2]/div[1]/div[1]/div[1]/script[0]@30781.author"},"assertionId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T15:33:05.605Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","contentSha256":"36408f37a3c297b107a1b69c06a05682ebd675003fbc45df91e12fee7d10585c","textHash":"d4ad702b35b8d9ea7de32566fdc7c42e314090677bd0781ad90de887e63e28a0","observedAt":"2026-10-29T15:33:05.605Z"}},{"actor":{"name":"Nathan Lambert","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Nathan Lambert","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","locator":"head/meta[name=author|citation_author]"},"assertionId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T15:33:05.605Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","contentSha256":"36408f37a3c297b107a1b69c06a05682ebd675003fbc45df91e12fee7d10585c","textHash":"d4ad702b35b8d9ea7de32566fdc7c42e314090677bd0781ad90de887e63e28a0","observedAt":"2026-10-29T15:33:05.605Z"}},{"actor":{"name":"Interconnects AI","url":"https://www.interconnects.ai/","kind":"unknown","actorId":null},"role":"publisher","scope":{"kind":"publication","locator":null},"literalCredit":"Interconnects AI","status":"stated","provenance":{"basis":"article-publisher","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","locator":"document/html[1]/body[2]/div[1]/div[1]/div[1]/script[0]@30781.publisher"},"assertionId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T15:33:05.605Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","contentSha256":"36408f37a3c297b107a1b69c06a05682ebd675003fbc45df91e12fee7d10585c","textHash":"d4ad702b35b8d9ea7de32566fdc7c42e314090677bd0781ad90de887e63e28a0","observedAt":"2026-10-29T15:33:05.605Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":{"name":"Interconnects AI","url":"https://www.interconnects.ai/"},"sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","snapshotId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184","contentSha256":"36408f37a3c297b107a1b69c06a05682ebd675003fbc45df91e12fee7d10585c","textHash":"d4ad702b35b8d9ea7de32566fdc7c42e314090677bd0781ad90de887e63e28a0","observedAt":"2026-10-29T15:33:05.605Z","basis":"publisher-page-metadata","sourceLinks":20,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/cdd4521abff1b84035fa2a73febfdbb79b8d693ed913a509789b3194ef25cb89/source"}},"canonicalUrl":"https://escapemeridian.com/readings/nathan-lambert-on-artificial-analysis-v5-and-the-benchmarker-s-cdd4521a","formats":{"html":"/readings/nathan-lambert-on-artificial-analysis-v5-and-the-benchmarker-s-cdd4521a","markdown":"/readings/nathan-lambert-on-artificial-analysis-v5-and-the-benchmarker-s-cdd4521a.md","json":"/readings/nathan-lambert-on-artificial-analysis-v5-and-the-benchmarker-s-cdd4521a.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"}],"entities":[{"id":"nathan-lambert","name":"Nathan Lambert","kind":"person","context":"Research-informed interpretation of model progress, training and limits; explicit alternative to runaway-superintelligence assumptions.","referenceUrls":["https://lexfridman.com/ai-sota-2026","https://lexfridman.com/deepseek-dylan-patel-nathan-lambert","https://www.interconnects.ai/p/i-expect-rapid-progress-but-not-towards","https://www.interconnects.ai/"],"path":"/entities/nathan-lambert","json":"/entities/nathan-lambert.json","markdown":"/entities/nathan-lambert.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"cdd4521abff1b84035fa2a73febfdbb79b8d693ed913a509789b3194ef25cb89","revision":"8b6438b1-2c35-42ee-8e13-c05834717a42","revisionKind":"publication-id","title":"Nathan Lambert on Artificial Analysis v5 and the benchmarker's dilemma","url":"https://escapemeridian.com/readings/nathan-lambert-on-artificial-analysis-v5-and-the-benchmarker-s-cdd4521a"},"source":{"type":"source","id":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","revision":"d4ad702b35b8d9ea7de32566fdc7c42e314090677bd0781ad90de887e63e28a0","revisionKind":"sha256-source-text","title":"Artificial Analysis v5 and the benchmarker's dilemma","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"cdd4521abff1b84035fa2a73febfdbb79b8d693ed913a509789b3194ef25cb89","title":"Nathan Lambert on Artificial Analysis v5 and the benchmarker's dilemma","argument":"Nathan Lambert argues that independent AI evaluation is difficult not only because scoring models is technically demanding, but because benchmark creators must choose what to measure and how to combine results. In his view, a single index is easy to discuss but cannot represent every capability fairly, and its choices can influence both public perceptions and model development.\n\nDiscussing Artificial Analysis’s Intelligence Index v5, Lambert says the ranking looks reasonable overall but reflects trade-offs in what counts as frontier capability. He argues that the index may understate the economic importance of open models if it emphasizes capabilities where closed models lead, while noting that open models have narrower strengths and remain behind on some science evaluations.\n\nLambert also questions whether private benchmarks, whose prompts and test environments are not public, are worth the loss of public scrutiny. Hidden tests may make it harder to tailor training to an evaluation, he writes, but they also prevent the community from finding and reporting flaws. He describes benchmark evaluation as a matter of trust, noting that commercial testing and public-interest institutions each involve trade-offs. Whether more specialized indices can broaden the picture without adding new biases remains open.","topicIds":["institutions-and-trust","intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"This is Lambert’s analysis and assessment of Artificial Analysis v5, not an independent verification of the index or its rankings. The source does not establish whether private benchmarks will reduce benchmark-focused training or how much public scrutiny would be lost in practice. Lambert’s discussion of open models’ future economic role and progress includes expectations and uncertainties, not demonstrated outcomes.","speaker":{"name":"Nathan Lambert","roleAtSource":"Article author"},"source":{"title":"Artificial Analysis v5 and the benchmarker's dilemma","publisher":"Interconnects AI","format":"Article","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","publishedAt":"2026-10-28T12:58:39+00:00","dateBasis":"Date stated by the publisher."},"evidence":[{"text":"Lambert says independent evaluation must balance a simple, comparable score against the trade-offs introduced by any aggregation method.","excerpt":"You need your benchmark to reduce to one single number, so that it is easy to discuss, use in reports, and compare models, but every method of aggregating scores comes with meaningful trade-offs.","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","relationship":"Source-stated"},{"text":"Lambert says the v5 ranking appears reasonable overall, but argues that the index reflects a choice about which capabilities matter.","excerpt":"At first glance, in aggregate, their model ranking definitely looks reasonable. Still, it shows signs of strain and confusion, as any benchmark will","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","relationship":"Source-stated"},{"text":"Lambert argues that an index emphasizing frontier capabilities could miss the economic role of open models in existing workflows.","excerpt":"over time as existing workflows of open models become key factors of the economy, it could become a benchmark that misses where the core economic engine of the industry is.","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","relationship":"Source-stated"},{"text":"Lambert says private benchmarks may limit opportunities for the community to find flaws and improve evaluations.","excerpt":"you no longer get the benefit of the community finding issues in the benchmark and pushing the benchmark to improve itself – which happens a lot!","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","relationship":"Source-stated"},{"text":"Lambert describes benchmark evaluation as dependent on trust and subject to trade-offs between different kinds of institutions.","excerpt":"The entities I trust the most to be fair on evaluations are non-profits, academics, and government institutions like CAISI. Today, the institutions I trust to be accurate on the frontier evaluation are startups like Artificial Analysis and Arena.","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","relationship":"Source-stated"}],"sourceMedia":{"status":"found","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","checkedAt":"2026-10-29T15:33:05.605Z","items":[{"type":"image","url":"https://substackcdn.com/image/fetch/$s_!WPu_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25433598-9909-4aff-b43f-d380c3eb099b_1200x567.jpeg","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","attribution":"Interconnects AI","basis":"publisher-embed","availability":"verified","label":"Image","verifiedAt":"2026-10-29T15:33:05.919Z"}],"note":"Media declared by the original publisher; availability checked separately from the editorial reading."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"Will private benchmarks reduce benchmark-focused training without sacrificing the public scrutiny that Lambert says can expose flaws?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Nathan Lambert on Artificial Analysis v5 and the benchmarker's dilemma","summary":"Nathan Lambert argues that independent AI evaluation is difficult not only because scoring models is technically demanding, but because benchmark creators must choose what to measure and how to combine results. In his view, a single index is easy to discuss but cannot represent every capability fairly, and its choices can influence both public perceptions and model development.\n\nDiscussing Artificial Analysis’s Intelligence Index v5, Lambert says the ranking looks reasonable overall but reflects trade-offs in what counts as frontier capability. He argues that the index may understate the economic importance of open models if it emphasizes capabilities where closed models lead, while noting that open models have narrower strengths and remain behind on some science evaluations.\n\nLambert also questions whether private benchmarks, whose prompts and test environments are not public, are worth the loss of public scrutiny. Hidden tests may make it harder to tailor training to an evaluation, he writes, but they also prevent the community from finding and reporting flaws. He describes benchmark evaluation as a matter of trust, noting that commercial testing and public-interest institutions each involve trade-offs. Whether more specialized indices can broaden the picture without adding new biases remains open.","question":"Will private benchmarks reduce benchmark-focused training without sacrificing the public scrutiny that Lambert says can expose flaws?","claims":[{"text":"Lambert says independent evaluation must balance a simple, comparable score against the trade-offs introduced by any aggregation method.","excerpt":"You need your benchmark to reduce to one single number, so that it is easy to discuss, use in reports, and compare models, but every method of aggregating scores comes with meaningful trade-offs.","support":"source-stated"},{"text":"Lambert says the v5 ranking appears reasonable overall, but argues that the index reflects a choice about which capabilities matter.","excerpt":"At first glance, in aggregate, their model ranking definitely looks reasonable. Still, it shows signs of strain and confusion, as any benchmark will","support":"source-stated"},{"text":"Lambert argues that an index emphasizing frontier capabilities could miss the economic role of open models in existing workflows.","excerpt":"over time as existing workflows of open models become key factors of the economy, it could become a benchmark that misses where the core economic engine of the industry is.","support":"source-stated"},{"text":"Lambert says private benchmarks may limit opportunities for the community to find flaws and improve evaluations.","excerpt":"you no longer get the benefit of the community finding issues in the benchmark and pushing the benchmark to improve itself – which happens a lot!","support":"source-stated"},{"text":"Lambert describes benchmark evaluation as dependent on trust and subject to trade-offs between different kinds of institutions.","excerpt":"The entities I trust the most to be fair on evaluations are non-profits, academics, and government institutions like CAISI. Today, the institutions I trust to be accurate on the frontier evaluation are startups like Artificial Analysis and Arena.","support":"source-stated"}],"limitations":["This is Lambert’s analysis and assessment of Artificial Analysis v5, not an independent verification of the index or its rankings.","The source does not establish whether private benchmarks will reduce benchmark-focused training or how much public scrutiny would be lost in practice.","Lambert’s discussion of open models’ future economic role and progress includes expectations and uncertainties, not demonstrated outcomes."],"mediaId":"887c59818f68e469bbe98a7ced922e847e223dd634ae5b949c58d6b02e99ea8b","correctionReason":null,"id":"cdd4521abff1b84035fa2a73febfdbb79b8d693ed913a509789b3194ef25cb89","publicationId":"8b6438b1-2c35-42ee-8e13-c05834717a42","version":1,"sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","sourceTitle":"Artificial Analysis v5 and the benchmarker's dilemma","publisher":"Nathan Lambert","publishedAt":"2026-10-28T12:58:39+00:00","originalPublished":{"value":"2026-10-28T12:58:39+00:00","precision":"instant","basis":"publisher_page"},"firstObservedAt":"2026-10-28T13:10:38.657Z","createdAt":"2026-10-29T15:33:49.440Z","historical":false,"entityIds":["nathan-lambert"],"topicIds":["institutions-and-trust","intelligence-and-discovery"],"media":[{"id":"887c59818f68e469bbe98a7ced922e847e223dd634ae5b949c58d6b02e99ea8b","kind":"article","url":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-29T15:33:05.919Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."},{"id":"6d836bda4cb472eafdc7186a01154dd344af019bd1a87315cdf49c80363695e2","kind":"image","url":"https://substackcdn.com/image/fetch/$s_!WPu_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25433598-9909-4aff-b43f-d380c3eb099b_1200x567.jpeg","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","label":"Image","provenance":"publisher_page","verification":{"status":"verified","checkedAt":"2026-10-29T15:33:05.919Z","method":"publisher_association_and_head","contentType":"image/jpeg","cacheUntil":1793288285919},"recommendationReason":"Observed on the acquired publisher source; availability checked."}],"sourceMedia":{"status":"found","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","checkedAt":"2026-10-29T15:33:05.605Z","items":[{"type":"image","url":"https://substackcdn.com/image/fetch/$s_!WPu_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F25433598-9909-4aff-b43f-d380c3eb099b_1200x567.jpeg","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","attribution":"Interconnects AI","basis":"publisher-embed","availability":"verified","label":"Image","verifiedAt":"2026-10-29T15:33:05.919Z"}],"note":"Media declared by the original publisher; availability checked separately from the editorial reading."},"editorialDecision":{"outcome":"accept","reasonCodes":["useful_perspective"],"rubricVersion":"stream-admission-v5","mediaId":"887c59818f68e469bbe98a7ced922e847e223dd634ae5b949c58d6b02e99ea8b","confidence":0.9800000000000001,"provider":"typesafe","model":"jev-1.13.0","inputHash":"d692824ad15270ea5d011e720b505790cf8d86248f9f046c317c520c9a026ad9","decidedAt":"2026-10-29T15:33:07.925Z"},"provenance":{"textHash":"d4ad702b35b8d9ea7de32566fdc7c42e314090677bd0781ad90de887e63e28a0","observationId":"348f89c4e8140cc0b63986826c073e27036c49efff5f31c7020b56b713997e9f","retrievedAt":"2026-10-29T15:33:05.605Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"79c268c2-cc6d-4f1d-8c1f-d38c8705962c"},"reviewer":{"model":"gpt-6-luna","runId":"f088b903-cb98-43e3-8a21-06e8e4efb21c"}},"audit":{"status":"passed","reviewedAt":"2026-10-29T15:33:49.440Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Nathan Lambert","url":"https://substack.com/@natolambert","kind":"Person"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Nathan Lambert","url":"https://substack.com/@natolambert","kind":"Person","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Nathan Lambert","status":"stated","provenance":{"basis":"article-jsonld","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","locator":"document/html[1]/body[2]/div[1]/div[1]/div[1]/script[0]@30781.author"},"assertionId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T15:33:05.605Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","contentSha256":"36408f37a3c297b107a1b69c06a05682ebd675003fbc45df91e12fee7d10585c","textHash":"d4ad702b35b8d9ea7de32566fdc7c42e314090677bd0781ad90de887e63e28a0","observedAt":"2026-10-29T15:33:05.605Z"}},{"actor":{"name":"Nathan Lambert","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Nathan Lambert","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","locator":"head/meta[name=author|citation_author]"},"assertionId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T15:33:05.605Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","contentSha256":"36408f37a3c297b107a1b69c06a05682ebd675003fbc45df91e12fee7d10585c","textHash":"d4ad702b35b8d9ea7de32566fdc7c42e314090677bd0781ad90de887e63e28a0","observedAt":"2026-10-29T15:33:05.605Z"}},{"actor":{"name":"Interconnects AI","url":"https://www.interconnects.ai/","kind":"unknown","actorId":null},"role":"publisher","scope":{"kind":"publication","locator":null},"literalCredit":"Interconnects AI","status":"stated","provenance":{"basis":"article-publisher","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","locator":"document/html[1]/body[2]/div[1]/div[1]/div[1]/script[0]@30781.publisher"},"assertionId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-10-29T15:33:05.605Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184","sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","contentSha256":"36408f37a3c297b107a1b69c06a05682ebd675003fbc45df91e12fee7d10585c","textHash":"d4ad702b35b8d9ea7de32566fdc7c42e314090677bd0781ad90de887e63e28a0","observedAt":"2026-10-29T15:33:05.605Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":{"name":"Interconnects AI","url":"https://www.interconnects.ai/"},"sourceUrl":"https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers","snapshotId":"a09e7f968de23399db3ad929018190fe6b184edab0c8fd1d7744f10321ddb184","contentSha256":"36408f37a3c297b107a1b69c06a05682ebd675003fbc45df91e12fee7d10585c","textHash":"d4ad702b35b8d9ea7de32566fdc7c42e314090677bd0781ad90de887e63e28a0","observedAt":"2026-10-29T15:33:05.605Z","basis":"publisher-page-metadata","sourceLinks":20,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/cdd4521abff1b84035fa2a73febfdbb79b8d693ed913a509789b3194ef25cb89/source"}}}],"audits":[{"publicationId":"8b6438b1-2c35-42ee-8e13-c05834717a42","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the arguments to Nathan Lambert, the byline author of this Interconnects post. It correctly identifies Artificial Analysis as the creator of the Intelligence Index v5 being discussed.","excerpt":"My prompt for this article is Artificial Analysis’s recent update to their Intelligence Index, v5."},{"dimension":"summary","status":"pass","explanation":"The summary accurately covers the aggregation dilemma, the v5 ranking, the open-versus-closed model argument, the trade-offs of private benchmarks, and trust in different kinds of institutions. A general reader can follow the actor, the development and its stakes. Terms such as open models are left unexplained, but this does not block understanding. The question follows from the source's private-benchmark discussion and is not leading.","excerpt":"All in, this is a change that directionally I think is defensible, but it’s not unanimous."},{"dimension":"date","status":"pass","explanation":"The originalPublished value of 2026-10-28 matches the byline date on the page and the feed entry.","excerpt":"Nathan Lambert\nOct 28, 2026"},{"dimension":"support","status":"pass","explanation":"Every claim is attributed to Lambert and backed by a verbatim excerpt. The open-model economic point is presented as his conditional argument. The limitations properly separate his assessment from independent verification and do not overstate outcomes.","excerpt":"There is enough money on the line, where I’d rather see transparency in benchmarking and enabling many eyes to find bugs."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the original article. It is the most direct way for readers to access the source.","excerpt":null}],"createdAt":"2026-10-29T15:33:49.440Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}