Epoch AI describes comparing open- and closed-weight models on cybersecurity benchmarks

Epoch AI describes comparing open- and closed-weight models on cybersecurity benchmarks

Epoch AI’s page was published on October 16, 2026. It describes comparing open- and closed-weight models with the Cyber ECI, a cybersecurity-specific version of the Epoch Capabilities Index (ECI). Epoch AI says it keeps benchmark parameters from a joint fit using general and cyber benchmarks, then estimates each model’s Cyber ECI from its cybersecurity results alone; scores use the same scale as the general ECI. The page links to data, analysis, assumptions and limitations, and methodology documentation, but the retrieved text does not include the results or explain the size of the gap named in the page title. The retrieved text therefore leaves unknown what Cyber ECI scores and analysis support the stated comparison between open- and closed-weight models. The comparison describes a benchmark-based measure, not evidence here about models’ performance in real-world cybersecurity work.

Publication: 3f3b1987-9724-ce0b-96b0-3866e5b88d9b

attribution — pass: The page is a single Epoch AI data insight describing its own Cyber ECI comparison, not a listing of other pieces. The reading attributes the methodology statements to Epoch AI, which matches the publisher and the first-person text.

We compare open- and closed-weight models using the Cyber ECI

summary — uncertain: The reading explains the Cyber ECI, and the definitions cover the ECI and benchmarks. However, the central contrast between open-weight and closed-weight models is never explained in plain language, and no term-library definition covers it. A general reader may not know that open-weight means the trained model parameters are publicly released. Smallest repair: add an open-weight / closed-weight term-library tooltip. The reading appropriately notes that the retrieved text does not substantiate the 6–7 month gap named in the title.

We compare open- and closed-weight models using the Cyber ECI

date — pass: The reading states October 16, 2026, which matches the UTC day of originalPublished. It uses no later events or hindsight framing. The CSV note dated Oct. 8, 2026 precedes publication, and the reading does not rely on it. The evidence kind is publisher-page, so no archive disclosure is needed.

support — pass: All three claims quote the source text verbatim and stay attributed. The caveat that the retrieved text lacks the results, and that a benchmark measure is not evidence of real-world cybersecurity performance, is a faithful statement of a gap rather than an invented claim. No unsupported benefits or costs are implied.

Scores are on the same scale as the general ECI.

media — pass: No media is selected and no verified media candidates were supplied, so the null selection is appropriate.