· Nov 12, 2025
Ethan Mollick makes the case for interviewing AI models in One Useful Thing
Markdown · JSON · Original source
’s piece was published November 12, 2025. He argues that can track broad improvements in AI ability, but often do not tell people which model will suit their particular work. Informal tests—his “vibes”—can reveal differences in writing, judgment, and recurring errors, but results vary and depend on the prompt. For organizations, Mollick says, choosing a model is like hiring: they should test it on realistic tasks and have experts assess its work, rather than rely on general benchmark scores. He points to ’s GDPval study as an example of comparing AI and human performance on work-like tasks, while noting that performance varies by task. His guacamole-drone example highlights another difference: models can give consistently different advice about an ambiguous business idea. At scale, he argues, those differences can steer many decisions in different directions. His conclusion is that organizations need repeated, task-specific evaluations as new models appear; the work is worth doing when an AI will advise many decisions. The piece does not establish how well this testing approach predicts performance after deployment.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Nov 12, 2025 (2025-11-12T02:46:43.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-10T05:03:26.338Z. First observed: 2026-09-23T05:43:00.089Z. Retrieved: 2026-10-23T16:15:41.932Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
Text as retrieved on October 23, 2026; originally published November 12, 2025. The piece gives examples and argues for task-specific evaluation, but does not establish how well such evaluations predict performance after deployment. Its account of GDPval and the model comparisons is Mollick’s presentation of those examples.
Open question
How well do task-specific evaluations predict an AI model’s performance and advice after deployment?
Source-stated
Ethan Mollick argues that public benchmarks have limitations and may not show which AI model best suits a particular user’s needs.
But what you actually care about is which model would be best for YOUR needs.
Source-stated
Mollick says informal, individualized tests can reveal model differences, but their variability and reliance on subjective impressions make them insufficient on their own.
Benchmarking through vibes - whether that is stories or code or otters - is a great way for an individual to get a feel for AI models, but it is also very idiosyncratic.
Source-stated
Mollick presents OpenAI’s GDPval research as an example of testing AI and human experts on realistic tasks, with experts grading the results without knowing who produced them.
OpenAI had a third group of experts grade the results, not knowing which answers came from the AI and which from the human
Source-stated
Mollick says models differed in their assessments of a guacamole-drone business idea and argues that different advice can steer decisions differently when given at scale.
When your AI is giving advice at scale, consistently rating ideas 3–4 points higher or lower means consistently steering you in a different direction.
Source-stated
Mollick concludes that organizations should test models on realistic tasks and judgments relevant to their work, and repeat evaluations as new models appear.
You need to systematically test your AI on the actual work it will do and the actual judgments it will make.
Source-stated
The source states that it was published on November 12, 2025.
Nov 12, 2025
Source and contributor context
One Useful Thing · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Ethan Mollick
Ethan Mollick is a Wharton researcher and author exploring how AI changes work, learning and decision-making.
One Useful Thing
One Useful Thing is Ethan Mollick’s publication about AI in work, education and everyday practice.
OpenAI
OpenAI develops AI models and products and publishes research, release announcements and evaluations.
Editorial provenance
Version 1. Publication ID: 873773e6-a922-94cb-165e-1cb50f1a8db7.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-10T04:58:50.343Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_screen.
Observation: 8dfe60e2324d312eb5345b9bce8104fcec5a80eac18cada896364000a3decf03. Source text hash: 19ce91d3d9b190fe8a81fc61ecde4c05e0e4940a03264bc8b1d0e62e223fefda.
Revision and source-check history
Version 1 · 2026-11-10T05:03:26.338Z
Ethan Mollick makes the case for interviewing AI models in One Useful Thing
’s piece was published November 12, 2025. He argues that can track broad improvements in AI ability, but often do not tell people which model will suit their particular work. Informal tests—his “vibes”—can reveal differences in writing, judgment, and recurring errors, but results vary and depend on the prompt. For organizations, Mollick says, choosing a model is like hiring: they should test it on realistic tasks and have experts assess its work, rather than rely on general benchmark scores. He points to ’s GDPval study as an example of comparing AI and human performance on work-like tasks, while noting that performance varies by task. His guacamole-drone example highlights another difference: models can give consistently different advice about an ambiguous business idea. At scale, he argues, those differences can steer many decisions in different directions. His conclusion is that organizations need repeated, task-specific evaluations as new models appear; the work is worth doing when an AI will advise many decisions. The piece does not establish how well this testing approach predicts performance after deployment.
Original source
· Article
Source details
Original source date: Nov 12, 2025 (2025-11-12T02:46:43.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-10T05:03:26.338Z. First observed: 2026-09-23T05:43:00.089Z. Retrieved: 2026-10-23T16:15:41.932Z.
Limits
Text as retrieved on October 23, 2026; originally published November 12, 2025. The piece gives examples and argues for task-specific evaluation, but does not establish how well such evaluations predict performance after deployment. Its account of GDPval and the model comparisons is Mollick’s presentation of those examples.
Open question
How well do task-specific evaluations predict an AI model’s performance and advice after deployment?
Source-stated
Ethan Mollick argues that public benchmarks have limitations and may not show which AI model best suits a particular user’s needs.
But what you actually care about is which model would be best for YOUR needs.
Source-stated
Mollick says informal, individualized tests can reveal model differences, but their variability and reliance on subjective impressions make them insufficient on their own.
Benchmarking through vibes - whether that is stories or code or otters - is a great way for an individual to get a feel for AI models, but it is also very idiosyncratic.
Source-stated
Mollick presents OpenAI’s GDPval research as an example of testing AI and human experts on realistic tasks, with experts grading the results without knowing who produced them.
OpenAI had a third group of experts grade the results, not knowing which answers came from the AI and which from the human
Source-stated
Mollick says models differed in their assessments of a guacamole-drone business idea and argues that different advice can steer decisions differently when given at scale.
When your AI is giving advice at scale, consistently rating ideas 3–4 points higher or lower means consistently steering you in a different direction.
Source-stated
Mollick concludes that organizations should test models on realistic tasks and judgments relevant to their work, and repeat evaluations as new models appear.
You need to systematically test your AI on the actual work it will do and the actual judgments it will make.
Source-stated
The source states that it was published on November 12, 2025.
Nov 12, 2025
Source and contributor context
One Useful Thing · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media retrieval status: unchecked.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Ethan Mollick
Ethan Mollick is a Wharton researcher and author exploring how AI changes work, learning and decision-making.
One Useful Thing
One Useful Thing is Ethan Mollick’s publication about AI in work, education and everyday practice.
OpenAI
OpenAI develops AI models and products and publishes research, release announcements and evaluations.
Editorial provenance
Version 1. Publication ID: 873773e6-a922-94cb-165e-1cb50f1a8db7.
Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-10T04:58:50.343Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: bridge_screen.
Observation: 8dfe60e2324d312eb5345b9bce8104fcec5a80eac18cada896364000a3decf03. Source text hash: 19ce91d3d9b190fe8a81fc61ecde4c05e0e4940a03264bc8b1d0e62e223fefda.
passed · 2026-11-10T04:58:50.343Z · anthropic/claude-opus-5.5
Publication: 873773e6-a922-94cb-165e-1cb50f1a8db7
attribution — pass: The reading attributes the argument to Ethan Mollick in One Useful Thing, matching the byline and publisher. Claims about GDPval and the model comparisons are framed as Mollick's presentation, not as independent facts.
Giving your AI a Job Interview As AI advice becomes more important, we are going to need to get better at assessing it Ethan Mollick
summary — pass: The summary keeps Mollick's thesis: interview your AI as you would a hire. It covers the limits of benchmarks, the value and limits of vibes testing, GDPval, the GuacaDrone example and the call for repeated task-specific evaluation. Benchmark is defined on the page, and vibes and GDPval are explained in context. The question follows from the evidence and is not leading.
To figure this out for yourself, you are going to need to interview your AI.
date — pass: originalPublished is 2025-11-12T02:46:43Z, whose UTC day is November 12, 2025, matching the source dateline. The page is a publisher page, not an archive copy, and its modification time equals its publication time. The reading uses no later events or hindsight framing.
Nov 12, 2025
support — pass: Each claim matches its quoted excerpt. The point that performance varies by task is stated in the source. The closing limitation, that the piece does not establish how well such testing predicts performance after deployment, is marked as a gap rather than presented as a claim. No unsupported benefits or costs are implied.
even among top models, performance varies significantly by task
media — pass: No media was selected, and no verified media candidates were supplied, so no media identity or relevance issue arises.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
One Useful Thing is Ethan Mollick’s publication about AI in work, education and everyday practice.
Explore One Useful ThingEthan Mollick is a Wharton researcher and author exploring how AI changes work, learning and decision-making.
Explore Ethan MollickA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceOpenAI develops AI models and products and publishes research, release announcements and evaluations.
Explore OpenAIOne Useful Thing is Ethan Mollick’s publication about AI in work, education and everyday practice.
Explore One Useful ThingEthan Mollick is a Wharton researcher and author exploring how AI changes work, learning and decision-making.
Explore Ethan MollickA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceOpenAI develops AI models and products and publishes research, release announcements and evaluations.
Explore OpenAIOne Useful Thing is Ethan Mollick’s publication about AI in work, education and everyday practice.
Explore One Useful Thing