# Ethan Mollick makes the case for interviewing AI models in One Useful Thing [Canonical reading](https://escapemeridian.com/readings/ethan-mollick-makes-the-case-for-interviewing-ai-models-in-one-d6bc7bc2) Ethan Mollick’s piece was published November 12, 2025. He argues that benchmarks can track broad improvements in AI ability, but often do not tell people which model will suit their particular work. Informal tests—his “vibes”—can reveal differences in writing, judgment, and recurring errors, but results vary and depend on the prompt. For organizations, Mollick says, choosing a model is like hiring: they should test it on realistic tasks and have experts assess its work, rather than rely on general benchmark scores. He points to OpenAI’s GDPval study as an example of comparing AI and human performance on work-like tasks, while noting that performance varies by task. His guacamole-drone example highlights another difference: models can give consistently different advice about an ambiguous business idea. At scale, he argues, those differences can steer many decisions in different directions. His conclusion is that organizations need repeated, task-specific evaluations as new models appear; the work is worth doing when an AI will advise many decisions. The piece does not establish how well this testing approach predicts performance after deployment. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source One Useful Thing · Article Original source date: Nov 12, 2025 (2025-11-12T02:46:43.000Z). Date basis: Date supplied by the publisher’s feed. Added to Escape Meridian: 2026-11-10T05:03:26.338Z. First observed: 2026-09-23T05:43:00.089Z. Retrieved: 2026-10-23T16:15:41.932Z. - [Giving your AI a Job Interview](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Limits Text as retrieved on October 23, 2026; originally published November 12, 2025. The piece gives examples and argues for task-specific evaluation, but does not establish how well such evaluations predict performance after deployment. Its account of GDPval and the model comparisons is Mollick’s presentation of those examples. ## Open question How well do task-specific evaluations predict an AI model’s performance and advice after deployment? ## Source-stated Ethan Mollick argues that public benchmarks have limitations and may not show which AI model best suits a particular user’s needs. > But what you actually care about is which model would be best for YOUR needs. - [Supporting source](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Source-stated Mollick says informal, individualized tests can reveal model differences, but their variability and reliance on subjective impressions make them insufficient on their own. > Benchmarking through vibes - whether that is stories or code or otters - is a great way for an individual to get a feel for AI models, but it is also very idiosyncratic. - [Supporting source](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Source-stated Mollick presents OpenAI’s GDPval research as an example of testing AI and human experts on realistic tasks, with experts grading the results without knowing who produced them. > OpenAI had a third group of experts grade the results, not knowing which answers came from the AI and which from the human - [Supporting source](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Source-stated Mollick says models differed in their assessments of a guacamole-drone business idea and argues that different advice can steer decisions differently when given at scale. > When your AI is giving advice at scale, consistently rating ideas 3–4 points higher or lower means consistently steering you in a different direction. - [Supporting source](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Source-stated Mollick concludes that organizations should test models on realistic tasks and judgments relevant to their work, and repeat evaluations as new models appear. > You need to systematically test your AI on the actual work it will do and the actual judgments it will make. - [Supporting source](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Source-stated The source states that it was published on November 12, 2025. > Nov 12, 2025 - [Supporting source](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Source and contributor context One Useful Thing · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media retrieval status: unchecked. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Ethan Mollick Ethan Mollick is a Wharton researcher and author exploring how AI changes work, learning and decision-making. - [Explore this context](/entities/ethan-mollick) - [Reference](https://mgmt.wharton.upenn.edu/profile/emollick/) - [Reference](https://www.oneusefulthing.org/) ## One Useful Thing One Useful Thing is Ethan Mollick’s publication about AI in work, education and everyday practice. - [Explore this context](/entities/one-useful-thing) - [Reference](https://www.oneusefulthing.org/) ## OpenAI OpenAI develops AI models and products and publishes research, release announcements and evaluations. - [Explore this context](/entities/openai) - [Reference](https://openai.com/news/) ## Editorial provenance Version 1. Publication ID: 873773e6-a922-94cb-165e-1cb50f1a8db7. Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-10T04:58:50.343Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: bridge\_screen. Observation: 8dfe60e2324d312eb5345b9bce8104fcec5a80eac18cada896364000a3decf03. Source text hash: 19ce91d3d9b190fe8a81fc61ecde4c05e0e4940a03264bc8b1d0e62e223fefda. ## Revision and source-check history ### Version 1 — 2026-11-10T05:03:26.338Z Ethan Mollick’s piece was published November 12, 2025. He argues that benchmarks can track broad improvements in AI ability, but often do not tell people which model will suit their particular work. Informal tests—his “vibes”—can reveal differences in writing, judgment, and recurring errors, but results vary and depend on the prompt. For organizations, Mollick says, choosing a model is like hiring: they should test it on realistic tasks and have experts assess its work, rather than rely on general benchmark scores. He points to OpenAI’s GDPval study as an example of comparing AI and human performance on work-like tasks, while noting that performance varies by task. His guacamole-drone example highlights another difference: models can give consistently different advice about an ambiguous business idea. At scale, he argues, those differences can steer many decisions in different directions. His conclusion is that organizations need repeated, task-specific evaluations as new models appear; the work is worth doing when an AI will advise many decisions. The piece does not establish how well this testing approach predicts performance after deployment. ## Original source One Useful Thing · Article Original source date: Nov 12, 2025 (2025-11-12T02:46:43.000Z). Date basis: Date supplied by the publisher’s feed. Added to Escape Meridian: 2026-11-10T05:03:26.338Z. First observed: 2026-09-23T05:43:00.089Z. Retrieved: 2026-10-23T16:15:41.932Z. - [Giving your AI a Job Interview](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Limits Text as retrieved on October 23, 2026; originally published November 12, 2025. The piece gives examples and argues for task-specific evaluation, but does not establish how well such evaluations predict performance after deployment. Its account of GDPval and the model comparisons is Mollick’s presentation of those examples. ## Open question How well do task-specific evaluations predict an AI model’s performance and advice after deployment? ## Source-stated Ethan Mollick argues that public benchmarks have limitations and may not show which AI model best suits a particular user’s needs. > But what you actually care about is which model would be best for YOUR needs. - [Supporting source](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Source-stated Mollick says informal, individualized tests can reveal model differences, but their variability and reliance on subjective impressions make them insufficient on their own. > Benchmarking through vibes - whether that is stories or code or otters - is a great way for an individual to get a feel for AI models, but it is also very idiosyncratic. - [Supporting source](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Source-stated Mollick presents OpenAI’s GDPval research as an example of testing AI and human experts on realistic tasks, with experts grading the results without knowing who produced them. > OpenAI had a third group of experts grade the results, not knowing which answers came from the AI and which from the human - [Supporting source](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Source-stated Mollick says models differed in their assessments of a guacamole-drone business idea and argues that different advice can steer decisions differently when given at scale. > When your AI is giving advice at scale, consistently rating ideas 3–4 points higher or lower means consistently steering you in a different direction. - [Supporting source](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Source-stated Mollick concludes that organizations should test models on realistic tasks and judgments relevant to their work, and repeat evaluations as new models appear. > You need to systematically test your AI on the actual work it will do and the actual judgments it will make. - [Supporting source](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Source-stated The source states that it was published on November 12, 2025. > Nov 12, 2025 - [Supporting source](https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview) ## Source and contributor context One Useful Thing · Publisher The publisher is identified here. No individual author has been inferred. ## Source media Media retrieval status: unchecked. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Ethan Mollick Ethan Mollick is a Wharton researcher and author exploring how AI changes work, learning and decision-making. - [Explore this context](/entities/ethan-mollick) - [Reference](https://mgmt.wharton.upenn.edu/profile/emollick/) - [Reference](https://www.oneusefulthing.org/) ## One Useful Thing One Useful Thing is Ethan Mollick’s publication about AI in work, education and everyday practice. - [Explore this context](/entities/one-useful-thing) - [Reference](https://www.oneusefulthing.org/) ## OpenAI OpenAI develops AI models and products and publishes research, release announcements and evaluations. - [Explore this context](/entities/openai) - [Reference](https://openai.com/news/) ## Editorial provenance Version 1. Publication ID: 873773e6-a922-94cb-165e-1cb50f1a8db7. Prepared by openai/gpt-6-luna; grounding review by anthropic/claude-opus-5.5. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-10T04:58:50.343Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: bridge\_screen. Observation: 8dfe60e2324d312eb5345b9bce8104fcec5a80eac18cada896364000a3decf03. Source text hash: 19ce91d3d9b190fe8a81fc61ecde4c05e0e4940a03264bc8b1d0e62e223fefda. ### passed — 2026-11-10T04:58:50.343Z — anthropic/claude-opus-5.5 Publication: 873773e6-a922-94cb-165e-1cb50f1a8db7 attribution — pass: The reading attributes the argument to Ethan Mollick in One Useful Thing, matching the byline and publisher. Claims about GDPval and the model comparisons are framed as Mollick's presentation, not as independent facts. > Giving your AI a Job Interview As AI advice becomes more important, we are going to need to get better at assessing it Ethan Mollick summary — pass: The summary keeps Mollick's thesis: interview your AI as you would a hire. It covers the limits of benchmarks, the value and limits of vibes testing, GDPval, the GuacaDrone example and the call for repeated task-specific evaluation. Benchmark is defined on the page, and vibes and GDPval are explained in context. The question follows from the evidence and is not leading. > To figure this out for yourself, you are going to need to interview your AI. date — pass: originalPublished is 2025-11-12T02:46:43Z, whose UTC day is November 12, 2025, matching the source dateline. The page is a publisher page, not an archive copy, and its modification time equals its publication time. The reading uses no later events or hindsight framing. > Nov 12, 2025 support — pass: Each claim matches its quoted excerpt. The point that performance varies by task is stated in the source. The closing limitation, that the piece does not establish how well such testing predicts performance after deployment, is marked as a gap rather than presented as a claim. No unsupported benefits or costs are implied. > even among top models, performance varies significantly by task media — pass: No media was selected, and no verified media candidates were supplied, so no media identity or relevance issue arises. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/d6bc7bc2300708e2c61b5a063c4e9bdd8c86ee1ae9ddde4cb82068231c4a2c2b/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/ethan-mollick-makes-the-case-for-interviewing-ai-models-in-one-d6bc7bc2.json)