{"id":"0b170b33a3ef932766538e3376073189e22620935e62fdf27f79796fe9d9aa54","title":"MIT Technology Review on a tool for judging chatbot answers","argument":"MIT Technology Review published this item on April 26, 2024. Will Douglas Heaven reports that Cleanlab, an AI startup spun out of a quantum computing lab at MIT, created the Trustworthy Language Model to score large language model outputs for reliability. The score is between 0 and 1, and is meant to help people decide which responses to trust or discard. Heaven frames the tool as a possible response to businesses’ concern that chatbots make things up, but says it is unlikely to be perfect. The item does not explain how Cleanlab calculates the score or provide evidence of how well it identifies unreliable answers. It presents the tool’s potential usefulness and limits, not a demonstrated reduction in business risk.","topicIds":["investment-booms-and-busts"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 29, 2026; originally published April 26, 2024. The item links to a full story but does not include its contents, so the tool's scoring method and evidence of performance are not available here.","speaker":{"name":"MIT Technology Review","roleAtSource":"Publisher"},"source":{"title":"The Download: how to tell when a chatbot is lying, and RIP my biotech plants","publisher":"MIT Technology Review","format":"Article","url":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","publishedAt":"2024-04-26T12:10:00Z","dateBasis":"Date from the publisher’s own records, through its API."},"evidence":[{"text":"Will Douglas Heaven reports that Cleanlab created the Trustworthy Language Model to help high-stakes users assess large language model outputs.","excerpt":"A new tool created by Cleanlab, an AI startup spun out of a quantum computing lab at MIT, is designed to give high-stakes users a clearer sense of how trustworthy these models really are.","url":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","relationship":"Source-stated"},{"text":"The Trustworthy Language Model assigns a score between 0 and 1 to an output, which people can use to decide whether to trust or discard it.","excerpt":"The Trustworthy Language Model gives any output generated by a large language model a score between 0 and 1, according to its reliability.","url":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","relationship":"Source-stated"},{"text":"Heaven says the tool could appeal to businesses worried about fabricated answers, while cautioning that it is unlikely to be perfect.","excerpt":"Cleanlab hopes that its tool will make large language models more attractive to businesses worried about how much stuff they invent. But while the approach could be useful, it’s unlikely to be perfect.","url":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How does Cleanlab calculate the reliability score, and what evidence shows how well it distinguishes trustworthy answers from unreliable ones?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"MIT Technology Review on a tool for judging chatbot answers","summary":"MIT Technology Review published this item on April 26, 2024. Will Douglas Heaven reports that Cleanlab, an AI startup spun out of a quantum computing lab at MIT, created the Trustworthy Language Model to score large language model outputs for reliability. The score is between 0 and 1, and is meant to help people decide which responses to trust or discard. Heaven frames the tool as a possible response to businesses’ concern that chatbots make things up, but says it is unlikely to be perfect. The item does not explain how Cleanlab calculates the score or provide evidence of how well it identifies unreliable answers. It presents the tool’s potential usefulness and limits, not a demonstrated reduction in business risk.","question":"How does Cleanlab calculate the reliability score, and what evidence shows how well it distinguishes trustworthy answers from unreliable ones?","claims":[{"text":"Will Douglas Heaven reports that Cleanlab created the Trustworthy Language Model to help high-stakes users assess large language model outputs.","excerpt":"A new tool created by Cleanlab, an AI startup spun out of a quantum computing lab at MIT, is designed to give high-stakes users a clearer sense of how trustworthy these models really are.","support":"source-stated"},{"text":"The Trustworthy Language Model assigns a score between 0 and 1 to an output, which people can use to decide whether to trust or discard it.","excerpt":"The Trustworthy Language Model gives any output generated by a large language model a score between 0 and 1, according to its reliability.","support":"source-stated"},{"text":"Heaven says the tool could appeal to businesses worried about fabricated answers, while cautioning that it is unlikely to be perfect.","excerpt":"Cleanlab hopes that its tool will make large language models more attractive to businesses worried about how much stuff they invent. But while the approach could be useful, it’s unlikely to be perfect.","support":"source-stated"}],"limitations":["Text as retrieved on October 29, 2026; originally published April 26, 2024.","The item links to a full story but does not include its contents, so the tool's scoring method and evidence of performance are not available here."],"mediaId":null,"correctionReason":null,"id":"0b170b33a3ef932766538e3376073189e22620935e62fdf27f79796fe9d9aa54","publicationId":"5e73acfa-0e09-306c-5024-cbf8c284fcad","version":1,"sourceUrl":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","sourceTitle":"The Download: how to tell when a chatbot is lying, and RIP my biotech plants","publisher":"MIT Technology Review","publishedAt":"2024-04-26T12:10:00Z","originalPublished":{"value":"2024-04-26T12:10:00Z","precision":"instant","basis":"backfill:api_published"},"firstObservedAt":"2026-10-29T10:12:01.669Z","createdAt":"2026-11-20T05:48:09.839Z","historical":true,"entityIds":[],"topicIds":["investment-booms-and-busts"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"84b10bd1cc0aae3bb91d30e6c96d7e65950aea0b10b32fa1b5a57728ecddd117","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-29T10:11:58.991Z"},"provenance":{"textHash":"be088073710e06a68c6f978475175f0785dc709c877d6cbedae457d0d139d94f","observationId":"cc00300058eb382cb28cbe1e491ac741e372421c45c07e1c2f8c9161bc798e1b","retrievedAt":"2026-10-29T10:13:30.329Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"7893986e-2ad4-1943-f9e7-d28de87e28a4"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"4cf87d85-e523-9924-22d7-211118fafcd4"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-20T05:43:27.329Z","model":"anthropic/claude-opus-5.5"}},"canonicalUrl":"https://escapemeridian.com/readings/mit-technology-review-on-a-tool-for-judging-chatbot-answers-0b170b33","formats":{"html":"/readings/mit-technology-review-on-a-tool-for-judging-chatbot-answers-0b170b33","markdown":"/readings/mit-technology-review-on-a-tool-for-judging-chatbot-answers-0b170b33.md","json":"/readings/mit-technology-review-on-a-tool-for-judging-chatbot-answers-0b170b33.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"}],"entities":[]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"0b170b33a3ef932766538e3376073189e22620935e62fdf27f79796fe9d9aa54","revision":"5e73acfa-0e09-306c-5024-cbf8c284fcad","revisionKind":"publication-id","title":"MIT Technology Review on a tool for judging chatbot answers","url":"https://escapemeridian.com/readings/mit-technology-review-on-a-tool-for-judging-chatbot-answers-0b170b33"},"source":{"type":"source","id":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","revision":"be088073710e06a68c6f978475175f0785dc709c877d6cbedae457d0d139d94f","revisionKind":"sha256-source-text","title":"The Download: how to tell when a chatbot is lying, and RIP my biotech plants","url":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"0b170b33a3ef932766538e3376073189e22620935e62fdf27f79796fe9d9aa54","title":"MIT Technology Review on a tool for judging chatbot answers","argument":"MIT Technology Review published this item on April 26, 2024. Will Douglas Heaven reports that Cleanlab, an AI startup spun out of a quantum computing lab at MIT, created the Trustworthy Language Model to score large language model outputs for reliability. The score is between 0 and 1, and is meant to help people decide which responses to trust or discard. Heaven frames the tool as a possible response to businesses’ concern that chatbots make things up, but says it is unlikely to be perfect. The item does not explain how Cleanlab calculates the score or provide evidence of how well it identifies unreliable answers. It presents the tool’s potential usefulness and limits, not a demonstrated reduction in business risk.","topicIds":["investment-booms-and-busts"],"domain":"other","claimType":"attributed reading","limits":"Text as retrieved on October 29, 2026; originally published April 26, 2024. The item links to a full story but does not include its contents, so the tool's scoring method and evidence of performance are not available here.","speaker":{"name":"MIT Technology Review","roleAtSource":"Publisher"},"source":{"title":"The Download: how to tell when a chatbot is lying, and RIP my biotech plants","publisher":"MIT Technology Review","format":"Article","url":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","publishedAt":"2024-04-26T12:10:00Z","dateBasis":"Date from the publisher’s own records, through its API."},"evidence":[{"text":"Will Douglas Heaven reports that Cleanlab created the Trustworthy Language Model to help high-stakes users assess large language model outputs.","excerpt":"A new tool created by Cleanlab, an AI startup spun out of a quantum computing lab at MIT, is designed to give high-stakes users a clearer sense of how trustworthy these models really are.","url":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","relationship":"Source-stated"},{"text":"The Trustworthy Language Model assigns a score between 0 and 1 to an output, which people can use to decide whether to trust or discard it.","excerpt":"The Trustworthy Language Model gives any output generated by a large language model a score between 0 and 1, according to its reliability.","url":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","relationship":"Source-stated"},{"text":"Heaven says the tool could appeal to businesses worried about fabricated answers, while cautioning that it is unlikely to be perfect.","excerpt":"Cleanlab hopes that its tool will make large language models more attractive to businesses worried about how much stuff they invent. But while the approach could be useful, it’s unlikely to be perfect.","url":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","relationship":"Source-stated"}],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","checkedAt":null,"items":[]},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How does Cleanlab calculate the reliability score, and what evidence shows how well it distinguishes trustworthy answers from unreliable ones?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"MIT Technology Review on a tool for judging chatbot answers","summary":"MIT Technology Review published this item on April 26, 2024. Will Douglas Heaven reports that Cleanlab, an AI startup spun out of a quantum computing lab at MIT, created the Trustworthy Language Model to score large language model outputs for reliability. The score is between 0 and 1, and is meant to help people decide which responses to trust or discard. Heaven frames the tool as a possible response to businesses’ concern that chatbots make things up, but says it is unlikely to be perfect. The item does not explain how Cleanlab calculates the score or provide evidence of how well it identifies unreliable answers. It presents the tool’s potential usefulness and limits, not a demonstrated reduction in business risk.","question":"How does Cleanlab calculate the reliability score, and what evidence shows how well it distinguishes trustworthy answers from unreliable ones?","claims":[{"text":"Will Douglas Heaven reports that Cleanlab created the Trustworthy Language Model to help high-stakes users assess large language model outputs.","excerpt":"A new tool created by Cleanlab, an AI startup spun out of a quantum computing lab at MIT, is designed to give high-stakes users a clearer sense of how trustworthy these models really are.","support":"source-stated"},{"text":"The Trustworthy Language Model assigns a score between 0 and 1 to an output, which people can use to decide whether to trust or discard it.","excerpt":"The Trustworthy Language Model gives any output generated by a large language model a score between 0 and 1, according to its reliability.","support":"source-stated"},{"text":"Heaven says the tool could appeal to businesses worried about fabricated answers, while cautioning that it is unlikely to be perfect.","excerpt":"Cleanlab hopes that its tool will make large language models more attractive to businesses worried about how much stuff they invent. But while the approach could be useful, it’s unlikely to be perfect.","support":"source-stated"}],"limitations":["Text as retrieved on October 29, 2026; originally published April 26, 2024.","The item links to a full story but does not include its contents, so the tool's scoring method and evidence of performance are not available here."],"mediaId":null,"correctionReason":null,"id":"0b170b33a3ef932766538e3376073189e22620935e62fdf27f79796fe9d9aa54","publicationId":"5e73acfa-0e09-306c-5024-cbf8c284fcad","version":1,"sourceUrl":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","sourceTitle":"The Download: how to tell when a chatbot is lying, and RIP my biotech plants","publisher":"MIT Technology Review","publishedAt":"2024-04-26T12:10:00Z","originalPublished":{"value":"2024-04-26T12:10:00Z","precision":"instant","basis":"backfill:api_published"},"firstObservedAt":"2026-10-29T10:12:01.669Z","createdAt":"2026-11-20T05:48:09.839Z","historical":true,"entityIds":[],"topicIds":["investment-booms-and-busts"],"media":[],"sourceMedia":{"status":"unchecked","sourceUrl":"https://www.technologyreview.com/2024/04/26/1091875/the-download-how-to-tell-when-a-chatbot-is-lying-and-rip-my-biotech-plants/","checkedAt":null,"items":[]},"editorialDecision":{"outcome":"accept","mediaId":null,"inputHash":"84b10bd1cc0aae3bb91d30e6c96d7e65950aea0b10b32fa1b5a57728ecddd117","rubricVersion":"unrecorded","provider":"anthropic","model":"claude-opus-5-5","reasonCodes":["bridge_admitted_by_model","second_opinion_categorical"],"confidence":1,"decidedAt":"2026-10-29T10:11:58.991Z"},"provenance":{"textHash":"be088073710e06a68c6f978475175f0785dc709c877d6cbedae457d0d139d94f","observationId":"cc00300058eb382cb28cbe1e491ac741e372421c45c07e1c2f8c9161bc798e1b","retrievedAt":"2026-10-29T10:13:30.329Z","kind":"publisher-page","preparer":{"model":"openai/gpt-6-luna","runId":"7893986e-2ad4-1943-f9e7-d28de87e28a4"},"reviewer":{"model":"anthropic/claude-opus-5.5","runId":"4cf87d85-e523-9924-22d7-211118fafcd4"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-20T05:43:27.329Z","model":"anthropic/claude-opus-5.5"}}}],"audits":[{"publicationId":"5e73acfa-0e09-306c-5024-cbf8c284fcad","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"This is a newsletter issue that comments on items in its own words, so it counts as one piece. The reading focuses on the first item and correctly credits it to MIT Technology Review and Will Douglas Heaven, whose signature follows that summary. Cleanlab is correctly named as the tool's creator.","excerpt":"—Will Douglas Heaven"},{"dimension":"summary","status":"pass","explanation":"The reading states the actor (Cleanlab), the development (the Trustworthy Language Model's 0–1 reliability score) and the business concern about fabricated answers. Large language model is defined on the page, and the reading notes the item's limits. It drops the source's punchier framing ('a BS-o-meter for chatbots'); restoring it would be a small voice improvement, not a fidelity failure. The question follows from what the source leaves unexplained.","excerpt":"In other words: a BS-o-meter for chatbots."},{"dimension":"date","status":"pass","explanation":"April 26, 2024 matches the UTC day of originalPublished. modifiedAt (11:46Z) falls before the publication instant. The evidence is a publisher page, so no archive disclosure is needed. The Deep Dive links, such as the OpenAI math findings, may be later additions to the page, but the reading neither uses nor relies on them. There is no hindsight framing.","excerpt":null},{"dimension":"support","status":"uncertain","explanation":"Claim 3 and the summary sentence on business appeal attribute to Heaven a view the source gives as Cleanlab's hope. Heaven's own assessment covers only 'could be useful' and 'unlikely to be perfect'. Smallest repair: 'Cleanlab hopes the tool will make LLMs more attractive to businesses worried about invented answers; Heaven says the approach could be useful but is unlikely to be perfect.' The other claims are directly supported.","excerpt":"Cleanlab hopes that its tool will make large language models more attractive to businesses worried about how much stuff they invent."},{"dimension":"media","status":"pass","explanation":"No media was selected, and no verified media candidates were supplied.","excerpt":null}],"createdAt":"2026-11-20T05:43:27.329Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}