# arXiv cs.AI: TherapyGym scores therapy chatbots on clinical fidelity and safety [Canonical reading](https://escapemeridian.com/readings/arxiv-cs-ai-therapygym-scores-therapy-chatbots-on-clinical-6aba85ab) The authors of TherapyGym argue that common chatbot evaluations—fluency metrics, preference tests and generic dialogue benchmarks—miss clinically important aspects of psychotherapy. Their framework evaluates fidelity to CBT techniques across multi-turn sessions using the Cognitive Therapy Rating Scale (CTRS), and assesses safety with labels for therapy-specific risks, such as failing to address harm or abuse. To audit LLM-based judges, they introduce THERAPYJUDGEBENCH, a set of 116 dialogues with 1,270 expert ratings for comparison and calibration against licensed clinicians. They also use CTRS and safety-based rewards in RL with configurable patient simulations. The authors report that models trained with TherapyGym improved on expert ratings: average CTRS rose from 0.10 to 0.60, and from 0.16 to 0.59 under LLM judges. These are improvements on the paper’s reported ratings, not reported outcomes in real therapy. The work’s central idea is to train against measures of therapeutic practice and safety, rather than general dialogue quality alone. Whether those measures and gains carry over to real-world use remains unresolved. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv cs.AI · Article Original source date: Feb 23, 2026 (2026-02-23). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-09T06:32:19.725Z. First observed: 2026-11-05T05:06:38.847Z. Retrieved: 2026-11-09T06:31:30.083Z. - [TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots](https://arxiv.org/abs/2603.18008) ## Limits The abstract reports rating improvements but does not give details of the evaluation setup or establish whether those gains carry over to real-world therapy use. ## Open question How well do TherapyGym’s reported rating gains carry over to real-world therapy use? ## Source-stated The authors argue that common evaluation methods miss clinically important dimensions of psychotherapy. > prevailing evaluation methods--fluency metrics, preference tests, and generic dialogue benchmarks--fail to capture the clinically critical dimensions of psychotherapy. - [Supporting source](https://arxiv.org/abs/2603.18008) ## Source-stated TherapyGym measures fidelity to CBT techniques across multi-turn sessions and assesses therapy-specific safety risks. > Fidelity is measured using the Cognitive Therapy Rating Scale (CTRS), implemented as an automated pipeline that scores adherence to CBT techniques over multi-turn sessions. Safety is assessed using a multi-label annotation scheme, covering therapy-specific risks - [Supporting source](https://arxiv.org/abs/2603.18008) ## Source-stated The authors introduce THERAPYJUDGEBENCH, comprising 116 dialogues and 1,270 expert ratings, to audit and calibrate LLM-based judges against licensed clinicians. > a validation set of 116 dialogues with 1,270 expert ratings for auditing and calibration against licensed clinicians. - [Supporting source](https://arxiv.org/abs/2603.18008) ## Source-stated The authors report that models trained in TherapyGym improved on expert ratings, with the stated average CTRS scores rising from 0.10 to 0.60 and from 0.16 to 0.59 under LLM judges. > Models trained in THERAPYGYM improve on expert ratings, with average CTRS rising from 0.10 to 0.60 (and 0.16 to 0.59 under LLM judges). - [Supporting source](https://arxiv.org/abs/2603.18008) ## Source and contributor context Huang, Fangrui, Chbeir, Souhad, Khatua, Arpandeep, Wang, Sheng, Tan, Sijun, Ye, Kenan, Bailey, Lily, Daniel, Merryn, Louie, Ryan, Koyejo, Sanmi, Adeli, Ehsan · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: aaa82c4c-af33-4d92-8600-e31de179214c. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T06:32:19.725Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: b70c94e9e269a6f8edbe23796810b1d712eeabdc13619c42785ffe89a83f2de6. Source text hash: ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3. ## Revision and source-check history ### Version 1 — 2026-11-09T06:32:19.725Z The authors of TherapyGym argue that common chatbot evaluations—fluency metrics, preference tests and generic dialogue benchmarks—miss clinically important aspects of psychotherapy. Their framework evaluates fidelity to CBT techniques across multi-turn sessions using the Cognitive Therapy Rating Scale (CTRS), and assesses safety with labels for therapy-specific risks, such as failing to address harm or abuse. To audit LLM-based judges, they introduce THERAPYJUDGEBENCH, a set of 116 dialogues with 1,270 expert ratings for comparison and calibration against licensed clinicians. They also use CTRS and safety-based rewards in RL with configurable patient simulations. The authors report that models trained with TherapyGym improved on expert ratings: average CTRS rose from 0.10 to 0.60, and from 0.16 to 0.59 under LLM judges. These are improvements on the paper’s reported ratings, not reported outcomes in real therapy. The work’s central idea is to train against measures of therapeutic practice and safety, rather than general dialogue quality alone. Whether those measures and gains carry over to real-world use remains unresolved. ## Original source arXiv cs.AI · Article Original source date: Feb 23, 2026 (2026-02-23). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-09T06:32:19.725Z. First observed: 2026-11-05T05:06:38.847Z. Retrieved: 2026-11-09T06:31:30.083Z. - [TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots](https://arxiv.org/abs/2603.18008) ## Limits The abstract reports rating improvements but does not give details of the evaluation setup or establish whether those gains carry over to real-world therapy use. ## Open question How well do TherapyGym’s reported rating gains carry over to real-world therapy use? ## Source-stated The authors argue that common evaluation methods miss clinically important dimensions of psychotherapy. > prevailing evaluation methods--fluency metrics, preference tests, and generic dialogue benchmarks--fail to capture the clinically critical dimensions of psychotherapy. - [Supporting source](https://arxiv.org/abs/2603.18008) ## Source-stated TherapyGym measures fidelity to CBT techniques across multi-turn sessions and assesses therapy-specific safety risks. > Fidelity is measured using the Cognitive Therapy Rating Scale (CTRS), implemented as an automated pipeline that scores adherence to CBT techniques over multi-turn sessions. Safety is assessed using a multi-label annotation scheme, covering therapy-specific risks - [Supporting source](https://arxiv.org/abs/2603.18008) ## Source-stated The authors introduce THERAPYJUDGEBENCH, comprising 116 dialogues and 1,270 expert ratings, to audit and calibrate LLM-based judges against licensed clinicians. > a validation set of 116 dialogues with 1,270 expert ratings for auditing and calibration against licensed clinicians. - [Supporting source](https://arxiv.org/abs/2603.18008) ## Source-stated The authors report that models trained in TherapyGym improved on expert ratings, with the stated average CTRS scores rising from 0.10 to 0.60 and from 0.16 to 0.59 under LLM judges. > Models trained in THERAPYGYM improve on expert ratings, with average CTRS rising from 0.10 to 0.60 (and 0.16 to 0.59 under LLM judges). - [Supporting source](https://arxiv.org/abs/2603.18008) ## Source and contributor context Huang, Fangrui, Chbeir, Souhad, Khatua, Arpandeep, Wang, Sheng, Tan, Sijun, Ye, Kenan, Bailey, Lily, Daniel, Merryn, Louie, Ryan, Koyejo, Sanmi, Adeli, Ehsan · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Benchmark A defined set of tests used to compare systems. Results depend on the tasks and measures selected. - [Definition reference](https://arxiv.org/abs/2211.09110) ## Large language model A language model with many learned numerical settings, used to process and generate language. - [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model) ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: aaa82c4c-af33-4d92-8600-e31de179214c. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: passed by anthropic/claude-opus-5.5 on 2026-11-09T06:32:19.725Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: b70c94e9e269a6f8edbe23796810b1d712eeabdc13619c42785ffe89a83f2de6. Source text hash: ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3. ### passed — 2026-11-09T06:32:19.725Z — anthropic/claude-opus-5.5 Publication: aaa82c4c-af33-4d92-8600-e31de179214c attribution — pass: The framework, critique of existing evaluations, benchmark and results are attributed to the paper's authors. Phrases like 'argue', 'introduce' and 'report' keep the claims as theirs. The arXiv cs.AI label matches the publisher field. > We introduce THERAPYGYM, a framework that evaluates and improves therapy chatbots along two clinical pillars: fidelity and safety. summary — pass: The reading accurately follows the abstract: the critique of fluency, preference and generic benchmarks; CTRS-based fidelity; safety labels; THERAPYJUDGEBENCH; RL training with simulated patients; and the reported score gains. The title and question are fair. CBT, LLM and RL are central acronyms the text does not define, so term-library tooltips should cover them. The surrounding sentence about training against measures of therapeutic practice makes the gist understandable. Omitting the NeurIPS acceptance note does not distort the meaning. > CTRS and safety-based rewards drive RL with configurable patient simulations spanning diverse symptom profiles. date — pass: originalPublished 2026-02-23 matches the v1 submission date. It correctly ignores the 3 Nov 2026 revision and the November feed date. > \[Submitted on 23 Feb 2026 (v1), last revised 3 Nov 2026 (this version, v2)\] support — pass: All four claims quote the abstract accurately, and the numbers match. The draft treats the gains as rating improvements, not real therapy outcomes, and leaves open whether they transfer to real-world use. It does not assert the abstract's broader 'safer in high-stakes use' framing as fact. It also does not imply measured safety gains, since the abstract reports only CTRS numbers. No unsupported beneficiaries or costs are invented. > Models trained in THERAPYGYM improve on expert ratings, with average CTRS rising from 0.10 to 0.60 (and 0.16 to 0.59 under LLM judges). media — pass: The selected media is the verified direct link to the arXiv abstract page for this paper. It is the most direct way to access the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/6aba85ab90d2b9e867a55e0128d712de1886a60722a140a2d49e3f8f99b67e39/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-cs-ai-therapygym-scores-therapy-chatbots-on-clinical-6aba85ab.json)