{"id":"6aba85ab90d2b9e867a55e0128d712de1886a60722a140a2d49e3f8f99b67e39","title":"arXiv cs.AI: TherapyGym scores therapy chatbots on clinical fidelity and safety","argument":"The authors of TherapyGym argue that common chatbot evaluations—fluency metrics, preference tests and generic dialogue benchmarks—miss clinically important aspects of psychotherapy. Their framework evaluates fidelity to CBT techniques across multi-turn sessions using the Cognitive Therapy Rating Scale (CTRS), and assesses safety with labels for therapy-specific risks, such as failing to address harm or abuse. To audit LLM-based judges, they introduce THERAPYJUDGEBENCH, a set of 116 dialogues with 1,270 expert ratings for comparison and calibration against licensed clinicians. They also use CTRS and safety-based rewards in RL with configurable patient simulations. The authors report that models trained with TherapyGym improved on expert ratings: average CTRS rose from 0.10 to 0.60, and from 0.16 to 0.59 under LLM judges. These are improvements on the paper’s reported ratings, not reported outcomes in real therapy. The work’s central idea is to train against measures of therapeutic practice and safety, rather than general dialogue quality alone. Whether those measures and gains carry over to real-world use remains unresolved.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract reports rating improvements but does not give details of the evaluation setup or establish whether those gains carry over to real-world therapy use.","speaker":{"name":"Huang, Fangrui, Chbeir, Souhad, Khatua, Arpandeep, Wang, Sheng, Tan, Sijun, Ye, Kenan, Bailey, Lily, Daniel, Merryn, Louie, Ryan, Koyejo, Sanmi, Adeli, Ehsan","roleAtSource":"Article author"},"source":{"title":"TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2603.18008","publishedAt":"2026-02-23","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that common evaluation methods miss clinically important dimensions of psychotherapy.","excerpt":"prevailing evaluation methods--fluency metrics, preference tests, and generic dialogue benchmarks--fail to capture the clinically critical dimensions of psychotherapy.","url":"https://arxiv.org/abs/2603.18008","relationship":"Source-stated"},{"text":"TherapyGym measures fidelity to CBT techniques across multi-turn sessions and assesses therapy-specific safety risks.","excerpt":"Fidelity is measured using the Cognitive Therapy Rating Scale (CTRS), implemented as an automated pipeline that scores adherence to CBT techniques over multi-turn sessions. Safety is assessed using a multi-label annotation scheme, covering therapy-specific risks","url":"https://arxiv.org/abs/2603.18008","relationship":"Source-stated"},{"text":"The authors introduce THERAPYJUDGEBENCH, comprising 116 dialogues and 1,270 expert ratings, to audit and calibrate LLM-based judges against licensed clinicians.","excerpt":"a validation set of 116 dialogues with 1,270 expert ratings for auditing and calibration against licensed clinicians.","url":"https://arxiv.org/abs/2603.18008","relationship":"Source-stated"},{"text":"The authors report that models trained in TherapyGym improved on expert ratings, with the stated average CTRS scores rising from 0.10 to 0.60 and from 0.16 to 0.59 under LLM judges.","excerpt":"Models trained in THERAPYGYM improve on expert ratings, with average CTRS rising from 0.10 to 0.60 (and 0.16 to 0.59 under LLM judges).","url":"https://arxiv.org/abs/2603.18008","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2603.18008","checkedAt":"2026-11-09T23:36:07.651Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do TherapyGym’s reported rating gains carry over to real-world therapy use?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: TherapyGym scores therapy chatbots on clinical fidelity and safety","summary":"The authors of TherapyGym argue that common chatbot evaluations—fluency metrics, preference tests and generic dialogue benchmarks—miss clinically important aspects of psychotherapy. Their framework evaluates fidelity to CBT techniques across multi-turn sessions using the Cognitive Therapy Rating Scale (CTRS), and assesses safety with labels for therapy-specific risks, such as failing to address harm or abuse. To audit LLM-based judges, they introduce THERAPYJUDGEBENCH, a set of 116 dialogues with 1,270 expert ratings for comparison and calibration against licensed clinicians. They also use CTRS and safety-based rewards in RL with configurable patient simulations. The authors report that models trained with TherapyGym improved on expert ratings: average CTRS rose from 0.10 to 0.60, and from 0.16 to 0.59 under LLM judges. These are improvements on the paper’s reported ratings, not reported outcomes in real therapy. The work’s central idea is to train against measures of therapeutic practice and safety, rather than general dialogue quality alone. Whether those measures and gains carry over to real-world use remains unresolved.","question":"How well do TherapyGym’s reported rating gains carry over to real-world therapy use?","claims":[{"text":"The authors argue that common evaluation methods miss clinically important dimensions of psychotherapy.","excerpt":"prevailing evaluation methods--fluency metrics, preference tests, and generic dialogue benchmarks--fail to capture the clinically critical dimensions of psychotherapy.","support":"source-stated"},{"text":"TherapyGym measures fidelity to CBT techniques across multi-turn sessions and assesses therapy-specific safety risks.","excerpt":"Fidelity is measured using the Cognitive Therapy Rating Scale (CTRS), implemented as an automated pipeline that scores adherence to CBT techniques over multi-turn sessions. Safety is assessed using a multi-label annotation scheme, covering therapy-specific risks","support":"source-stated"},{"text":"The authors introduce THERAPYJUDGEBENCH, comprising 116 dialogues and 1,270 expert ratings, to audit and calibrate LLM-based judges against licensed clinicians.","excerpt":"a validation set of 116 dialogues with 1,270 expert ratings for auditing and calibration against licensed clinicians.","support":"source-stated"},{"text":"The authors report that models trained in TherapyGym improved on expert ratings, with the stated average CTRS scores rising from 0.10 to 0.60 and from 0.16 to 0.59 under LLM judges.","excerpt":"Models trained in THERAPYGYM improve on expert ratings, with average CTRS rising from 0.10 to 0.60 (and 0.16 to 0.59 under LLM judges).","support":"source-stated"}],"limitations":["The abstract reports rating improvements but does not give details of the evaluation setup or establish whether those gains carry over to real-world therapy use."],"mediaId":"76579242217ea543cbbbbef69205b4092b19a8bb4b79ce6ec1cb3472ff2ffc8e","correctionReason":null,"id":"6aba85ab90d2b9e867a55e0128d712de1886a60722a140a2d49e3f8f99b67e39","publicationId":"aaa82c4c-af33-4d92-8600-e31de179214c","version":1,"sourceUrl":"https://arxiv.org/abs/2603.18008","sourceTitle":"TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-02-23","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:06:38.847Z","createdAt":"2026-11-09T06:32:19.725Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"76579242217ea543cbbbbef69205b4092b19a8bb4b79ce6ec1cb3472ff2ffc8e","kind":"article","url":"https://arxiv.org/abs/2603.18008","sourceUrl":"https://arxiv.org/abs/2603.18008","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T06:31:30.172Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2603.18008","checkedAt":"2026-11-09T23:36:07.651Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"76579242217ea543cbbbbef69205b4092b19a8bb4b79ce6ec1cb3472ff2ffc8e","confidence":0.89,"provider":"typesafe","model":"jev-1.13.0","inputHash":"8d44eb245ebfabd160d4ca22e429f7f37db4cab9ee58813449e3969fa711fe90","decidedAt":"2026-11-09T06:31:30.753Z"},"provenance":{"textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observationId":"b70c94e9e269a6f8edbe23796810b1d712eeabdc13619c42785ffe89a83f2de6","retrievedAt":"2026-11-09T06:31:30.083Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"2203d4f6-82b5-4436-8cb2-ac8664557e3a"},"reviewer":{"model":"gpt-6-luna","runId":"57f684d3-a1dc-4bfd-81b2-9cfb5b239aef"}},"audit":{"status":"passed","reviewedAt":"2026-11-09T06:32:19.725Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Huang, Fangrui","url":null,"kind":"unknown"},{"name":"Chbeir, Souhad","url":null,"kind":"unknown"},{"name":"Khatua, Arpandeep","url":null,"kind":"unknown"},{"name":"Wang, Sheng","url":null,"kind":"unknown"},{"name":"Tan, Sijun","url":null,"kind":"unknown"},{"name":"Ye, Kenan","url":null,"kind":"unknown"},{"name":"Bailey, Lily","url":null,"kind":"unknown"},{"name":"Daniel, Merryn","url":null,"kind":"unknown"},{"name":"Louie, Ryan","url":null,"kind":"unknown"},{"name":"Koyejo, Sanmi","url":null,"kind":"unknown"},{"name":"Adeli, Ehsan","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Huang, Fangrui","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Huang, Fangrui","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Chbeir, Souhad","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chbeir, Souhad","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Khatua, Arpandeep","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Khatua, Arpandeep","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Wang, Sheng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wang, Sheng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Tan, Sijun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tan, Sijun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Ye, Kenan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ye, Kenan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Bailey, Lily","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bailey, Lily","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Daniel, Merryn","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Daniel, Merryn","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Louie, Ryan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Louie, Ryan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Koyejo, Sanmi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Koyejo, Sanmi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:9","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Adeli, Ehsan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Adeli, Ehsan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:10","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}}],"participationCoverage":{"total":11,"returned":11,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2603.18008","snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z","basis":"publisher-page-metadata","sourceLinks":53,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/6aba85ab90d2b9e867a55e0128d712de1886a60722a140a2d49e3f8f99b67e39/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-therapygym-scores-therapy-chatbots-on-clinical-6aba85ab","formats":{"html":"/readings/arxiv-cs-ai-therapygym-scores-therapy-chatbots-on-clinical-6aba85ab","markdown":"/readings/arxiv-cs-ai-therapygym-scores-therapy-chatbots-on-clinical-6aba85ab.md","json":"/readings/arxiv-cs-ai-therapygym-scores-therapy-chatbots-on-clinical-6aba85ab.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-reinforcement-learning","name":"Reinforcement learning","explanation":"Training by trial and error, in which a system is rewarded for the outcomes its designers want.","referenceUrl":"https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"6aba85ab90d2b9e867a55e0128d712de1886a60722a140a2d49e3f8f99b67e39","revision":"aaa82c4c-af33-4d92-8600-e31de179214c","revisionKind":"publication-id","title":"arXiv cs.AI: TherapyGym scores therapy chatbots on clinical fidelity and safety","url":"https://escapemeridian.com/readings/arxiv-cs-ai-therapygym-scores-therapy-chatbots-on-clinical-6aba85ab"},"source":{"type":"source","id":"https://arxiv.org/abs/2603.18008","revision":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","revisionKind":"sha256-source-text","title":"TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots","url":"https://arxiv.org/abs/2603.18008"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"6aba85ab90d2b9e867a55e0128d712de1886a60722a140a2d49e3f8f99b67e39","title":"arXiv cs.AI: TherapyGym scores therapy chatbots on clinical fidelity and safety","argument":"The authors of TherapyGym argue that common chatbot evaluations—fluency metrics, preference tests and generic dialogue benchmarks—miss clinically important aspects of psychotherapy. Their framework evaluates fidelity to CBT techniques across multi-turn sessions using the Cognitive Therapy Rating Scale (CTRS), and assesses safety with labels for therapy-specific risks, such as failing to address harm or abuse. To audit LLM-based judges, they introduce THERAPYJUDGEBENCH, a set of 116 dialogues with 1,270 expert ratings for comparison and calibration against licensed clinicians. They also use CTRS and safety-based rewards in RL with configurable patient simulations. The authors report that models trained with TherapyGym improved on expert ratings: average CTRS rose from 0.10 to 0.60, and from 0.16 to 0.59 under LLM judges. These are improvements on the paper’s reported ratings, not reported outcomes in real therapy. The work’s central idea is to train against measures of therapeutic practice and safety, rather than general dialogue quality alone. Whether those measures and gains carry over to real-world use remains unresolved.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The abstract reports rating improvements but does not give details of the evaluation setup or establish whether those gains carry over to real-world therapy use.","speaker":{"name":"Huang, Fangrui, Chbeir, Souhad, Khatua, Arpandeep, Wang, Sheng, Tan, Sijun, Ye, Kenan, Bailey, Lily, Daniel, Merryn, Louie, Ryan, Koyejo, Sanmi, Adeli, Ehsan","roleAtSource":"Article author"},"source":{"title":"TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2603.18008","publishedAt":"2026-02-23","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that common evaluation methods miss clinically important dimensions of psychotherapy.","excerpt":"prevailing evaluation methods--fluency metrics, preference tests, and generic dialogue benchmarks--fail to capture the clinically critical dimensions of psychotherapy.","url":"https://arxiv.org/abs/2603.18008","relationship":"Source-stated"},{"text":"TherapyGym measures fidelity to CBT techniques across multi-turn sessions and assesses therapy-specific safety risks.","excerpt":"Fidelity is measured using the Cognitive Therapy Rating Scale (CTRS), implemented as an automated pipeline that scores adherence to CBT techniques over multi-turn sessions. Safety is assessed using a multi-label annotation scheme, covering therapy-specific risks","url":"https://arxiv.org/abs/2603.18008","relationship":"Source-stated"},{"text":"The authors introduce THERAPYJUDGEBENCH, comprising 116 dialogues and 1,270 expert ratings, to audit and calibrate LLM-based judges against licensed clinicians.","excerpt":"a validation set of 116 dialogues with 1,270 expert ratings for auditing and calibration against licensed clinicians.","url":"https://arxiv.org/abs/2603.18008","relationship":"Source-stated"},{"text":"The authors report that models trained in TherapyGym improved on expert ratings, with the stated average CTRS scores rising from 0.10 to 0.60 and from 0.16 to 0.59 under LLM judges.","excerpt":"Models trained in THERAPYGYM improve on expert ratings, with average CTRS rising from 0.10 to 0.60 (and 0.16 to 0.59 under LLM judges).","url":"https://arxiv.org/abs/2603.18008","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2603.18008","checkedAt":"2026-11-09T23:36:07.651Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do TherapyGym’s reported rating gains carry over to real-world therapy use?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: TherapyGym scores therapy chatbots on clinical fidelity and safety","summary":"The authors of TherapyGym argue that common chatbot evaluations—fluency metrics, preference tests and generic dialogue benchmarks—miss clinically important aspects of psychotherapy. Their framework evaluates fidelity to CBT techniques across multi-turn sessions using the Cognitive Therapy Rating Scale (CTRS), and assesses safety with labels for therapy-specific risks, such as failing to address harm or abuse. To audit LLM-based judges, they introduce THERAPYJUDGEBENCH, a set of 116 dialogues with 1,270 expert ratings for comparison and calibration against licensed clinicians. They also use CTRS and safety-based rewards in RL with configurable patient simulations. The authors report that models trained with TherapyGym improved on expert ratings: average CTRS rose from 0.10 to 0.60, and from 0.16 to 0.59 under LLM judges. These are improvements on the paper’s reported ratings, not reported outcomes in real therapy. The work’s central idea is to train against measures of therapeutic practice and safety, rather than general dialogue quality alone. Whether those measures and gains carry over to real-world use remains unresolved.","question":"How well do TherapyGym’s reported rating gains carry over to real-world therapy use?","claims":[{"text":"The authors argue that common evaluation methods miss clinically important dimensions of psychotherapy.","excerpt":"prevailing evaluation methods--fluency metrics, preference tests, and generic dialogue benchmarks--fail to capture the clinically critical dimensions of psychotherapy.","support":"source-stated"},{"text":"TherapyGym measures fidelity to CBT techniques across multi-turn sessions and assesses therapy-specific safety risks.","excerpt":"Fidelity is measured using the Cognitive Therapy Rating Scale (CTRS), implemented as an automated pipeline that scores adherence to CBT techniques over multi-turn sessions. Safety is assessed using a multi-label annotation scheme, covering therapy-specific risks","support":"source-stated"},{"text":"The authors introduce THERAPYJUDGEBENCH, comprising 116 dialogues and 1,270 expert ratings, to audit and calibrate LLM-based judges against licensed clinicians.","excerpt":"a validation set of 116 dialogues with 1,270 expert ratings for auditing and calibration against licensed clinicians.","support":"source-stated"},{"text":"The authors report that models trained in TherapyGym improved on expert ratings, with the stated average CTRS scores rising from 0.10 to 0.60 and from 0.16 to 0.59 under LLM judges.","excerpt":"Models trained in THERAPYGYM improve on expert ratings, with average CTRS rising from 0.10 to 0.60 (and 0.16 to 0.59 under LLM judges).","support":"source-stated"}],"limitations":["The abstract reports rating improvements but does not give details of the evaluation setup or establish whether those gains carry over to real-world therapy use."],"mediaId":"76579242217ea543cbbbbef69205b4092b19a8bb4b79ce6ec1cb3472ff2ffc8e","correctionReason":null,"id":"6aba85ab90d2b9e867a55e0128d712de1886a60722a140a2d49e3f8f99b67e39","publicationId":"aaa82c4c-af33-4d92-8600-e31de179214c","version":1,"sourceUrl":"https://arxiv.org/abs/2603.18008","sourceTitle":"TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-02-23","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-05T05:06:38.847Z","createdAt":"2026-11-09T06:32:19.725Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"76579242217ea543cbbbbef69205b4092b19a8bb4b79ce6ec1cb3472ff2ffc8e","kind":"article","url":"https://arxiv.org/abs/2603.18008","sourceUrl":"https://arxiv.org/abs/2603.18008","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-09T06:31:30.172Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2603.18008","checkedAt":"2026-11-09T23:36:07.651Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"76579242217ea543cbbbbef69205b4092b19a8bb4b79ce6ec1cb3472ff2ffc8e","confidence":0.89,"provider":"typesafe","model":"jev-1.13.0","inputHash":"8d44eb245ebfabd160d4ca22e429f7f37db4cab9ee58813449e3969fa711fe90","decidedAt":"2026-11-09T06:31:30.753Z"},"provenance":{"textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observationId":"b70c94e9e269a6f8edbe23796810b1d712eeabdc13619c42785ffe89a83f2de6","retrievedAt":"2026-11-09T06:31:30.083Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"2203d4f6-82b5-4436-8cb2-ac8664557e3a"},"reviewer":{"model":"gpt-6-luna","runId":"57f684d3-a1dc-4bfd-81b2-9cfb5b239aef"}},"audit":{"status":"passed","reviewedAt":"2026-11-09T06:32:19.725Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Huang, Fangrui","url":null,"kind":"unknown"},{"name":"Chbeir, Souhad","url":null,"kind":"unknown"},{"name":"Khatua, Arpandeep","url":null,"kind":"unknown"},{"name":"Wang, Sheng","url":null,"kind":"unknown"},{"name":"Tan, Sijun","url":null,"kind":"unknown"},{"name":"Ye, Kenan","url":null,"kind":"unknown"},{"name":"Bailey, Lily","url":null,"kind":"unknown"},{"name":"Daniel, Merryn","url":null,"kind":"unknown"},{"name":"Louie, Ryan","url":null,"kind":"unknown"},{"name":"Koyejo, Sanmi","url":null,"kind":"unknown"},{"name":"Adeli, Ehsan","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Huang, Fangrui","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Huang, Fangrui","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Chbeir, Souhad","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Chbeir, Souhad","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Khatua, Arpandeep","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Khatua, Arpandeep","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Wang, Sheng","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wang, Sheng","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Tan, Sijun","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tan, Sijun","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Ye, Kenan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ye, Kenan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Bailey, Lily","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Bailey, Lily","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:6","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Daniel, Merryn","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Daniel, Merryn","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:7","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Louie, Ryan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Louie, Ryan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:8","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Koyejo, Sanmi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Koyejo, Sanmi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:9","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}},{"actor":{"name":"Adeli, Ehsan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Adeli, Ehsan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2603.18008","locator":"head/meta[name=author|citation_author]"},"assertionId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f:participation:10","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-09T23:36:07.651Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","sourceUrl":"https://arxiv.org/abs/2603.18008","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z"}}],"participationCoverage":{"total":11,"returned":11,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2603.18008","snapshotId":"60c9cc15c7b775357f19d18458811899e117fd207e699ccb0f9e7d9d4473012f","contentSha256":"510b76fca558b32713808a68608819a33e639fcb91d1a32b49bf0c012f3008bf","textHash":"ecc1aa29d984fc9f8849423e629ea36f873025289cbc5ff29031a7882549f2e3","observedAt":"2026-11-09T23:36:07.651Z","basis":"publisher-page-metadata","sourceLinks":53,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/6aba85ab90d2b9e867a55e0128d712de1886a60722a140a2d49e3f8f99b67e39/source"}}}],"audits":[{"publicationId":"aaa82c4c-af33-4d92-8600-e31de179214c","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The framework, critique of existing evaluations, benchmark and results are attributed to the paper's authors. Phrases like 'argue', 'introduce' and 'report' keep the claims as theirs. The arXiv cs.AI label matches the publisher field.","excerpt":"We introduce THERAPYGYM, a framework that evaluates and improves therapy chatbots along two clinical pillars: fidelity and safety."},{"dimension":"summary","status":"pass","explanation":"The reading accurately follows the abstract: the critique of fluency, preference and generic benchmarks; CTRS-based fidelity; safety labels; THERAPYJUDGEBENCH; RL training with simulated patients; and the reported score gains. The title and question are fair. CBT, LLM and RL are central acronyms the text does not define, so term-library tooltips should cover them. The surrounding sentence about training against measures of therapeutic practice makes the gist understandable. Omitting the NeurIPS acceptance note does not distort the meaning.","excerpt":"CTRS and safety-based rewards drive RL with configurable patient simulations spanning diverse symptom profiles."},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-02-23 matches the v1 submission date. It correctly ignores the 3 Nov 2026 revision and the November feed date.","excerpt":"[Submitted on 23 Feb 2026 (v1), last revised 3 Nov 2026 (this version, v2)]"},{"dimension":"support","status":"pass","explanation":"All four claims quote the abstract accurately, and the numbers match. The draft treats the gains as rating improvements, not real therapy outcomes, and leaves open whether they transfer to real-world use. It does not assert the abstract's broader 'safer in high-stakes use' framing as fact. It also does not imply measured safety gains, since the abstract reports only CTRS numbers. No unsupported beneficiaries or costs are invented.","excerpt":"Models trained in THERAPYGYM improve on expert ratings, with average CTRS rising from 0.10 to 0.60 (and 0.16 to 0.59 under LLM judges)."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the arXiv abstract page for this paper. It is the most direct way to access the source.","excerpt":null}],"createdAt":"2026-11-09T06:32:19.725Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}