{"id":"9de785bc807b31a23774990c5277e7d1f564bdf64d9d792843fccfad24a989f2","title":"arXiv cs.AI: Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?","argument":"The paper’s authors compare LLM annotations of veteran-related social media posts with judgments from eight Veteran Subject Matter Experts (SMEs). They examine 15 open-source and 10 proprietary models on 149 tweets across five severity categories, testing prompt variants, inter-model agreement, and detection of high-severity crises. The authors report that selected LLMs agreed more consistently with one another than the human reviewers did, but warn that consistency did not necessarily mean agreement with human judgments or reliable detection of rare high-severity cases. Proprietary models generally agreed better with expert annotations and were more stable than open-source models. GPT-5.6-Terra had the best overall performance, while Qwen2.5-7B performed strongest at identifying Acute Crisis with Suicidal Ideation cases. Label-flipped demonstration experiments exposed model-specific sensitivity to misleading in-context examples; explanatory demonstrations did not consistently improve performance. The authors present LLMs as possible scalable complements to expert annotation, not established replacements, and say careful evaluation is needed before high-stakes mental health use. The abstract reports summary results, but does not give enough detail to assess how reliably models detect rare crises beyond this evaluation.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence presents the paper’s abstract rather than its full methods and results; it does not provide detailed performance measures for assessing detection of rare high-severity cases. The abstract describes results on 149 tweets and does not establish how well they generalize beyond the evaluated posts.","speaker":{"name":"Islam, K M Sajjadul, Yaramalla, Pranav, Johnson, Nadiyah, Flower, Mark, Madiraju, Praveen, Franco, Zeno","roleAtSource":"Article author"},"source":{"title":"Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.10048","publishedAt":"2026-11-05","dateBasis":"Date basis not provided."},"evidence":[{"text":"The study evaluates LLM crisis-severity annotations against judgments from eight Veteran Subject Matter Experts, using 149 tweets and five severity categories.","excerpt":"Using 149 tweets and five severity categories, we evaluate 15 open-source and 10 proprietary LLMs across eight prompt variants","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"},{"text":"Human reviewers had higher ordinal than nominal agreement, which the authors say indicates disagreements often involved neighboring severity levels.","excerpt":"Human reviewers showed higher ordinal than nominal agreement ({\\alpha} = 0.494), indicating that disagreements often occurred between neighboring severity levels.","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"},{"text":"Selected LLMs showed higher inter-model consistency, but the authors caution that consistency did not necessarily mean agreement with human judgments or reliable detection of rare high-severity cases.","excerpt":"However, higher model consistency did not necessarily correspond to agreement with human judgments or reliable detection of rare high-severity cases.","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"},{"text":"Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models; GPT-5.6-Terra performed best overall.","excerpt":"Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models. GPT-5.6-Terra achieved the best overall performance","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"},{"text":"Qwen2.5-7B, an open-source model, performed strongest in identifying Acute Crisis with Suicidal Ideation cases.","excerpt":"the open-source Qwen2.5-7B showed the strongest performance in identifying Acute Crisis with Suicidal Ideation cases.","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"},{"text":"Label-flipped demonstrations revealed model-specific sensitivity to misleading in-context examples; explanatory demonstrations did not consistently improve performance.","excerpt":"Label-flipped demonstration experiments also revealed model-specific sensitivity to misleading in-context examples, while explanatory demonstrations did not consistently improve performance.","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"},{"text":"The authors describe LLMs as promising scalable complements to expert annotation, while calling for careful evaluation before high-stakes mental health use.","excerpt":"Overall, LLMs show promise as scalable complements to expert annotation, but require careful evaluation before use in high-stakes mental health applications.","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10048","checkedAt":"2026-11-12T10:50:08.824Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How reliably would the models detect rare high-severity crises beyond the 149 tweets evaluated in this study?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?","summary":"The paper’s authors compare LLM annotations of veteran-related social media posts with judgments from eight Veteran Subject Matter Experts (SMEs). They examine 15 open-source and 10 proprietary models on 149 tweets across five severity categories, testing prompt variants, inter-model agreement, and detection of high-severity crises. The authors report that selected LLMs agreed more consistently with one another than the human reviewers did, but warn that consistency did not necessarily mean agreement with human judgments or reliable detection of rare high-severity cases. Proprietary models generally agreed better with expert annotations and were more stable than open-source models. GPT-5.6-Terra had the best overall performance, while Qwen2.5-7B performed strongest at identifying Acute Crisis with Suicidal Ideation cases. Label-flipped demonstration experiments exposed model-specific sensitivity to misleading in-context examples; explanatory demonstrations did not consistently improve performance. The authors present LLMs as possible scalable complements to expert annotation, not established replacements, and say careful evaluation is needed before high-stakes mental health use. The abstract reports summary results, but does not give enough detail to assess how reliably models detect rare crises beyond this evaluation.","question":"How reliably would the models detect rare high-severity crises beyond the 149 tweets evaluated in this study?","claims":[{"text":"The study evaluates LLM crisis-severity annotations against judgments from eight Veteran Subject Matter Experts, using 149 tweets and five severity categories.","excerpt":"Using 149 tweets and five severity categories, we evaluate 15 open-source and 10 proprietary LLMs across eight prompt variants","support":"source-stated"},{"text":"Human reviewers had higher ordinal than nominal agreement, which the authors say indicates disagreements often involved neighboring severity levels.","excerpt":"Human reviewers showed higher ordinal than nominal agreement ({\\alpha} = 0.494), indicating that disagreements often occurred between neighboring severity levels.","support":"source-stated"},{"text":"Selected LLMs showed higher inter-model consistency, but the authors caution that consistency did not necessarily mean agreement with human judgments or reliable detection of rare high-severity cases.","excerpt":"However, higher model consistency did not necessarily correspond to agreement with human judgments or reliable detection of rare high-severity cases.","support":"source-stated"},{"text":"Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models; GPT-5.6-Terra performed best overall.","excerpt":"Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models. GPT-5.6-Terra achieved the best overall performance","support":"source-stated"},{"text":"Qwen2.5-7B, an open-source model, performed strongest in identifying Acute Crisis with Suicidal Ideation cases.","excerpt":"the open-source Qwen2.5-7B showed the strongest performance in identifying Acute Crisis with Suicidal Ideation cases.","support":"source-stated"},{"text":"Label-flipped demonstrations revealed model-specific sensitivity to misleading in-context examples; explanatory demonstrations did not consistently improve performance.","excerpt":"Label-flipped demonstration experiments also revealed model-specific sensitivity to misleading in-context examples, while explanatory demonstrations did not consistently improve performance.","support":"source-stated"},{"text":"The authors describe LLMs as promising scalable complements to expert annotation, while calling for careful evaluation before high-stakes mental health use.","excerpt":"Overall, LLMs show promise as scalable complements to expert annotation, but require careful evaluation before use in high-stakes mental health applications.","support":"source-stated"}],"limitations":["The supplied evidence presents the paper’s abstract rather than its full methods and results; it does not provide detailed performance measures for assessing detection of rare high-severity cases.","The abstract describes results on 149 tweets and does not establish how well they generalize beyond the evaluated posts."],"mediaId":"50653b36e1edda3c81dffa217ab94081bd4a637f91fa6d3c108434c8e732427e","correctionReason":null,"id":"9de785bc807b31a23774990c5277e7d1f564bdf64d9d792843fccfad24a989f2","publicationId":"bad7b11a-83fe-45e6-831b-f7b72fc87d85","version":1,"sourceUrl":"https://arxiv.org/abs/2611.10048","sourceTitle":"Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-05","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:39:44.132Z","createdAt":"2026-11-12T10:19:41.684Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"50653b36e1edda3c81dffa217ab94081bd4a637f91fa6d3c108434c8e732427e","kind":"article","url":"https://arxiv.org/abs/2611.10048","sourceUrl":"https://arxiv.org/abs/2611.10048","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T10:18:32.983Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10048","checkedAt":"2026-11-12T10:50:08.824Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"50653b36e1edda3c81dffa217ab94081bd4a637f91fa6d3c108434c8e732427e","confidence":0.91,"provider":"typesafe","model":"jev-1.13.0","inputHash":"3cf7d129aa2004f6a48b1d1513379720635425c356a274d8ddab9ca7f2998f2b","decidedAt":"2026-11-12T10:18:35.276Z"},"provenance":{"textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observationId":"a181d671df9ea083bc73fec80673651b49e587c03505165d6565af385ef95735","retrievedAt":"2026-11-12T10:18:32.657Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"fe0ce197-1237-4f77-8de5-8f31174cd289"},"reviewer":{"model":"gpt-6-luna","runId":"e491cd24-934f-46f6-8e79-1254ea3e8124"}},"audit":{"status":"passed","reviewedAt":"2026-11-12T10:19:41.684Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Islam, K M Sajjadul","url":null,"kind":"unknown"},{"name":"Yaramalla, Pranav","url":null,"kind":"unknown"},{"name":"Johnson, Nadiyah","url":null,"kind":"unknown"},{"name":"Flower, Mark","url":null,"kind":"unknown"},{"name":"Madiraju, Praveen","url":null,"kind":"unknown"},{"name":"Franco, Zeno","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Islam, K M Sajjadul","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Islam, K M Sajjadul","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10048","locator":"head/meta[name=author|citation_author]"},"assertionId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T10:50:08.824Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","sourceUrl":"https://arxiv.org/abs/2611.10048","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z"}},{"actor":{"name":"Yaramalla, Pranav","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yaramalla, Pranav","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10048","locator":"head/meta[name=author|citation_author]"},"assertionId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T10:50:08.824Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","sourceUrl":"https://arxiv.org/abs/2611.10048","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z"}},{"actor":{"name":"Johnson, Nadiyah","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Johnson, Nadiyah","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10048","locator":"head/meta[name=author|citation_author]"},"assertionId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T10:50:08.824Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","sourceUrl":"https://arxiv.org/abs/2611.10048","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z"}},{"actor":{"name":"Flower, Mark","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Flower, Mark","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10048","locator":"head/meta[name=author|citation_author]"},"assertionId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T10:50:08.824Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","sourceUrl":"https://arxiv.org/abs/2611.10048","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z"}},{"actor":{"name":"Madiraju, Praveen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Madiraju, Praveen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10048","locator":"head/meta[name=author|citation_author]"},"assertionId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T10:50:08.824Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","sourceUrl":"https://arxiv.org/abs/2611.10048","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z"}},{"actor":{"name":"Franco, Zeno","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Franco, Zeno","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10048","locator":"head/meta[name=author|citation_author]"},"assertionId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T10:50:08.824Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","sourceUrl":"https://arxiv.org/abs/2611.10048","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.10048","snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/9de785bc807b31a23774990c5277e7d1f564bdf64d9d792843fccfad24a989f2/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-veterans-mental-health-crisis-severity-annotation-9de785bc","formats":{"html":"/readings/arxiv-cs-ai-veterans-mental-health-crisis-severity-annotation-9de785bc","markdown":"/readings/arxiv-cs-ai-veterans-mental-health-crisis-severity-annotation-9de785bc.md","json":"/readings/arxiv-cs-ai-veterans-mental-health-crisis-severity-annotation-9de785bc.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"9de785bc807b31a23774990c5277e7d1f564bdf64d9d792843fccfad24a989f2","revision":"bad7b11a-83fe-45e6-831b-f7b72fc87d85","revisionKind":"publication-id","title":"arXiv cs.AI: Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?","url":"https://escapemeridian.com/readings/arxiv-cs-ai-veterans-mental-health-crisis-severity-annotation-9de785bc"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.10048","revision":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","revisionKind":"sha256-source-text","title":"Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?","url":"https://arxiv.org/abs/2611.10048"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"9de785bc807b31a23774990c5277e7d1f564bdf64d9d792843fccfad24a989f2","title":"arXiv cs.AI: Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?","argument":"The paper’s authors compare LLM annotations of veteran-related social media posts with judgments from eight Veteran Subject Matter Experts (SMEs). They examine 15 open-source and 10 proprietary models on 149 tweets across five severity categories, testing prompt variants, inter-model agreement, and detection of high-severity crises. The authors report that selected LLMs agreed more consistently with one another than the human reviewers did, but warn that consistency did not necessarily mean agreement with human judgments or reliable detection of rare high-severity cases. Proprietary models generally agreed better with expert annotations and were more stable than open-source models. GPT-5.6-Terra had the best overall performance, while Qwen2.5-7B performed strongest at identifying Acute Crisis with Suicidal Ideation cases. Label-flipped demonstration experiments exposed model-specific sensitivity to misleading in-context examples; explanatory demonstrations did not consistently improve performance. The authors present LLMs as possible scalable complements to expert annotation, not established replacements, and say careful evaluation is needed before high-stakes mental health use. The abstract reports summary results, but does not give enough detail to assess how reliably models detect rare crises beyond this evaluation.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The supplied evidence presents the paper’s abstract rather than its full methods and results; it does not provide detailed performance measures for assessing detection of rare high-severity cases. The abstract describes results on 149 tweets and does not establish how well they generalize beyond the evaluated posts.","speaker":{"name":"Islam, K M Sajjadul, Yaramalla, Pranav, Johnson, Nadiyah, Flower, Mark, Madiraju, Praveen, Franco, Zeno","roleAtSource":"Article author"},"source":{"title":"Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.10048","publishedAt":"2026-11-05","dateBasis":"Date basis not provided."},"evidence":[{"text":"The study evaluates LLM crisis-severity annotations against judgments from eight Veteran Subject Matter Experts, using 149 tweets and five severity categories.","excerpt":"Using 149 tweets and five severity categories, we evaluate 15 open-source and 10 proprietary LLMs across eight prompt variants","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"},{"text":"Human reviewers had higher ordinal than nominal agreement, which the authors say indicates disagreements often involved neighboring severity levels.","excerpt":"Human reviewers showed higher ordinal than nominal agreement ({\\alpha} = 0.494), indicating that disagreements often occurred between neighboring severity levels.","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"},{"text":"Selected LLMs showed higher inter-model consistency, but the authors caution that consistency did not necessarily mean agreement with human judgments or reliable detection of rare high-severity cases.","excerpt":"However, higher model consistency did not necessarily correspond to agreement with human judgments or reliable detection of rare high-severity cases.","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"},{"text":"Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models; GPT-5.6-Terra performed best overall.","excerpt":"Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models. GPT-5.6-Terra achieved the best overall performance","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"},{"text":"Qwen2.5-7B, an open-source model, performed strongest in identifying Acute Crisis with Suicidal Ideation cases.","excerpt":"the open-source Qwen2.5-7B showed the strongest performance in identifying Acute Crisis with Suicidal Ideation cases.","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"},{"text":"Label-flipped demonstrations revealed model-specific sensitivity to misleading in-context examples; explanatory demonstrations did not consistently improve performance.","excerpt":"Label-flipped demonstration experiments also revealed model-specific sensitivity to misleading in-context examples, while explanatory demonstrations did not consistently improve performance.","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"},{"text":"The authors describe LLMs as promising scalable complements to expert annotation, while calling for careful evaluation before high-stakes mental health use.","excerpt":"Overall, LLMs show promise as scalable complements to expert annotation, but require careful evaluation before use in high-stakes mental health applications.","url":"https://arxiv.org/abs/2611.10048","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10048","checkedAt":"2026-11-12T10:50:08.824Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How reliably would the models detect rare high-severity crises beyond the 149 tweets evaluated in this study?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?","summary":"The paper’s authors compare LLM annotations of veteran-related social media posts with judgments from eight Veteran Subject Matter Experts (SMEs). They examine 15 open-source and 10 proprietary models on 149 tweets across five severity categories, testing prompt variants, inter-model agreement, and detection of high-severity crises. The authors report that selected LLMs agreed more consistently with one another than the human reviewers did, but warn that consistency did not necessarily mean agreement with human judgments or reliable detection of rare high-severity cases. Proprietary models generally agreed better with expert annotations and were more stable than open-source models. GPT-5.6-Terra had the best overall performance, while Qwen2.5-7B performed strongest at identifying Acute Crisis with Suicidal Ideation cases. Label-flipped demonstration experiments exposed model-specific sensitivity to misleading in-context examples; explanatory demonstrations did not consistently improve performance. The authors present LLMs as possible scalable complements to expert annotation, not established replacements, and say careful evaluation is needed before high-stakes mental health use. The abstract reports summary results, but does not give enough detail to assess how reliably models detect rare crises beyond this evaluation.","question":"How reliably would the models detect rare high-severity crises beyond the 149 tweets evaluated in this study?","claims":[{"text":"The study evaluates LLM crisis-severity annotations against judgments from eight Veteran Subject Matter Experts, using 149 tweets and five severity categories.","excerpt":"Using 149 tweets and five severity categories, we evaluate 15 open-source and 10 proprietary LLMs across eight prompt variants","support":"source-stated"},{"text":"Human reviewers had higher ordinal than nominal agreement, which the authors say indicates disagreements often involved neighboring severity levels.","excerpt":"Human reviewers showed higher ordinal than nominal agreement ({\\alpha} = 0.494), indicating that disagreements often occurred between neighboring severity levels.","support":"source-stated"},{"text":"Selected LLMs showed higher inter-model consistency, but the authors caution that consistency did not necessarily mean agreement with human judgments or reliable detection of rare high-severity cases.","excerpt":"However, higher model consistency did not necessarily correspond to agreement with human judgments or reliable detection of rare high-severity cases.","support":"source-stated"},{"text":"Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models; GPT-5.6-Terra performed best overall.","excerpt":"Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models. GPT-5.6-Terra achieved the best overall performance","support":"source-stated"},{"text":"Qwen2.5-7B, an open-source model, performed strongest in identifying Acute Crisis with Suicidal Ideation cases.","excerpt":"the open-source Qwen2.5-7B showed the strongest performance in identifying Acute Crisis with Suicidal Ideation cases.","support":"source-stated"},{"text":"Label-flipped demonstrations revealed model-specific sensitivity to misleading in-context examples; explanatory demonstrations did not consistently improve performance.","excerpt":"Label-flipped demonstration experiments also revealed model-specific sensitivity to misleading in-context examples, while explanatory demonstrations did not consistently improve performance.","support":"source-stated"},{"text":"The authors describe LLMs as promising scalable complements to expert annotation, while calling for careful evaluation before high-stakes mental health use.","excerpt":"Overall, LLMs show promise as scalable complements to expert annotation, but require careful evaluation before use in high-stakes mental health applications.","support":"source-stated"}],"limitations":["The supplied evidence presents the paper’s abstract rather than its full methods and results; it does not provide detailed performance measures for assessing detection of rare high-severity cases.","The abstract describes results on 149 tweets and does not establish how well they generalize beyond the evaluated posts."],"mediaId":"50653b36e1edda3c81dffa217ab94081bd4a637f91fa6d3c108434c8e732427e","correctionReason":null,"id":"9de785bc807b31a23774990c5277e7d1f564bdf64d9d792843fccfad24a989f2","publicationId":"bad7b11a-83fe-45e6-831b-f7b72fc87d85","version":1,"sourceUrl":"https://arxiv.org/abs/2611.10048","sourceTitle":"Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-05","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:39:44.132Z","createdAt":"2026-11-12T10:19:41.684Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"50653b36e1edda3c81dffa217ab94081bd4a637f91fa6d3c108434c8e732427e","kind":"article","url":"https://arxiv.org/abs/2611.10048","sourceUrl":"https://arxiv.org/abs/2611.10048","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T10:18:32.983Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10048","checkedAt":"2026-11-12T10:50:08.824Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"50653b36e1edda3c81dffa217ab94081bd4a637f91fa6d3c108434c8e732427e","confidence":0.91,"provider":"typesafe","model":"jev-1.13.0","inputHash":"3cf7d129aa2004f6a48b1d1513379720635425c356a274d8ddab9ca7f2998f2b","decidedAt":"2026-11-12T10:18:35.276Z"},"provenance":{"textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observationId":"a181d671df9ea083bc73fec80673651b49e587c03505165d6565af385ef95735","retrievedAt":"2026-11-12T10:18:32.657Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"fe0ce197-1237-4f77-8de5-8f31174cd289"},"reviewer":{"model":"gpt-6-luna","runId":"e491cd24-934f-46f6-8e79-1254ea3e8124"}},"audit":{"status":"passed","reviewedAt":"2026-11-12T10:19:41.684Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Islam, K M Sajjadul","url":null,"kind":"unknown"},{"name":"Yaramalla, Pranav","url":null,"kind":"unknown"},{"name":"Johnson, Nadiyah","url":null,"kind":"unknown"},{"name":"Flower, Mark","url":null,"kind":"unknown"},{"name":"Madiraju, Praveen","url":null,"kind":"unknown"},{"name":"Franco, Zeno","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Islam, K M Sajjadul","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Islam, K M Sajjadul","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10048","locator":"head/meta[name=author|citation_author]"},"assertionId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T10:50:08.824Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","sourceUrl":"https://arxiv.org/abs/2611.10048","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z"}},{"actor":{"name":"Yaramalla, Pranav","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Yaramalla, Pranav","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10048","locator":"head/meta[name=author|citation_author]"},"assertionId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T10:50:08.824Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","sourceUrl":"https://arxiv.org/abs/2611.10048","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z"}},{"actor":{"name":"Johnson, Nadiyah","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Johnson, Nadiyah","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10048","locator":"head/meta[name=author|citation_author]"},"assertionId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T10:50:08.824Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","sourceUrl":"https://arxiv.org/abs/2611.10048","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z"}},{"actor":{"name":"Flower, Mark","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Flower, Mark","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10048","locator":"head/meta[name=author|citation_author]"},"assertionId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T10:50:08.824Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","sourceUrl":"https://arxiv.org/abs/2611.10048","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z"}},{"actor":{"name":"Madiraju, Praveen","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Madiraju, Praveen","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10048","locator":"head/meta[name=author|citation_author]"},"assertionId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T10:50:08.824Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","sourceUrl":"https://arxiv.org/abs/2611.10048","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z"}},{"actor":{"name":"Franco, Zeno","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Franco, Zeno","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10048","locator":"head/meta[name=author|citation_author]"},"assertionId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T10:50:08.824Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","sourceUrl":"https://arxiv.org/abs/2611.10048","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.10048","snapshotId":"ba4fa9811d5e84e3cc357fafa313a313792a002d4b3feefb1c0fb50d812d11a0","contentSha256":"8fa25cf288bcd33d922d1cc595e480078f7551616b99e18987d7311a1d08b4e6","textHash":"c30e267f90f54f34d360b9ff2ba4cfe397b75a9a7bad73bb24c6aad9b71c79e4","observedAt":"2026-11-12T10:50:08.824Z","basis":"publisher-page-metadata","sourceLinks":46,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/9de785bc807b31a23774990c5277e7d1f564bdf64d9d792843fccfad24a989f2/source"}}}],"audits":[{"publicationId":"bad7b11a-83fe-45e6-831b-f7b72fc87d85","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading attributes the study, its findings and its cautions to the paper's authors on arXiv cs.AI. It does not present the results as established fact or as Escape Meridian's view.","excerpt":"This study evaluates LLM-based crisis-severity annotation of veteran-related social media posts against annotations from eight Veteran Subject Matter Experts (SMEs)."},{"dimension":"summary","status":"pass","explanation":"The reading faithfully covers the setup, the agreement contrast, the proprietary vs open-source results, the two standout models, the demonstration experiments and the authors' hedged conclusion. It keeps the central tension: models agree with each other more than humans do, but that agreement does not mean they match experts or catch rare crises. A general reader can follow it. Terms such as 'label-flipped demonstration' and 'in-context examples' are kept exactly. Tooltips are suggested for them, but the core meaning does not depend on them.","excerpt":"However, higher model consistency did not necessarily correspond to agreement with human judgments or reliable detection of rare high-severity cases."},{"dimension":"date","status":"pass","explanation":"The originalPublished value, 2026-11-05, matches the v1 submission date. It is not the 10 Nov v2 revision date or the feed date.","excerpt":"Submitted on 5 Nov 2026 (v1), last revised 10 Nov 2026 (this version, v2)"},{"dimension":"support","status":"pass","explanation":"Every claim matches the abstract text. The comparison of LLM inter-model agreement (0.8336) with human agreement (0.494) is drawn from stated figures. The question and limitations stay within the evidence: the abstract-only scope and the 149-tweet sample. They do not invent benefits or harms.","excerpt":"selected LLMs demonstrated substantially higher inter-model consistency, with ordinal agreement ({\\alpha}) reaching 0.8336 under the baseline prompt"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct arXiv abstract page for this paper. It is the most direct way for a reader to reach the source.","excerpt":null}],"createdAt":"2026-11-12T10:19:41.684Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}