Wang, Zongqi, Wang, Rui, Wu, Yuchuan, Yang, Hua, Qiu, Bingxue, Gu, Tianle, Zhao, Bing, Yang, Yujiu, Ye, Jieping, Li, Yongbin · Oct 29, 2026
SocialArena authors propose a benchmark for LLM social intelligence
Markdown · JSON · Original source
The authors of SocialArena propose a for evaluating social intelligence in (LLMs), arguing that existing evaluations focus on isolated capabilities or single scenarios and that their reliability is largely unverified. Their framework defines social intelligence through role-playing ability, mental state reasoning, and behavioral reasoning. The authors use a role-playing dialogue paradigm with explicit character thinking to assess these capabilities together, and introduce U-TRIMF to synthesize social contexts with broad coverage. For evaluation, they say they invested $132.4K to recruit and train human annotators, who compare model responses and provide preference labels and natural-language critiques. The work aims to guide future LLM improvements; the abstract describes the benchmark and its evaluation approach, but gives no results showing how models perform or how reliable the evaluations are.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Oct 29, 2026 (2026-10-29).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T08:54:26.843Z. First observed: 2026-10-30T06:45:27.726Z. Retrieved: 2026-10-31T07:04:11.318Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The abstract describes the and evaluation approach but provides no results showing model performance or evaluation reliability. The authors describe U-TRIMF as guiding broad coverage of social contexts; the abstract does not specify the contexts included.
Open question
How does SocialArena’s human-annotated arena-style protocol perform in assessing evaluation reliability across the social contexts its authors seek to cover?
Source-stated
The authors introduce SocialArena as a benchmark intended to define and evaluate social intelligence in LLMs.
We introduce SocialArena, a benchmark for systematically defining and evaluating the social intelligence of large language models (LLMs).
Source-stated
The authors identify three capability dimensions: role-playing ability, mental state reasoning, and behavioral reasoning.
a unified three-tier framework that defines social intelligence along three capability dimensions: role-playing ability, mental state reasoning, and behavioral reasoning.
Source-stated
Their role-playing dialogue paradigm uses explicit character thinking to assess the three capabilities together in interactions they describe as realistic.
we design a role-playing dialogue paradigm with explicit character thinking for evaluation, enabling simultaneous assessment of these capabilities in a manner that mirrors realistic human interactions.
Source-stated
The authors introduce U-TRIMF to guide the synthesis of social contexts with broad coverage.
we introduce the U-TRIMF framework, which guides us in synthesizing social contexts with broad coverage.
Source-stated
The authors say they spent $132.4K recruiting and training human annotators for an arena-style evaluation protocol.
we invest $132.4K to recruit and train human annotators and adopt a human-annotated arena-style protocol
Source-stated
The protocol has annotators compare model responses and provide preference labels and natural-language critiques.
annotators compare model responses and provide preference labels along with natural-language critiques.
Source and contributor context
Wang, Zongqi, Wang, Rui, Wu, Yuchuan, Yang, Hua, Qiu, Bingxue, Gu, Tianle, Zhao, Bing, Yang, Yujiu, Ye, Jieping, Li, Yongbin · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 39634894-1a1b-4e4b-8e09-a1d610c29a40.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T08:54:26.843Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 7c0dc4cd74a84856f8266295c382d824ba6d8266888cf59892117f50fcd18bac. Source text hash: 4e9f7d57474d06e9cb5445d0534495e742a1c23e01dd7765fba8879e466c696f.
Revision and source-check history
Version 1 · 2026-11-11T08:54:26.843Z
SocialArena authors propose a benchmark for LLM social intelligence
The authors of SocialArena propose a for evaluating social intelligence in (LLMs), arguing that existing evaluations focus on isolated capabilities or single scenarios and that their reliability is largely unverified. Their framework defines social intelligence through role-playing ability, mental state reasoning, and behavioral reasoning. The authors use a role-playing dialogue paradigm with explicit character thinking to assess these capabilities together, and introduce U-TRIMF to synthesize social contexts with broad coverage. For evaluation, they say they invested $132.4K to recruit and train human annotators, who compare model responses and provide preference labels and natural-language critiques. The work aims to guide future LLM improvements; the abstract describes the benchmark and its evaluation approach, but gives no results showing how models perform or how reliable the evaluations are.
Original source
cs.AI · Article
Source details
Original source date: Oct 29, 2026 (2026-10-29).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-11T08:54:26.843Z. First observed: 2026-10-30T06:45:27.726Z. Retrieved: 2026-10-31T07:04:11.318Z.
Limits
The abstract describes the and evaluation approach but provides no results showing model performance or evaluation reliability. The authors describe U-TRIMF as guiding broad coverage of social contexts; the abstract does not specify the contexts included.
Open question
How does SocialArena’s human-annotated arena-style protocol perform in assessing evaluation reliability across the social contexts its authors seek to cover?
Source-stated
The authors introduce SocialArena as a benchmark intended to define and evaluate social intelligence in LLMs.
We introduce SocialArena, a benchmark for systematically defining and evaluating the social intelligence of large language models (LLMs).
Source-stated
The authors identify three capability dimensions: role-playing ability, mental state reasoning, and behavioral reasoning.
a unified three-tier framework that defines social intelligence along three capability dimensions: role-playing ability, mental state reasoning, and behavioral reasoning.
Source-stated
Their role-playing dialogue paradigm uses explicit character thinking to assess the three capabilities together in interactions they describe as realistic.
we design a role-playing dialogue paradigm with explicit character thinking for evaluation, enabling simultaneous assessment of these capabilities in a manner that mirrors realistic human interactions.
Source-stated
The authors introduce U-TRIMF to guide the synthesis of social contexts with broad coverage.
we introduce the U-TRIMF framework, which guides us in synthesizing social contexts with broad coverage.
Source-stated
The authors say they spent $132.4K recruiting and training human annotators for an arena-style evaluation protocol.
we invest $132.4K to recruit and train human annotators and adopt a human-annotated arena-style protocol
Source-stated
The protocol has annotators compare model responses and provide preference labels and natural-language critiques.
annotators compare model responses and provide preference labels along with natural-language critiques.
Source and contributor context
Wang, Zongqi, Wang, Rui, Wu, Yuchuan, Yang, Hua, Qiu, Bingxue, Gu, Tianle, Zhao, Bing, Yang, Yujiu, Ye, Jieping, Li, Yongbin · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Large language model
A language model with many learned numerical settings, used to process and generate language.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 39634894-1a1b-4e4b-8e09-a1d610c29a40.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-11T08:54:26.843Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 7c0dc4cd74a84856f8266295c382d824ba6d8266888cf59892117f50fcd18bac. Source text hash: 4e9f7d57474d06e9cb5445d0534495e742a1c23e01dd7765fba8879e466c696f.
uncertain · 2026-11-11T08:54:26.843Z · anthropic/claude-opus-5.5
Publication: 39634894-1a1b-4e4b-8e09-a1d610c29a40
attribution — pass: The reading consistently attributes the benchmark, framework, U-TRIMF, the $132.4K spending and the stated aims to the paper's authors. It does not present their assertions as established fact.
We introduce SocialArena, a benchmark for systematically defining and evaluating the social intelligence of large language models (LLMs).
summary — uncertain: The summary omits the abstract's stated reason the work matters: LLMs are increasingly used in mental health counseling, companionship chatbots and tutoring. Without this, a general reader gets a list of components but not the human stakes. Arena-style and mental state reasoning are also left unexplained, and U-TRIMF is not described beyond its role. Smallest repair: add one sentence citing these deployment settings as the authors' motivation, and add tooltips for arena-style and mental state reasoning. The title and question are otherwise faithful.
LLMs are increasingly deployed in socially interactive applications such as mental health counseling, companionship chatbots, and educational tutoring
date — pass: originalPublished is 2026-10-29. This matches the arXiv v1 submission date of 29 Oct 2026, not the later feed date of Oct 31.
[Submitted on 29 Oct 2026]
support — pass: Each claim matches the abstract verbatim in substance. The limitation is accurate: the abstract gives no performance or reliability results. The question about evaluation reliability follows from the authors' own stated concern and does not assume an outcome. No unsupported benefits or costs are implied.
existing evaluations remain fragmented focusing on isolated capabilities or single scenarios and evaluation reliability is largely unverified
media — pass: The selected media is the verified direct arXiv abstract page for this paper. It is the most direct way to access the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceA language model with many learned numerical settings, used to process and generate language.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition reference