{"id":"8c5e00837a6a17c4ac27df950c5e4bce6c0260cc4ef6294b31a397e5359a81b9","title":"arXiv cs.AI: Toward Safety-Aware AI Agents in Realistic Social Engineering Attack Scenarios","argument":"The authors propose SEA (Social Engineering Attack Benchmark), an interactive environment for evaluating how AI agents handle social engineering attacks. Their motivation is that personal AI agents can take actions involving real money and sensitive data, while their security against these attacks has not been systematically evaluated. SEA covers daily-life domains and simulated applications, with tasks designed in consultation with anti-fraud specialists across Chinese and English settings. In evaluations of open-source and closed-source models, the authors report that every model was breached to some degree; proactive verification rates were below 32%. They also propose META-SEA, a framework in which attack tasks and defenders co-evolve. The authors report that training with META-SEA brought Qwen3-8B to defense performance comparable to closed-source models and reduced ungrounded action claims. Escape Meridian’s interpretation is that the work focuses attention on risks that may arise when agents act on a user’s behalf, beyond producing misleading text. The reported results come from the benchmark’s simulated tasks, so they do not by themselves establish how agents perform in real-world attacks. The abstract also reports GPT-6 results only in Chinese, limiting what this evidence says about that model across both settings.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence provided is the arXiv abstract, not the paper’s full methods or results; it does not provide detail for assessing the evaluation procedure. The reported evaluations use simulated applications and tasks, so their relationship to real-world attacks remains unclear. GPT-6 was evaluated only in Chinese, so the abstract provides no GPT-6 evaluation result for the English setting.","speaker":{"name":"He, Zhitao, Han, Chengru, Zhang, Chi Him, Fung, Yi R","roleAtSource":"Article author"},"source":{"title":"Toward Safety-Aware AI Agents in Realistic Social Engineering Attack Scenarios","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.10037","publishedAt":"2026-11-05","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors propose SEA as an interactive environment for evaluating AI agents in social engineering attack scenarios.","excerpt":"We propose SEA (Social Engineering Attack Benchmark), the first interactive environment for evaluating the intrinsic security capability of AI agents in social engineering attack scenarios","url":"https://arxiv.org/abs/2611.10037","relationship":"Source-stated"},{"text":"SEA includes tasks spanning Chinese and English settings, designed in consultation with anti-fraud specialists.","excerpt":"320 evaluation tasks designed in consultation with anti-fraud specialists and spanning two culturally distinct settings in Chinese and English.","url":"https://arxiv.org/abs/2611.10037","relationship":"Source-stated"},{"text":"The authors report that all evaluated models were breached to varying degrees and that proactive verification rates were below 32%.","excerpt":"Evaluation on 11 frontier open-source and closed-source models, with GPT-6 evaluated only in Chinese, reveals that all models are breached to varying degrees (Chinese attack success rates ranging from 16.9% to 84.3%) proactive verification rates universally below 32%.","url":"https://arxiv.org/abs/2611.10037","relationship":"Source-stated"},{"text":"The authors report that META-SEA training gave Qwen3-8B defense performance comparable to closed-source models and reduced ungrounded action claims.","excerpt":"With META-SEA training, Qwen3-8B achieves defense performance comparable to closed-source models while effectively reducing ungrounded action claims.","url":"https://arxiv.org/abs/2611.10037","relationship":"Source-stated"},{"text":"Because the reported evaluation uses simulated applications and tasks, its results alone do not establish performance in real-world attacks.","excerpt":"covering 8 daily-life domains, 30+ simulated applications, a 4-level difficulty gradient, and a 5-dimensional evaluation framework, with 320 evaluation tasks","url":"https://arxiv.org/abs/2611.10037","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10037","checkedAt":"2026-11-10T22:38:09.425Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would SEA’s findings and META-SEA training results carry over from the benchmark’s simulated tasks to real-world social engineering attacks?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Toward Safety-Aware AI Agents in Realistic Social Engineering Attack Scenarios","summary":"The authors propose SEA (Social Engineering Attack Benchmark), an interactive environment for evaluating how AI agents handle social engineering attacks. Their motivation is that personal AI agents can take actions involving real money and sensitive data, while their security against these attacks has not been systematically evaluated. SEA covers daily-life domains and simulated applications, with tasks designed in consultation with anti-fraud specialists across Chinese and English settings. In evaluations of open-source and closed-source models, the authors report that every model was breached to some degree; proactive verification rates were below 32%. They also propose META-SEA, a framework in which attack tasks and defenders co-evolve. The authors report that training with META-SEA brought Qwen3-8B to defense performance comparable to closed-source models and reduced ungrounded action claims. Escape Meridian’s interpretation is that the work focuses attention on risks that may arise when agents act on a user’s behalf, beyond producing misleading text. The reported results come from the benchmark’s simulated tasks, so they do not by themselves establish how agents perform in real-world attacks. The abstract also reports GPT-6 results only in Chinese, limiting what this evidence says about that model across both settings.","question":"How well would SEA’s findings and META-SEA training results carry over from the benchmark’s simulated tasks to real-world social engineering attacks?","claims":[{"text":"The authors propose SEA as an interactive environment for evaluating AI agents in social engineering attack scenarios.","excerpt":"We propose SEA (Social Engineering Attack Benchmark), the first interactive environment for evaluating the intrinsic security capability of AI agents in social engineering attack scenarios","support":"source-stated"},{"text":"SEA includes tasks spanning Chinese and English settings, designed in consultation with anti-fraud specialists.","excerpt":"320 evaluation tasks designed in consultation with anti-fraud specialists and spanning two culturally distinct settings in Chinese and English.","support":"source-stated"},{"text":"The authors report that all evaluated models were breached to varying degrees and that proactive verification rates were below 32%.","excerpt":"Evaluation on 11 frontier open-source and closed-source models, with GPT-6 evaluated only in Chinese, reveals that all models are breached to varying degrees (Chinese attack success rates ranging from 16.9% to 84.3%) proactive verification rates universally below 32%.","support":"source-stated"},{"text":"The authors report that META-SEA training gave Qwen3-8B defense performance comparable to closed-source models and reduced ungrounded action claims.","excerpt":"With META-SEA training, Qwen3-8B achieves defense performance comparable to closed-source models while effectively reducing ungrounded action claims.","support":"source-stated"},{"text":"Because the reported evaluation uses simulated applications and tasks, its results alone do not establish performance in real-world attacks.","excerpt":"covering 8 daily-life domains, 30+ simulated applications, a 4-level difficulty gradient, and a 5-dimensional evaluation framework, with 320 evaluation tasks","support":"interpretation"}],"limitations":["The evidence provided is the arXiv abstract, not the paper’s full methods or results; it does not provide detail for assessing the evaluation procedure.","The reported evaluations use simulated applications and tasks, so their relationship to real-world attacks remains unclear.","GPT-6 was evaluated only in Chinese, so the abstract provides no GPT-6 evaluation result for the English setting."],"mediaId":"b2f4156ff6af9612c9094fcee43b2a5b7c30ceb9bb6161f906e8181461238246","correctionReason":null,"id":"8c5e00837a6a17c4ac27df950c5e4bce6c0260cc4ef6294b31a397e5359a81b9","publicationId":"174d75a1-bbaa-4ae7-88d3-7ad3c9d42fa8","version":1,"sourceUrl":"https://arxiv.org/abs/2611.10037","sourceTitle":"Toward Safety-Aware AI Agents in Realistic Social Engineering Attack Scenarios","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-05","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:39:36.514Z","createdAt":"2026-11-10T21:34:03.078Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"b2f4156ff6af9612c9094fcee43b2a5b7c30ceb9bb6161f906e8181461238246","kind":"article","url":"https://arxiv.org/abs/2611.10037","sourceUrl":"https://arxiv.org/abs/2611.10037","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-10T21:33:06.362Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10037","checkedAt":"2026-11-10T22:38:09.425Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"b2f4156ff6af9612c9094fcee43b2a5b7c30ceb9bb6161f906e8181461238246","confidence":0.86,"provider":"typesafe","model":"jev-1.13.0","inputHash":"9e3c8bcf802401df6b6e367cf273d48412f1608264cb542536174b37b5d94178","decidedAt":"2026-11-10T21:33:07.676Z"},"provenance":{"textHash":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","observationId":"8a24d84333a0b06237e76abe8f057b589f5ee61b5798232495aab50987f49be1","retrievedAt":"2026-11-10T21:33:06.105Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"5519cc59-d689-4c27-8c0c-220b45dd2468"},"reviewer":{"model":"gpt-6-luna","runId":"64918e6f-62d0-4cc0-8d0a-e4a88caa5d8f"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-10T21:34:03.078Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"He, Zhitao","url":null,"kind":"unknown"},{"name":"Han, Chengru","url":null,"kind":"unknown"},{"name":"Zhang, Chi Him","url":null,"kind":"unknown"},{"name":"Fung, Yi R","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"He, Zhitao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"He, Zhitao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T22:38:09.425Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f","sourceUrl":"https://arxiv.org/abs/2611.10037","contentSha256":"f090ba1fffd7d96bb4a75946e0e41be83869dca8a9b114306801e1bf561f71d7","textHash":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","observedAt":"2026-11-10T22:38:09.425Z"}},{"actor":{"name":"Han, Chengru","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Han, Chengru","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T22:38:09.425Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f","sourceUrl":"https://arxiv.org/abs/2611.10037","contentSha256":"f090ba1fffd7d96bb4a75946e0e41be83869dca8a9b114306801e1bf561f71d7","textHash":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","observedAt":"2026-11-10T22:38:09.425Z"}},{"actor":{"name":"Zhang, Chi Him","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Chi Him","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T22:38:09.425Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f","sourceUrl":"https://arxiv.org/abs/2611.10037","contentSha256":"f090ba1fffd7d96bb4a75946e0e41be83869dca8a9b114306801e1bf561f71d7","textHash":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","observedAt":"2026-11-10T22:38:09.425Z"}},{"actor":{"name":"Fung, Yi R","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Fung, Yi R","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T22:38:09.425Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f","sourceUrl":"https://arxiv.org/abs/2611.10037","contentSha256":"f090ba1fffd7d96bb4a75946e0e41be83869dca8a9b114306801e1bf561f71d7","textHash":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","observedAt":"2026-11-10T22:38:09.425Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.10037","snapshotId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f","contentSha256":"f090ba1fffd7d96bb4a75946e0e41be83869dca8a9b114306801e1bf561f71d7","textHash":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","observedAt":"2026-11-10T22:38:09.425Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/8c5e00837a6a17c4ac27df950c5e4bce6c0260cc4ef6294b31a397e5359a81b9/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-toward-safety-aware-ai-agents-in-realistic-social-8c5e0083","formats":{"html":"/readings/arxiv-cs-ai-toward-safety-aware-ai-agents-in-realistic-social-8c5e0083","markdown":"/readings/arxiv-cs-ai-toward-safety-aware-ai-agents-in-realistic-social-8c5e0083.md","json":"/readings/arxiv-cs-ai-toward-safety-aware-ai-agents-in-realistic-social-8c5e0083.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-ai-agent","name":"AI agent","explanation":"An AI system that chooses and carries out actions toward a goal, often using software tools.","referenceUrl":"https://www.ibm.com/think/topics/ai-agents"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"8c5e00837a6a17c4ac27df950c5e4bce6c0260cc4ef6294b31a397e5359a81b9","revision":"174d75a1-bbaa-4ae7-88d3-7ad3c9d42fa8","revisionKind":"publication-id","title":"arXiv cs.AI: Toward Safety-Aware AI Agents in Realistic Social Engineering Attack Scenarios","url":"https://escapemeridian.com/readings/arxiv-cs-ai-toward-safety-aware-ai-agents-in-realistic-social-8c5e0083"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.10037","revision":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","revisionKind":"sha256-source-text","title":"Toward Safety-Aware AI Agents in Realistic Social Engineering Attack Scenarios","url":"https://arxiv.org/abs/2611.10037"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"8c5e00837a6a17c4ac27df950c5e4bce6c0260cc4ef6294b31a397e5359a81b9","title":"arXiv cs.AI: Toward Safety-Aware AI Agents in Realistic Social Engineering Attack Scenarios","argument":"The authors propose SEA (Social Engineering Attack Benchmark), an interactive environment for evaluating how AI agents handle social engineering attacks. Their motivation is that personal AI agents can take actions involving real money and sensitive data, while their security against these attacks has not been systematically evaluated. SEA covers daily-life domains and simulated applications, with tasks designed in consultation with anti-fraud specialists across Chinese and English settings. In evaluations of open-source and closed-source models, the authors report that every model was breached to some degree; proactive verification rates were below 32%. They also propose META-SEA, a framework in which attack tasks and defenders co-evolve. The authors report that training with META-SEA brought Qwen3-8B to defense performance comparable to closed-source models and reduced ungrounded action claims. Escape Meridian’s interpretation is that the work focuses attention on risks that may arise when agents act on a user’s behalf, beyond producing misleading text. The reported results come from the benchmark’s simulated tasks, so they do not by themselves establish how agents perform in real-world attacks. The abstract also reports GPT-6 results only in Chinese, limiting what this evidence says about that model across both settings.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence provided is the arXiv abstract, not the paper’s full methods or results; it does not provide detail for assessing the evaluation procedure. The reported evaluations use simulated applications and tasks, so their relationship to real-world attacks remains unclear. GPT-6 was evaluated only in Chinese, so the abstract provides no GPT-6 evaluation result for the English setting.","speaker":{"name":"He, Zhitao, Han, Chengru, Zhang, Chi Him, Fung, Yi R","roleAtSource":"Article author"},"source":{"title":"Toward Safety-Aware AI Agents in Realistic Social Engineering Attack Scenarios","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.10037","publishedAt":"2026-11-05","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors propose SEA as an interactive environment for evaluating AI agents in social engineering attack scenarios.","excerpt":"We propose SEA (Social Engineering Attack Benchmark), the first interactive environment for evaluating the intrinsic security capability of AI agents in social engineering attack scenarios","url":"https://arxiv.org/abs/2611.10037","relationship":"Source-stated"},{"text":"SEA includes tasks spanning Chinese and English settings, designed in consultation with anti-fraud specialists.","excerpt":"320 evaluation tasks designed in consultation with anti-fraud specialists and spanning two culturally distinct settings in Chinese and English.","url":"https://arxiv.org/abs/2611.10037","relationship":"Source-stated"},{"text":"The authors report that all evaluated models were breached to varying degrees and that proactive verification rates were below 32%.","excerpt":"Evaluation on 11 frontier open-source and closed-source models, with GPT-6 evaluated only in Chinese, reveals that all models are breached to varying degrees (Chinese attack success rates ranging from 16.9% to 84.3%) proactive verification rates universally below 32%.","url":"https://arxiv.org/abs/2611.10037","relationship":"Source-stated"},{"text":"The authors report that META-SEA training gave Qwen3-8B defense performance comparable to closed-source models and reduced ungrounded action claims.","excerpt":"With META-SEA training, Qwen3-8B achieves defense performance comparable to closed-source models while effectively reducing ungrounded action claims.","url":"https://arxiv.org/abs/2611.10037","relationship":"Source-stated"},{"text":"Because the reported evaluation uses simulated applications and tasks, its results alone do not establish performance in real-world attacks.","excerpt":"covering 8 daily-life domains, 30+ simulated applications, a 4-level difficulty gradient, and a 5-dimensional evaluation framework, with 320 evaluation tasks","url":"https://arxiv.org/abs/2611.10037","relationship":"Escape Meridian interpretation"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10037","checkedAt":"2026-11-10T22:38:09.425Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well would SEA’s findings and META-SEA training results carry over from the benchmark’s simulated tasks to real-world social engineering attacks?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: Toward Safety-Aware AI Agents in Realistic Social Engineering Attack Scenarios","summary":"The authors propose SEA (Social Engineering Attack Benchmark), an interactive environment for evaluating how AI agents handle social engineering attacks. Their motivation is that personal AI agents can take actions involving real money and sensitive data, while their security against these attacks has not been systematically evaluated. SEA covers daily-life domains and simulated applications, with tasks designed in consultation with anti-fraud specialists across Chinese and English settings. In evaluations of open-source and closed-source models, the authors report that every model was breached to some degree; proactive verification rates were below 32%. They also propose META-SEA, a framework in which attack tasks and defenders co-evolve. The authors report that training with META-SEA brought Qwen3-8B to defense performance comparable to closed-source models and reduced ungrounded action claims. Escape Meridian’s interpretation is that the work focuses attention on risks that may arise when agents act on a user’s behalf, beyond producing misleading text. The reported results come from the benchmark’s simulated tasks, so they do not by themselves establish how agents perform in real-world attacks. The abstract also reports GPT-6 results only in Chinese, limiting what this evidence says about that model across both settings.","question":"How well would SEA’s findings and META-SEA training results carry over from the benchmark’s simulated tasks to real-world social engineering attacks?","claims":[{"text":"The authors propose SEA as an interactive environment for evaluating AI agents in social engineering attack scenarios.","excerpt":"We propose SEA (Social Engineering Attack Benchmark), the first interactive environment for evaluating the intrinsic security capability of AI agents in social engineering attack scenarios","support":"source-stated"},{"text":"SEA includes tasks spanning Chinese and English settings, designed in consultation with anti-fraud specialists.","excerpt":"320 evaluation tasks designed in consultation with anti-fraud specialists and spanning two culturally distinct settings in Chinese and English.","support":"source-stated"},{"text":"The authors report that all evaluated models were breached to varying degrees and that proactive verification rates were below 32%.","excerpt":"Evaluation on 11 frontier open-source and closed-source models, with GPT-6 evaluated only in Chinese, reveals that all models are breached to varying degrees (Chinese attack success rates ranging from 16.9% to 84.3%) proactive verification rates universally below 32%.","support":"source-stated"},{"text":"The authors report that META-SEA training gave Qwen3-8B defense performance comparable to closed-source models and reduced ungrounded action claims.","excerpt":"With META-SEA training, Qwen3-8B achieves defense performance comparable to closed-source models while effectively reducing ungrounded action claims.","support":"source-stated"},{"text":"Because the reported evaluation uses simulated applications and tasks, its results alone do not establish performance in real-world attacks.","excerpt":"covering 8 daily-life domains, 30+ simulated applications, a 4-level difficulty gradient, and a 5-dimensional evaluation framework, with 320 evaluation tasks","support":"interpretation"}],"limitations":["The evidence provided is the arXiv abstract, not the paper’s full methods or results; it does not provide detail for assessing the evaluation procedure.","The reported evaluations use simulated applications and tasks, so their relationship to real-world attacks remains unclear.","GPT-6 was evaluated only in Chinese, so the abstract provides no GPT-6 evaluation result for the English setting."],"mediaId":"b2f4156ff6af9612c9094fcee43b2a5b7c30ceb9bb6161f906e8181461238246","correctionReason":null,"id":"8c5e00837a6a17c4ac27df950c5e4bce6c0260cc4ef6294b31a397e5359a81b9","publicationId":"174d75a1-bbaa-4ae7-88d3-7ad3c9d42fa8","version":1,"sourceUrl":"https://arxiv.org/abs/2611.10037","sourceTitle":"Toward Safety-Aware AI Agents in Realistic Social Engineering Attack Scenarios","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-05","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:39:36.514Z","createdAt":"2026-11-10T21:34:03.078Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"b2f4156ff6af9612c9094fcee43b2a5b7c30ceb9bb6161f906e8181461238246","kind":"article","url":"https://arxiv.org/abs/2611.10037","sourceUrl":"https://arxiv.org/abs/2611.10037","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-10T21:33:06.362Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10037","checkedAt":"2026-11-10T22:38:09.425Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"b2f4156ff6af9612c9094fcee43b2a5b7c30ceb9bb6161f906e8181461238246","confidence":0.86,"provider":"typesafe","model":"jev-1.13.0","inputHash":"9e3c8bcf802401df6b6e367cf273d48412f1608264cb542536174b37b5d94178","decidedAt":"2026-11-10T21:33:07.676Z"},"provenance":{"textHash":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","observationId":"8a24d84333a0b06237e76abe8f057b589f5ee61b5798232495aab50987f49be1","retrievedAt":"2026-11-10T21:33:06.105Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"5519cc59-d689-4c27-8c0c-220b45dd2468"},"reviewer":{"model":"gpt-6-luna","runId":"64918e6f-62d0-4cc0-8d0a-e4a88caa5d8f"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-10T21:34:03.078Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"He, Zhitao","url":null,"kind":"unknown"},{"name":"Han, Chengru","url":null,"kind":"unknown"},{"name":"Zhang, Chi Him","url":null,"kind":"unknown"},{"name":"Fung, Yi R","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"He, Zhitao","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"He, Zhitao","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T22:38:09.425Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f","sourceUrl":"https://arxiv.org/abs/2611.10037","contentSha256":"f090ba1fffd7d96bb4a75946e0e41be83869dca8a9b114306801e1bf561f71d7","textHash":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","observedAt":"2026-11-10T22:38:09.425Z"}},{"actor":{"name":"Han, Chengru","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Han, Chengru","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T22:38:09.425Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f","sourceUrl":"https://arxiv.org/abs/2611.10037","contentSha256":"f090ba1fffd7d96bb4a75946e0e41be83869dca8a9b114306801e1bf561f71d7","textHash":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","observedAt":"2026-11-10T22:38:09.425Z"}},{"actor":{"name":"Zhang, Chi Him","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhang, Chi Him","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T22:38:09.425Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f","sourceUrl":"https://arxiv.org/abs/2611.10037","contentSha256":"f090ba1fffd7d96bb4a75946e0e41be83869dca8a9b114306801e1bf561f71d7","textHash":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","observedAt":"2026-11-10T22:38:09.425Z"}},{"actor":{"name":"Fung, Yi R","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Fung, Yi R","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10037","locator":"head/meta[name=author|citation_author]"},"assertionId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-10T22:38:09.425Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f","sourceUrl":"https://arxiv.org/abs/2611.10037","contentSha256":"f090ba1fffd7d96bb4a75946e0e41be83869dca8a9b114306801e1bf561f71d7","textHash":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","observedAt":"2026-11-10T22:38:09.425Z"}}],"participationCoverage":{"total":4,"returned":4,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.10037","snapshotId":"ec68c1a91a9df78f2e0d9fb232d36b663d89b21f483f6afdbdf1ba252adab51f","contentSha256":"f090ba1fffd7d96bb4a75946e0e41be83869dca8a9b114306801e1bf561f71d7","textHash":"ab23df1fccc5fd8a649faadf7f55db225864376df0286d7414bb00f85f37d5ad","observedAt":"2026-11-10T22:38:09.425Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/8c5e00837a6a17c4ac27df950c5e4bce6c0260cc4ef6294b31a397e5359a81b9/source"}}}],"audits":[{"publicationId":"174d75a1-bbaa-4ae7-88d3-7ad3c9d42fa8","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading consistently attributes SEA, META-SEA and the reported results to the paper's authors as their proposals and reports. The arXiv listing identifies four authors. The title prefix uses the feed label 'arXiv cs.AI' even though the primary category is cs.CR, but cs.AI is a listed cross-category, so this does not misattribute the source.","excerpt":"Authors:Zhitao He, Chengru Han, Chi Him Zhang, Yi R Fung"},{"dimension":"summary","status":"pass","explanation":"The summary leads with the actor and the development: a benchmark plus a training framework. It gives the motivation and reports the headline results without inflating them. The title and question are faithful and non-leading. Terms such as 'proactive verification' and 'ungrounded action claims' are kept exactly and are suitable for term-library tooltips rather than paraphrase. Omitting the 16.9% to 84.3% Chinese attack success range and the count of 11 models is a minor loss of detail, not a distortion.","excerpt":"all models are breached to varying degrees (Chinese attack success rates ranging from 16.9% to 84.3%)"},{"dimension":"date","status":"pass","explanation":"originalPublished is 2026-11-05, which matches the arXiv v1 submission date. The feed date of 2026-11-10 is correctly not used as the publication date.","excerpt":"[Submitted on 5 Nov 2026]"},{"dimension":"support","status":"uncertain","explanation":"Most claims are directly source-stated. The labeled EM interpretation contrasts action risks with 'producing misleading text', but the abstract never mentions text outputs. That contrast is not backed by any of the draft's claim excerpts. It is only loosely grounded in the abstract's statement that agents execute actions involving money and data. Smallest repair: drop 'beyond producing misleading text', or add a claim with this excerpt and keep the inference conditional. The simulated-task caveat and the GPT-6 limitation are properly supported.","excerpt":"Personal AI agents are increasingly deployed on user devices to execute actions involving real money and sensitive data"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the original arXiv abstract page. This is the source itself and the most direct way to access it.","excerpt":null}],"createdAt":"2026-11-10T21:34:03.078Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}