{"id":"b4e0d324ca60d77fd8e9851070bbce5448894268d6c7624c34d48dd440532cd0","title":"Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study","argument":"Ana Clara Zoppi Serpa, Anderson Rocha and Hélio Pedrini report an audit of instruction-hierarchy enforcement across 363 text-in/text-out models in the OpenRouter catalog. Each model had a secret 16-character flag in its system prompt; an AI agent, steered by a human researcher, tried to persuade the model to emit it. The flag was exfiltrated from 296 catalog models, plus four models released after the catalog freeze, for 300 captures total. The researchers say each capture shows that a model ranked a user request above its system instruction at least once. Of the captures, 170 involved a non-English language pivot, most often French. The authors caution that the experiment does not show that models resist untested attacks, consistently leak semantically sensitive data, or are vulnerable to harmful-content jailbreaks. For practitioners, they recommend not storing secrets in system prompts, enforcing critical behavior outside the model with output validation and deterministic checks, and not letting an LLM judge decide consequential outcomes alone. The results concern this flag-extraction experiment; the abstract does not establish how models would behave under other attacks or with other secrets.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported result concerns attempts to extract a 16-character flag embedded in a system prompt; the abstract cautions that it does not establish behavior on untested attacks or consistent leakage of semantically sensitive data. The abstract describes the study and its findings but does not provide the attack techniques or per-model results in detail.","speaker":{"name":"Serpa, Ana Clara Zoppi, Rocha, Anderson, Pedrini, Hélio","roleAtSource":"Article author"},"source":{"title":"Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.21956","publishedAt":"2026-11-16","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors audited instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog.","excerpt":"We audit instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog.","url":"https://arxiv.org/abs/2611.21956","relationship":"Source-stated"},{"text":"The experiment placed a 16-character flag in each model’s system prompt and used a human-steered AI agent to try to elicit it.","excerpt":"Each model guards a secret 16-character flag set in its system prompt, while an artificial intelligence (AI) agent (OpenCode, powered by DeepSeek V4.1 Flash), steered by a human researcher, tries to persuade it to emit the flag.","url":"https://arxiv.org/abs/2611.21956","relationship":"Source-stated"},{"text":"The flag was captured from 296 of the 363 catalog models, plus four models released after the catalog freeze, for 300 captures total.","excerpt":"The flag was exfiltrated from 296 of the 363 models, plus four models released after the initial catalog freeze, for 300 captures in total.","url":"https://arxiv.org/abs/2611.21956","relationship":"Source-stated"},{"text":"The authors say every capture shows that a model ranked a user request above its system instruction at least once.","excerpt":"Each capture shows that a model ranked a user request above its system instruction at least once.","url":"https://arxiv.org/abs/2611.21956","relationship":"Source-stated"},{"text":"The experiment does not establish resistance to untested attacks, consistent leakage of semantically sensitive data, or vulnerability to harmful-content jailbreaks.","excerpt":"It does not show that the model provably resists untested attacks, nor that it would consistently leak semantically sensitive data, nor that it is vulnerable to harmful-content jailbreaks.","url":"https://arxiv.org/abs/2611.21956","relationship":"Source-stated"},{"text":"The authors recommend keeping secrets out of system prompts, enforcing critical behavior outside the model, and not relying solely on an LLM judge for consequential outcomes.","excerpt":"do not store secrets in system prompts, enforce critical behavior outside the model with output validation and deterministic checks, and do not let a large language model (LLM) judge be the sole decider of consequential outcomes.","url":"https://arxiv.org/abs/2611.21956","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21956","checkedAt":"2026-11-18T15:48:11.279Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would the results change under attacks beyond the techniques tested in this experiment?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study","summary":"Ana Clara Zoppi Serpa, Anderson Rocha and Hélio Pedrini report an audit of instruction-hierarchy enforcement across 363 text-in/text-out models in the OpenRouter catalog. Each model had a secret 16-character flag in its system prompt; an AI agent, steered by a human researcher, tried to persuade the model to emit it. The flag was exfiltrated from 296 catalog models, plus four models released after the catalog freeze, for 300 captures total. The researchers say each capture shows that a model ranked a user request above its system instruction at least once. Of the captures, 170 involved a non-English language pivot, most often French. The authors caution that the experiment does not show that models resist untested attacks, consistently leak semantically sensitive data, or are vulnerable to harmful-content jailbreaks. For practitioners, they recommend not storing secrets in system prompts, enforcing critical behavior outside the model with output validation and deterministic checks, and not letting an LLM judge decide consequential outcomes alone. The results concern this flag-extraction experiment; the abstract does not establish how models would behave under other attacks or with other secrets.","question":"How would the results change under attacks beyond the techniques tested in this experiment?","claims":[{"text":"The authors audited instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog.","excerpt":"We audit instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog.","support":"source-stated"},{"text":"The experiment placed a 16-character flag in each model’s system prompt and used a human-steered AI agent to try to elicit it.","excerpt":"Each model guards a secret 16-character flag set in its system prompt, while an artificial intelligence (AI) agent (OpenCode, powered by DeepSeek V4.1 Flash), steered by a human researcher, tries to persuade it to emit the flag.","support":"source-stated"},{"text":"The flag was captured from 296 of the 363 catalog models, plus four models released after the catalog freeze, for 300 captures total.","excerpt":"The flag was exfiltrated from 296 of the 363 models, plus four models released after the initial catalog freeze, for 300 captures in total.","support":"source-stated"},{"text":"The authors say every capture shows that a model ranked a user request above its system instruction at least once.","excerpt":"Each capture shows that a model ranked a user request above its system instruction at least once.","support":"source-stated"},{"text":"The experiment does not establish resistance to untested attacks, consistent leakage of semantically sensitive data, or vulnerability to harmful-content jailbreaks.","excerpt":"It does not show that the model provably resists untested attacks, nor that it would consistently leak semantically sensitive data, nor that it is vulnerable to harmful-content jailbreaks.","support":"source-stated"},{"text":"The authors recommend keeping secrets out of system prompts, enforcing critical behavior outside the model, and not relying solely on an LLM judge for consequential outcomes.","excerpt":"do not store secrets in system prompts, enforce critical behavior outside the model with output validation and deterministic checks, and do not let a large language model (LLM) judge be the sole decider of consequential outcomes.","support":"source-stated"}],"limitations":["The reported result concerns attempts to extract a 16-character flag embedded in a system prompt; the abstract cautions that it does not establish behavior on untested attacks or consistent leakage of semantically sensitive data.","The abstract describes the study and its findings but does not provide the attack techniques or per-model results in detail."],"mediaId":"5273e5daa7c44b40238132bd1f0636913d09ec1cc3b50916c9b2ba0c31018e2c","correctionReason":null,"id":"b4e0d324ca60d77fd8e9851070bbce5448894268d6c7624c34d48dd440532cd0","publicationId":"b2c84bcd-6570-4566-83f7-931ca1a0388d","version":1,"sourceUrl":"https://arxiv.org/abs/2611.21956","sourceTitle":"Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-16","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-18T05:04:04.391Z","createdAt":"2026-11-18T15:32:53.223Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"5273e5daa7c44b40238132bd1f0636913d09ec1cc3b50916c9b2ba0c31018e2c","kind":"article","url":"https://arxiv.org/abs/2611.21956","sourceUrl":"https://arxiv.org/abs/2611.21956","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-18T15:31:53.891Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21956","checkedAt":"2026-11-18T15:48:11.279Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"5273e5daa7c44b40238132bd1f0636913d09ec1cc3b50916c9b2ba0c31018e2c","confidence":0.84,"provider":"typesafe","model":"jev-1.13.0","inputHash":"edb5ecb5935a8132a8c93193f9e210da75813eb8661dcd92666616273af8f689","decidedAt":"2026-11-18T15:31:55.090Z"},"provenance":{"textHash":"a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8","observationId":"06632b260e351dbcd047060b3a3a7101d0759345b7e0a6aaf97fe63f2c8fc3bf","retrievedAt":"2026-11-18T15:31:53.696Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"88397058-ce8f-40d0-8109-4610c2eff744"},"reviewer":{"model":"gpt-6-luna","runId":"9a2fe709-cc37-4c94-86c3-5a023e47473f"}},"audit":{"status":"passed","reviewedAt":"2026-11-18T15:32:53.223Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Serpa, Ana Clara Zoppi","url":null,"kind":"unknown"},{"name":"Rocha, Anderson","url":null,"kind":"unknown"},{"name":"Pedrini, Hélio","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Serpa, Ana Clara Zoppi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Serpa, Ana Clara Zoppi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21956","locator":"head/meta[name=author|citation_author]"},"assertionId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T15:48:11.279Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee","sourceUrl":"https://arxiv.org/abs/2611.21956","contentSha256":"ede7c6eaf2dda802354188787ce5badedbe890ed1274387585ba36f6a4568d21","textHash":"a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8","observedAt":"2026-11-18T15:48:11.279Z"}},{"actor":{"name":"Rocha, Anderson","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Rocha, Anderson","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21956","locator":"head/meta[name=author|citation_author]"},"assertionId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T15:48:11.279Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee","sourceUrl":"https://arxiv.org/abs/2611.21956","contentSha256":"ede7c6eaf2dda802354188787ce5badedbe890ed1274387585ba36f6a4568d21","textHash":"a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8","observedAt":"2026-11-18T15:48:11.279Z"}},{"actor":{"name":"Pedrini, Hélio","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Pedrini, Hélio","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21956","locator":"head/meta[name=author|citation_author]"},"assertionId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T15:48:11.279Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee","sourceUrl":"https://arxiv.org/abs/2611.21956","contentSha256":"ede7c6eaf2dda802354188787ce5badedbe890ed1274387585ba36f6a4568d21","textHash":"a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8","observedAt":"2026-11-18T15:48:11.279Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.21956","snapshotId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee","contentSha256":"ede7c6eaf2dda802354188787ce5badedbe890ed1274387585ba36f6a4568d21","textHash":"a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8","observedAt":"2026-11-18T15:48:11.279Z","basis":"publisher-page-metadata","sourceLinks":40,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/b4e0d324ca60d77fd8e9851070bbce5448894268d6c7624c34d48dd440532cd0/source"}},"canonicalUrl":"https://escapemeridian.com/readings/eight-in-ten-llms-leak-system-prompt-secrets-nevan-an-agentic-b4e0d324","formats":{"html":"/readings/eight-in-ten-llms-leak-system-prompt-secrets-nevan-an-agentic-b4e0d324","markdown":"/readings/eight-in-ten-llms-leak-system-prompt-secrets-nevan-an-agentic-b4e0d324.md","json":"/readings/eight-in-ten-llms-leak-system-prompt-secrets-nevan-an-agentic-b4e0d324.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"conditions":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-agentic","name":"Agentic AI","explanation":"AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.","referenceUrl":"https://developers.google.com/machine-learning/glossary#agentic"},{"id":"term-ai-agent","name":"AI agent","explanation":"An AI system that chooses and carries out actions toward a goal, often using software tools.","referenceUrl":"https://www.ibm.com/think/topics/ai-agents"},{"id":"term-jailbreak","name":"Jailbreak","explanation":"A prompt written to trick an AI system into ignoring its safety rules.","referenceUrl":"https://arxiv.org/abs/2307.02483"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"b4e0d324ca60d77fd8e9851070bbce5448894268d6c7624c34d48dd440532cd0","revision":"b2c84bcd-6570-4566-83f7-931ca1a0388d","revisionKind":"publication-id","title":"Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study","url":"https://escapemeridian.com/readings/eight-in-ten-llms-leak-system-prompt-secrets-nevan-an-agentic-b4e0d324"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.21956","revision":"a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8","revisionKind":"sha256-source-text","title":"Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study","url":"https://arxiv.org/abs/2611.21956"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter."},"revisions":[{"id":"b4e0d324ca60d77fd8e9851070bbce5448894268d6c7624c34d48dd440532cd0","title":"Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study","argument":"Ana Clara Zoppi Serpa, Anderson Rocha and Hélio Pedrini report an audit of instruction-hierarchy enforcement across 363 text-in/text-out models in the OpenRouter catalog. Each model had a secret 16-character flag in its system prompt; an AI agent, steered by a human researcher, tried to persuade the model to emit it. The flag was exfiltrated from 296 catalog models, plus four models released after the catalog freeze, for 300 captures total. The researchers say each capture shows that a model ranked a user request above its system instruction at least once. Of the captures, 170 involved a non-English language pivot, most often French. The authors caution that the experiment does not show that models resist untested attacks, consistently leak semantically sensitive data, or are vulnerable to harmful-content jailbreaks. For practitioners, they recommend not storing secrets in system prompts, enforcing critical behavior outside the model with output validation and deterministic checks, and not letting an LLM judge decide consequential outcomes alone. The results concern this flag-extraction experiment; the abstract does not establish how models would behave under other attacks or with other secrets.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported result concerns attempts to extract a 16-character flag embedded in a system prompt; the abstract cautions that it does not establish behavior on untested attacks or consistent leakage of semantically sensitive data. The abstract describes the study and its findings but does not provide the attack techniques or per-model results in detail.","speaker":{"name":"Serpa, Ana Clara Zoppi, Rocha, Anderson, Pedrini, Hélio","roleAtSource":"Article author"},"source":{"title":"Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.21956","publishedAt":"2026-11-16","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors audited instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog.","excerpt":"We audit instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog.","url":"https://arxiv.org/abs/2611.21956","relationship":"Source-stated"},{"text":"The experiment placed a 16-character flag in each model’s system prompt and used a human-steered AI agent to try to elicit it.","excerpt":"Each model guards a secret 16-character flag set in its system prompt, while an artificial intelligence (AI) agent (OpenCode, powered by DeepSeek V4.1 Flash), steered by a human researcher, tries to persuade it to emit the flag.","url":"https://arxiv.org/abs/2611.21956","relationship":"Source-stated"},{"text":"The flag was captured from 296 of the 363 catalog models, plus four models released after the catalog freeze, for 300 captures total.","excerpt":"The flag was exfiltrated from 296 of the 363 models, plus four models released after the initial catalog freeze, for 300 captures in total.","url":"https://arxiv.org/abs/2611.21956","relationship":"Source-stated"},{"text":"The authors say every capture shows that a model ranked a user request above its system instruction at least once.","excerpt":"Each capture shows that a model ranked a user request above its system instruction at least once.","url":"https://arxiv.org/abs/2611.21956","relationship":"Source-stated"},{"text":"The experiment does not establish resistance to untested attacks, consistent leakage of semantically sensitive data, or vulnerability to harmful-content jailbreaks.","excerpt":"It does not show that the model provably resists untested attacks, nor that it would consistently leak semantically sensitive data, nor that it is vulnerable to harmful-content jailbreaks.","url":"https://arxiv.org/abs/2611.21956","relationship":"Source-stated"},{"text":"The authors recommend keeping secrets out of system prompts, enforcing critical behavior outside the model, and not relying solely on an LLM judge for consequential outcomes.","excerpt":"do not store secrets in system prompts, enforce critical behavior outside the model with output validation and deterministic checks, and do not let a large language model (LLM) judge be the sole decider of consequential outcomes.","url":"https://arxiv.org/abs/2611.21956","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21956","checkedAt":"2026-11-18T15:48:11.279Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How would the results change under attacks beyond the techniques tested in this experiment?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study","summary":"Ana Clara Zoppi Serpa, Anderson Rocha and Hélio Pedrini report an audit of instruction-hierarchy enforcement across 363 text-in/text-out models in the OpenRouter catalog. Each model had a secret 16-character flag in its system prompt; an AI agent, steered by a human researcher, tried to persuade the model to emit it. The flag was exfiltrated from 296 catalog models, plus four models released after the catalog freeze, for 300 captures total. The researchers say each capture shows that a model ranked a user request above its system instruction at least once. Of the captures, 170 involved a non-English language pivot, most often French. The authors caution that the experiment does not show that models resist untested attacks, consistently leak semantically sensitive data, or are vulnerable to harmful-content jailbreaks. For practitioners, they recommend not storing secrets in system prompts, enforcing critical behavior outside the model with output validation and deterministic checks, and not letting an LLM judge decide consequential outcomes alone. The results concern this flag-extraction experiment; the abstract does not establish how models would behave under other attacks or with other secrets.","question":"How would the results change under attacks beyond the techniques tested in this experiment?","claims":[{"text":"The authors audited instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog.","excerpt":"We audit instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog.","support":"source-stated"},{"text":"The experiment placed a 16-character flag in each model’s system prompt and used a human-steered AI agent to try to elicit it.","excerpt":"Each model guards a secret 16-character flag set in its system prompt, while an artificial intelligence (AI) agent (OpenCode, powered by DeepSeek V4.1 Flash), steered by a human researcher, tries to persuade it to emit the flag.","support":"source-stated"},{"text":"The flag was captured from 296 of the 363 catalog models, plus four models released after the catalog freeze, for 300 captures total.","excerpt":"The flag was exfiltrated from 296 of the 363 models, plus four models released after the initial catalog freeze, for 300 captures in total.","support":"source-stated"},{"text":"The authors say every capture shows that a model ranked a user request above its system instruction at least once.","excerpt":"Each capture shows that a model ranked a user request above its system instruction at least once.","support":"source-stated"},{"text":"The experiment does not establish resistance to untested attacks, consistent leakage of semantically sensitive data, or vulnerability to harmful-content jailbreaks.","excerpt":"It does not show that the model provably resists untested attacks, nor that it would consistently leak semantically sensitive data, nor that it is vulnerable to harmful-content jailbreaks.","support":"source-stated"},{"text":"The authors recommend keeping secrets out of system prompts, enforcing critical behavior outside the model, and not relying solely on an LLM judge for consequential outcomes.","excerpt":"do not store secrets in system prompts, enforce critical behavior outside the model with output validation and deterministic checks, and do not let a large language model (LLM) judge be the sole decider of consequential outcomes.","support":"source-stated"}],"limitations":["The reported result concerns attempts to extract a 16-character flag embedded in a system prompt; the abstract cautions that it does not establish behavior on untested attacks or consistent leakage of semantically sensitive data.","The abstract describes the study and its findings but does not provide the attack techniques or per-model results in detail."],"mediaId":"5273e5daa7c44b40238132bd1f0636913d09ec1cc3b50916c9b2ba0c31018e2c","correctionReason":null,"id":"b4e0d324ca60d77fd8e9851070bbce5448894268d6c7624c34d48dd440532cd0","publicationId":"b2c84bcd-6570-4566-83f7-931ca1a0388d","version":1,"sourceUrl":"https://arxiv.org/abs/2611.21956","sourceTitle":"Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-16","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-18T05:04:04.391Z","createdAt":"2026-11-18T15:32:53.223Z","historical":false,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"5273e5daa7c44b40238132bd1f0636913d09ec1cc3b50916c9b2ba0c31018e2c","kind":"article","url":"https://arxiv.org/abs/2611.21956","sourceUrl":"https://arxiv.org/abs/2611.21956","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-18T15:31:53.891Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.21956","checkedAt":"2026-11-18T15:48:11.279Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v9","mediaId":"5273e5daa7c44b40238132bd1f0636913d09ec1cc3b50916c9b2ba0c31018e2c","confidence":0.84,"provider":"typesafe","model":"jev-1.13.0","inputHash":"edb5ecb5935a8132a8c93193f9e210da75813eb8661dcd92666616273af8f689","decidedAt":"2026-11-18T15:31:55.090Z"},"provenance":{"textHash":"a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8","observationId":"06632b260e351dbcd047060b3a3a7101d0759345b7e0a6aaf97fe63f2c8fc3bf","retrievedAt":"2026-11-18T15:31:53.696Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"88397058-ce8f-40d0-8109-4610c2eff744"},"reviewer":{"model":"gpt-6-luna","runId":"9a2fe709-cc37-4c94-86c3-5a023e47473f"}},"audit":{"status":"passed","reviewedAt":"2026-11-18T15:32:53.223Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Serpa, Ana Clara Zoppi","url":null,"kind":"unknown"},{"name":"Rocha, Anderson","url":null,"kind":"unknown"},{"name":"Pedrini, Hélio","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Serpa, Ana Clara Zoppi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Serpa, Ana Clara Zoppi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21956","locator":"head/meta[name=author|citation_author]"},"assertionId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T15:48:11.279Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee","sourceUrl":"https://arxiv.org/abs/2611.21956","contentSha256":"ede7c6eaf2dda802354188787ce5badedbe890ed1274387585ba36f6a4568d21","textHash":"a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8","observedAt":"2026-11-18T15:48:11.279Z"}},{"actor":{"name":"Rocha, Anderson","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Rocha, Anderson","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21956","locator":"head/meta[name=author|citation_author]"},"assertionId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T15:48:11.279Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee","sourceUrl":"https://arxiv.org/abs/2611.21956","contentSha256":"ede7c6eaf2dda802354188787ce5badedbe890ed1274387585ba36f6a4568d21","textHash":"a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8","observedAt":"2026-11-18T15:48:11.279Z"}},{"actor":{"name":"Pedrini, Hélio","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Pedrini, Hélio","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.21956","locator":"head/meta[name=author|citation_author]"},"assertionId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-18T15:48:11.279Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee","sourceUrl":"https://arxiv.org/abs/2611.21956","contentSha256":"ede7c6eaf2dda802354188787ce5badedbe890ed1274387585ba36f6a4568d21","textHash":"a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8","observedAt":"2026-11-18T15:48:11.279Z"}}],"participationCoverage":{"total":3,"returned":3,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.21956","snapshotId":"9d0c4d89d171043b10de763f6d776f19e9836eb3075af8a6ee038121ab657eee","contentSha256":"ede7c6eaf2dda802354188787ce5badedbe890ed1274387585ba36f6a4568d21","textHash":"a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8","observedAt":"2026-11-18T15:48:11.279Z","basis":"publisher-page-metadata","sourceLinks":40,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/b4e0d324ca60d77fd8e9851070bbce5448894268d6c7624c34d48dd440532cd0/source"}}}],"audits":[{"publicationId":"b2c84bcd-6570-4566-83f7-931ca1a0388d","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The study is correctly attributed to Ana Clara Zoppi Serpa, Anderson Rocha and Hélio Pedrini. Findings, caveats and practitioner lessons are framed as the authors' report, not as established fact or Escape Meridian opinion.","excerpt":"Authors:Ana Clara Zoppi Serpa, Anderson Rocha, Hélio Pedrini"},{"dimension":"summary","status":"pass","explanation":"The summary leads with the actors and the experiment, a secret flag hidden in each model's system prompt that a human-steered agent tried to extract. It gives the result and the authors' caveats and lessons. Instruction hierarchy is explained in plain words as a model ranking a user request above its system instruction. Terms such as language pivot, exfiltrated and LLM judge would benefit from term-library tooltips but do not block understanding. The title is the source's own, and the question follows from the stated limits.","excerpt":"Each capture shows that a model ranked a user request above its system instruction at least once."},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-11-16 matches the v1 arXiv submission date. It is not the later feed date of 2026-11-18.","excerpt":"[Submitted on 16 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"All counts (363, 296, four later models, 300 captures, 170 non-English pivots), the setup, the caveats and the three lessons match the abstract. The caveat drops the source's qualifier 'provably'. Restoring it ('does not show that models provably resist untested attacks') would preserve exact scope, but the meaning is not materially changed. The closing limitation sentence stays within what the abstract supports.","excerpt":"It does not show that the model provably resists untested attacks, nor that it would consistently leak semantically sensitive data, nor that it is vulnerable to harmful-content jailbreaks."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified direct link to the original arXiv abstract page. It is the most direct way to reach the source.","excerpt":null}],"createdAt":"2026-11-18T15:32:53.223Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}