# Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study

[Canonical reading](https://escapemeridian.com/readings/eight-in-ten-llms-leak-system-prompt-secrets-nevan-an-agentic-b4e0d324)

Ana Clara Zoppi Serpa, Anderson Rocha and Hélio Pedrini report an audit of instruction-hierarchy enforcement across 363 text-in/text-out models in the OpenRouter catalog. Each model had a secret 16-character flag in its system prompt; an AI agent, steered by a human researcher, tried to persuade the model to emit it. The flag was exfiltrated from 296 catalog models, plus four models released after the catalog freeze, for 300 captures total. The researchers say each capture shows that a model ranked a user request above its system instruction at least once. Of the captures, 170 involved a non-English language pivot, most often French. The authors caution that the experiment does not show that models resist untested attacks, consistently leak semantically sensitive data, or are vulnerable to harmful-content jailbreaks. For practitioners, they recommend not storing secrets in system prompts, enforcing critical behavior outside the model with output validation and deterministic checks, and not letting an LLM judge decide consequential outcomes alone. The results concern this flag-extraction experiment; the abstract does not establish how models would behave under other attacks or with other secrets.

Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.

## Original source

arXiv cs.AI · Article

Original source date: Nov 16, 2026 (2026-11-16).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T15:32:53.223Z. First observed: 2026-11-18T05:04:04.391Z. Retrieved: 2026-11-18T15:31:53.696Z.

- [Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study](https://arxiv.org/abs/2611.21956)

## Limits

The reported result concerns attempts to extract a 16-character flag embedded in a system prompt; the abstract cautions that it does not establish behavior on untested attacks or consistent leakage of semantically sensitive data. The abstract describes the study and its findings but does not provide the attack techniques or per-model results in detail.


## Open question

How would the results change under attacks beyond the techniques tested in this experiment?


## Source-stated

The authors audited instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog.

> We audit instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog.

- [Supporting source](https://arxiv.org/abs/2611.21956)

## Source-stated

The experiment placed a 16-character flag in each model’s system prompt and used a human-steered AI agent to try to elicit it.

> Each model guards a secret 16-character flag set in its system prompt, while an artificial intelligence (AI) agent (OpenCode, powered by DeepSeek V4.1 Flash), steered by a human researcher, tries to persuade it to emit the flag.

- [Supporting source](https://arxiv.org/abs/2611.21956)

## Source-stated

The flag was captured from 296 of the 363 catalog models, plus four models released after the catalog freeze, for 300 captures total.

> The flag was exfiltrated from 296 of the 363 models, plus four models released after the initial catalog freeze, for 300 captures in total.

- [Supporting source](https://arxiv.org/abs/2611.21956)

## Source-stated

The authors say every capture shows that a model ranked a user request above its system instruction at least once.

> Each capture shows that a model ranked a user request above its system instruction at least once.

- [Supporting source](https://arxiv.org/abs/2611.21956)

## Source-stated

The experiment does not establish resistance to untested attacks, consistent leakage of semantically sensitive data, or vulnerability to harmful-content jailbreaks.

> It does not show that the model provably resists untested attacks, nor that it would consistently leak semantically sensitive data, nor that it is vulnerable to harmful-content jailbreaks.

- [Supporting source](https://arxiv.org/abs/2611.21956)

## Source-stated

The authors recommend keeping secrets out of system prompts, enforcing critical behavior outside the model, and not relying solely on an LLM judge for consequential outcomes.

> do not store secrets in system prompts, enforce critical behavior outside the model with output validation and deterministic checks, and do not let a large language model (LLM) judge be the sole decider of consequential outcomes.

- [Supporting source](https://arxiv.org/abs/2611.21956)

## Source and contributor context

Serpa, Ana Clara Zoppi, Rocha, Anderson, Pedrini, Hélio · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Agentic AI

AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.

- [Definition reference](https://developers.google.com/machine-learning/glossary#agentic)

## AI agent

An AI system that chooses and carries out actions toward a goal, often using software tools.

- [Definition reference](https://www.ibm.com/think/topics/ai-agents)

## Jailbreak

A prompt written to trick an AI system into ignoring its safety rules.

- [Definition reference](https://arxiv.org/abs/2307.02483)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: b2c84bcd-6570-4566-83f7-931ca1a0388d.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T15:32:53.223Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 06632b260e351dbcd047060b3a3a7101d0759345b7e0a6aaf97fe63f2c8fc3bf. Source text hash: a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8.


## Revision and source-check history

### Version 1 — 2026-11-18T15:32:53.223Z

Ana Clara Zoppi Serpa, Anderson Rocha and Hélio Pedrini report an audit of instruction-hierarchy enforcement across 363 text-in/text-out models in the OpenRouter catalog. Each model had a secret 16-character flag in its system prompt; an AI agent, steered by a human researcher, tried to persuade the model to emit it. The flag was exfiltrated from 296 catalog models, plus four models released after the catalog freeze, for 300 captures total. The researchers say each capture shows that a model ranked a user request above its system instruction at least once. Of the captures, 170 involved a non-English language pivot, most often French. The authors caution that the experiment does not show that models resist untested attacks, consistently leak semantically sensitive data, or are vulnerable to harmful-content jailbreaks. For practitioners, they recommend not storing secrets in system prompts, enforcing critical behavior outside the model with output validation and deterministic checks, and not letting an LLM judge decide consequential outcomes alone. The results concern this flag-extraction experiment; the abstract does not establish how models would behave under other attacks or with other secrets.

## Original source

arXiv cs.AI · Article

Original source date: Nov 16, 2026 (2026-11-16).

Date basis: Date basis not provided.

Added to Escape Meridian: 2026-11-18T15:32:53.223Z. First observed: 2026-11-18T05:04:04.391Z. Retrieved: 2026-11-18T15:31:53.696Z.

- [Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study](https://arxiv.org/abs/2611.21956)

## Limits

The reported result concerns attempts to extract a 16-character flag embedded in a system prompt; the abstract cautions that it does not establish behavior on untested attacks or consistent leakage of semantically sensitive data. The abstract describes the study and its findings but does not provide the attack techniques or per-model results in detail.


## Open question

How would the results change under attacks beyond the techniques tested in this experiment?


## Source-stated

The authors audited instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog.

> We audit instruction-hierarchy enforcement across 363 text-in/text-out models from the OpenRouter catalog.

- [Supporting source](https://arxiv.org/abs/2611.21956)

## Source-stated

The experiment placed a 16-character flag in each model’s system prompt and used a human-steered AI agent to try to elicit it.

> Each model guards a secret 16-character flag set in its system prompt, while an artificial intelligence (AI) agent (OpenCode, powered by DeepSeek V4.1 Flash), steered by a human researcher, tries to persuade it to emit the flag.

- [Supporting source](https://arxiv.org/abs/2611.21956)

## Source-stated

The flag was captured from 296 of the 363 catalog models, plus four models released after the catalog freeze, for 300 captures total.

> The flag was exfiltrated from 296 of the 363 models, plus four models released after the initial catalog freeze, for 300 captures in total.

- [Supporting source](https://arxiv.org/abs/2611.21956)

## Source-stated

The authors say every capture shows that a model ranked a user request above its system instruction at least once.

> Each capture shows that a model ranked a user request above its system instruction at least once.

- [Supporting source](https://arxiv.org/abs/2611.21956)

## Source-stated

The experiment does not establish resistance to untested attacks, consistent leakage of semantically sensitive data, or vulnerability to harmful-content jailbreaks.

> It does not show that the model provably resists untested attacks, nor that it would consistently leak semantically sensitive data, nor that it is vulnerable to harmful-content jailbreaks.

- [Supporting source](https://arxiv.org/abs/2611.21956)

## Source-stated

The authors recommend keeping secrets out of system prompts, enforcing critical behavior outside the model, and not relying solely on an LLM judge for consequential outcomes.

> do not store secrets in system prompts, enforce critical behavior outside the model with output validation and deterministic checks, and do not let a large language model (LLM) judge be the sole decider of consequential outcomes.

- [Supporting source](https://arxiv.org/abs/2611.21956)

## Source and contributor context

Serpa, Ana Clara Zoppi, Rocha, Anderson, Pedrini, Hélio · Article author

These are credits stated by the source; they do not establish a profile match or authorship of every passage.


## Source media

No suitable media was found by the supported page extraction.


## Terms and references

Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.


## Large language model

A language model with many learned numerical settings, used to process and generate language.

- [Definition reference](https://developers.google.com/machine-learning/glossary#large-language-model)

## Agentic AI

AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.

- [Definition reference](https://developers.google.com/machine-learning/glossary#agentic)

## AI agent

An AI system that chooses and carries out actions toward a goal, often using software tools.

- [Definition reference](https://www.ibm.com/think/topics/ai-agents)

## Jailbreak

A prompt written to trick an AI system into ignoring its safety rules.

- [Definition reference](https://arxiv.org/abs/2307.02483)

## arXiv

arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.

- [Explore this context](/entities/arxiv)
- [Reference](https://arxiv.org/)
- [Reference](https://info.arxiv.org/help/api/index.html)

## Editorial provenance

Version 1. Publication ID: b2c84bcd-6570-4566-83f7-931ca1a0388d.

Prepared by gpt-6-luna; grounding review by gpt-6-luna.

Source check: passed by anthropic/claude-opus-5.5 on 2026-11-18T15:32:53.223Z. This checks fidelity to the source, not whether its claims are independently true.

Editorial selection: accept; reasons: material\_development.

Observation: 06632b260e351dbcd047060b3a3a7101d0759345b7e0a6aaf97fe63f2c8fc3bf. Source text hash: a789061455f2a2d8470aa57070140ec342741c3bc46426e8fabd60569d9250a8.


### passed — 2026-11-18T15:32:53.223Z — anthropic/claude-opus-5.5

Publication: b2c84bcd-6570-4566-83f7-931ca1a0388d

attribution — pass: The study is correctly attributed to Ana Clara Zoppi Serpa, Anderson Rocha and Hélio Pedrini. Findings, caveats and practitioner lessons are framed as the authors' report, not as established fact or Escape Meridian opinion.

> Authors:Ana Clara Zoppi Serpa, Anderson Rocha, Hélio Pedrini

summary — pass: The summary leads with the actors and the experiment, a secret flag hidden in each model's system prompt that a human-steered agent tried to extract. It gives the result and the authors' caveats and lessons. Instruction hierarchy is explained in plain words as a model ranking a user request above its system instruction. Terms such as language pivot, exfiltrated and LLM judge would benefit from term-library tooltips but do not block understanding. The title is the source's own, and the question follows from the stated limits.

> Each capture shows that a model ranked a user request above its system instruction at least once.

date — pass: The originalPublished date of 2026-11-16 matches the v1 arXiv submission date. It is not the later feed date of 2026-11-18.

> \[Submitted on 16 Nov 2026\]

support — pass: All counts (363, 296, four later models, 300 captures, 170 non-English pivots), the setup, the caveats and the three lessons match the abstract. The caveat drops the source's qualifier 'provably'. Restoring it ('does not show that models provably resist untested attacks') would preserve exact scope, but the meaning is not materially changed. The closing limitation sentence stays within what the abstract supports.

> It does not show that the model provably resists untested attacks, nor that it would consistently leak semantically sensitive data, nor that it is vulnerable to harmful-content jailbreaks.

media — pass: The selected media is the verified direct link to the original arXiv abstract page. It is the most direct way to reach the source.

## Recorded references

Explicit citations in public readings to this reading. Shared topics are not citations. Other record types need an explicit reference adapter.

No public references recorded here yet.

[Reference data (JSON)](/readings/b4e0d324ca60d77fd8e9851070bbce5448894268d6c7624c34d48dd440532cd0/references.json)

## Corrections and challenges

[Send a correction or an opposing source](https://escapemeridian.com/corrections/)

[JSON record](https://escapemeridian.com/readings/eight-in-ten-llms-leak-system-prompt-secrets-nevan-an-agentic-b4e0d324.json)
