{"id":"7f4ebc895c801d9abea2dca55cdd7af952980ffd944d20110d9f8d3147e86500","title":"Redwood Research’s Screwtape finds monitor blind spots by editing real agent transcripts","argument":"Gosha Kozhevnikov and Tyler describe Screwtape, a pipeline that edits real AI-agent transcripts to create harmful actions that evade monitors—systems that inspect an agent’s actions but not its reasoning. An attacker repeatedly revises each edit using feedback from a monitor, while a separate language-model verifier checks whether the proposed attack is plausible and severe enough.\n\nIn tests on 94 transcripts, the team says it found attacks that evaded its own monitors and a held-out monitor from a frontier developer; human reviewers confirmed some of the latter. The authors see the clearest use as finding specific blind spots developers can patch, not measuring overall safety: the method searches only a limited part of the possible attack space, and finding no attack provides little assurance. They also describe a tradeoff: stronger monitors may catch harmful actions, but could also help developers advance capable systems without addressing underlying misalignment. The post says the pipeline does not yet handle reasoning traces and that its final attacks still require human review.","topicIds":["institutions-and-trust","intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The post also says human review confirmed 6 of 17 attacks that evaded the held-out monitor, while its earlier summary says six of 35 attacks evaded that monitor and were confirmed. The relationship between these denominators is not explained in the supplied text. The pipeline lacks access to the original environment, relies on an LLM verifier that can accept invalid attacks, and requires human review of resulting attacks. Screwtape currently generates transcripts without reasoning traces, limiting its applicability to monitors that inspect both actions and reasoning. The authors caution that the pipeline’s findings do not quantify real-world failure rates; failure to find an attack provides little information about safety.","speaker":{"name":"Gosha Kozhevnikov, Tyler","roleAtSource":"Article author"},"source":{"title":"Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming","publisher":"Redwood Research blog","format":"Article","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","publishedAt":"2026-10-16T16:01:26+00:00","dateBasis":"Date stated by the publisher."},"evidence":[{"text":"Screwtape edits real agent transcripts and iterates attacks against an action-only monitor, using a separate language-model verifier to assess whether edits are plausible and harmful.","excerpt":"a pipeline that turns agent transcripts into realistic attacks that evade action-only monitors","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","relationship":"Source-stated"},{"text":"The authors report that tests on 94 transcripts produced 35 attacks that evaded their monitors; six also evaded a held-out monitor and were confirmed through human review.","excerpt":"This produced 35 attacks that evaded our monitors. Six of these 35 attacks also evaded the frontier developer’s monitor, and we confirmed them as valid through human review.","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","relationship":"Source-stated"},{"text":"The authors say Screwtape is more useful for finding monitor weaknesses to patch than for estimating safety, because it cannot provide quantitative safety estimates and searches only a limited attack space.","excerpt":"In its current form, the pipeline can’t show how often a monitor fails in normal deployment or produce a numerical safety estimate.","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","relationship":"Source-stated"},{"text":"The authors are uncertain whether helping developers strengthen frontier monitors is beneficial, since they could improve short-term safety while allowing capability development to continue without addressing underlying misalignment.","excerpt":"Strong monitors increase short-term safety by catching malign actions. On the other hand, they might let developers keep improving capabilities without addressing the underlying misalignment","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","relationship":"Source-stated"}],"sourceMedia":{"status":"found","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","checkedAt":"2026-10-19T14:20:41.808Z","items":[{"type":"image","url":"https://substackcdn.com/image/fetch/$s_!Epxv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc311c042-341f-4cc2-8a87-de830f497cd9_2048x781.png","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","attribution":"Redwood Research blog","basis":"publisher-embed","availability":"verified","verifiedAt":"2026-10-19T14:20:41.808Z"}],"note":"Media declared by the original publisher; availability checked separately from the editorial reading."},"assumptions":[],"whatWouldChangeView":{"attribution":"Open question","text":"What additional methodology would be needed to turn attacks found by Screwtape into a credible estimate of how often monitors fail in deployment?"},"editorialComparisons":[],"live":{"title":"Redwood Research’s Screwtape finds monitor blind spots by editing real agent transcripts","summary":"Gosha Kozhevnikov and Tyler describe Screwtape, a pipeline that edits real AI-agent transcripts to create harmful actions that evade monitors—systems that inspect an agent’s actions but not its reasoning. An attacker repeatedly revises each edit using feedback from a monitor, while a separate language-model verifier checks whether the proposed attack is plausible and severe enough.\n\nIn tests on 94 transcripts, the team says it found attacks that evaded its own monitors and a held-out monitor from a frontier developer; human reviewers confirmed some of the latter. The authors see the clearest use as finding specific blind spots developers can patch, not measuring overall safety: the method searches only a limited part of the possible attack space, and finding no attack provides little assurance. They also describe a tradeoff: stronger monitors may catch harmful actions, but could also help developers advance capable systems without addressing underlying misalignment. The post says the pipeline does not yet handle reasoning traces and that its final attacks still require human review.","question":"What additional methodology would be needed to turn attacks found by Screwtape into a credible estimate of how often monitors fail in deployment?","claims":[{"text":"Screwtape edits real agent transcripts and iterates attacks against an action-only monitor, using a separate language-model verifier to assess whether edits are plausible and harmful.","excerpt":"a pipeline that turns agent transcripts into realistic attacks that evade action-only monitors","support":"source-stated"},{"text":"The authors report that tests on 94 transcripts produced 35 attacks that evaded their monitors; six also evaded a held-out monitor and were confirmed through human review.","excerpt":"This produced 35 attacks that evaded our monitors. Six of these 35 attacks also evaded the frontier developer’s monitor, and we confirmed them as valid through human review.","support":"source-stated"},{"text":"The authors say Screwtape is more useful for finding monitor weaknesses to patch than for estimating safety, because it cannot provide quantitative safety estimates and searches only a limited attack space.","excerpt":"In its current form, the pipeline can’t show how often a monitor fails in normal deployment or produce a numerical safety estimate.","support":"source-stated"},{"text":"The authors are uncertain whether helping developers strengthen frontier monitors is beneficial, since they could improve short-term safety while allowing capability development to continue without addressing underlying misalignment.","excerpt":"Strong monitors increase short-term safety by catching malign actions. On the other hand, they might let developers keep improving capabilities without addressing the underlying misalignment","support":"source-stated"}],"limitations":["The post also says human review confirmed 6 of 17 attacks that evaded the held-out monitor, while its earlier summary says six of 35 attacks evaded that monitor and were confirmed. The relationship between these denominators is not explained in the supplied text.","The pipeline lacks access to the original environment, relies on an LLM verifier that can accept invalid attacks, and requires human review of resulting attacks.","Screwtape currently generates transcripts without reasoning traces, limiting its applicability to monitors that inspect both actions and reasoning.","The authors caution that the pipeline’s findings do not quantify real-world failure rates; failure to find an attack provides little information about safety."],"mediaId":"69308e7ae62d6c8c27e929c3c4143abc9415e0fbb76cd9194884e8997c01d8c3","correctionReason":null,"id":"7f4ebc895c801d9abea2dca55cdd7af952980ffd944d20110d9f8d3147e86500","publicationId":"1a85fff9-d6a9-404c-8ea3-70f797d944c6","version":1,"sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","sourceTitle":"Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming","publisher":"Redwood Research","publishedAt":"2026-10-16T16:01:26+00:00","originalPublished":{"value":"2026-10-16T16:01:26+00:00","precision":"instant","basis":"publisher_page"},"firstObservedAt":"2026-10-18T02:34:54.300Z","createdAt":"2026-10-19T13:02:24.396Z","historical":false,"entityIds":["redwood-research"],"topicIds":["institutions-and-trust","intelligence-and-discovery"],"media":[{"id":"69308e7ae62d6c8c27e929c3c4143abc9415e0fbb76cd9194884e8997c01d8c3","kind":"article","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-19T13:01:29.093Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."},{"id":"941d337a739c86c6040468c70e45e4bdfcc5e096f6a5fb7f865145bc49f3aa92","kind":"image","url":"https://substackcdn.com/image/fetch/$s_!Epxv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc311c042-341f-4cc2-8a87-de830f497cd9_2048x781.png","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","label":"View source image","provenance":"publisher_page","verification":{"status":"verified","checkedAt":"2026-10-19T14:20:41.808Z","method":"publisher_association_and_head","contentType":"image/png","cacheUntil":1792419941808},"recommendationReason":"Observed on the acquired publisher source; availability checked."}],"sourceMedia":{"status":"found","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","checkedAt":"2026-10-19T14:20:41.808Z","items":[{"type":"image","url":"https://substackcdn.com/image/fetch/$s_!Epxv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc311c042-341f-4cc2-8a87-de830f497cd9_2048x781.png","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","attribution":"Redwood Research blog","basis":"publisher-embed","availability":"verified","verifiedAt":"2026-10-19T14:20:41.808Z"}],"note":"Media declared by the original publisher; availability checked separately from the editorial reading."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"69308e7ae62d6c8c27e929c3c4143abc9415e0fbb76cd9194884e8997c01d8c3","confidence":0.95,"provider":"vercel","model":"typesafe-ai/jev","inputHash":"7366281303dc193e38007b24b6a1d9537733ca9fd57893aa5b3795b9cf82230c","decidedAt":"2026-10-19T13:01:31.827Z"},"provenance":{"textHash":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","observationId":"731ec8394c162cbeb1068e6be6a500b6035a187988ac7fa9d1be7953052f4712","retrievedAt":"2026-10-19T13:01:28.795Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"c1199954-7844-43de-8bf1-c33fe4a33123"},"reviewer":{"model":"gpt-6-luna","runId":"e6c7bf88-3751-4343-81fe-16660cdd6dc9"}},"audit":{"status":"passed","reviewedAt":"2026-10-19T13:02:24.396Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Gosha Kozhevnikov","url":"https://substack.com/@baraltiva","kind":"Person"},{"name":"Tyler","url":"https://substack.com/@tylord321","kind":"Person"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Gosha Kozhevnikov","url":"https://substack.com/@baraltiva","kind":"Person","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gosha Kozhevnikov","status":"stated","provenance":{"basis":"article-jsonld","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","locator":"document/html[1]/body[2]/div[1]/div[1]/div[1]/script[0]@37513.author"},"assertionId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf:participation:0","sourceVersion":{"snapshotId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","contentSha256":"dfcb8ddbc3cb87d090be3591e64cc98a7869690900662273fc2b42bd1353f60d","textHash":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","observedAt":"2026-10-19T14:20:41.808Z"}},{"actor":{"name":"Tyler","url":"https://substack.com/@tylord321","kind":"Person","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tyler","status":"stated","provenance":{"basis":"article-jsonld","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","locator":"document/html[1]/body[2]/div[1]/div[1]/div[1]/script[0]@37513.author"},"assertionId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf:participation:1","sourceVersion":{"snapshotId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","contentSha256":"dfcb8ddbc3cb87d090be3591e64cc98a7869690900662273fc2b42bd1353f60d","textHash":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","observedAt":"2026-10-19T14:20:41.808Z"}},{"actor":{"name":"Gosha Kozhevnikov","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gosha Kozhevnikov","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","locator":"head/meta[name=author|citation_author]"},"assertionId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf:participation:2","sourceVersion":{"snapshotId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","contentSha256":"dfcb8ddbc3cb87d090be3591e64cc98a7869690900662273fc2b42bd1353f60d","textHash":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","observedAt":"2026-10-19T14:20:41.808Z"}},{"actor":{"name":"Redwood Research blog","url":"https://blog.redwoodresearch.org/","kind":"unknown","actorId":null},"role":"publisher","scope":{"kind":"publication","locator":null},"literalCredit":"Redwood Research blog","status":"stated","provenance":{"basis":"article-publisher","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","locator":"document/html[1]/body[2]/div[1]/div[1]/div[1]/script[0]@37513.publisher"},"assertionId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf:participation:3","sourceVersion":{"snapshotId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","contentSha256":"dfcb8ddbc3cb87d090be3591e64cc98a7869690900662273fc2b42bd1353f60d","textHash":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","observedAt":"2026-10-19T14:20:41.808Z"}}],"distribution":{"service":"Substack","basis":"registered-source-adapter","publicationUrl":"https://blog.redwoodresearch.org"},"publisher":{"name":"Redwood Research blog","url":"https://blog.redwoodresearch.org/"},"sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","snapshotId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf","contentSha256":"dfcb8ddbc3cb87d090be3591e64cc98a7869690900662273fc2b42bd1353f60d","textHash":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","observedAt":"2026-10-19T14:20:41.808Z","basis":"publisher-page-metadata","sourceLinks":27,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/7f4ebc895c801d9abea2dca55cdd7af952980ffd944d20110d9f8d3147e86500/source"}},"canonicalUrl":"https://escapemeridian.com/readings/7f4ebc895c801d9abea2dca55cdd7af952980ffd944d20110d9f8d3147e86500","formats":{"html":"/readings/7f4ebc895c801d9abea2dca55cdd7af952980ffd944d20110d9f8d3147e86500","markdown":"/readings/7f4ebc895c801d9abea2dca55cdd7af952980ffd944d20110d9f8d3147e86500.md","json":"/readings/7f4ebc895c801d9abea2dca55cdd7af952980ffd944d20110d9f8d3147e86500.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"}],"entities":[{"id":"redwood-research","name":"Redwood Research","kind":"organization","context":"Specific empirical AI-control and monitor-failure work, including counterexamples.","referenceUrls":["https://www.dwarkesh.com/p/ryan-greenblatt","https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","https://www.redwoodresearch.org/research"],"path":"/entities/redwood-research","json":"/entities/redwood-research.json","markdown":"/entities/redwood-research.md"}]},"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"7f4ebc895c801d9abea2dca55cdd7af952980ffd944d20110d9f8d3147e86500","revision":"1a85fff9-d6a9-404c-8ea3-70f797d944c6","revisionKind":"publication-id","title":"Redwood Research’s Screwtape finds monitor blind spots by editing real agent transcripts","url":"https://escapemeridian.com/readings/7f4ebc895c801d9abea2dca55cdd7af952980ffd944d20110d9f8d3147e86500"},"source":{"type":"source","id":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","revision":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","revisionKind":"sha256-source-text","title":"Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"7f4ebc895c801d9abea2dca55cdd7af952980ffd944d20110d9f8d3147e86500","title":"Redwood Research’s Screwtape finds monitor blind spots by editing real agent transcripts","argument":"Gosha Kozhevnikov and Tyler describe Screwtape, a pipeline that edits real AI-agent transcripts to create harmful actions that evade monitors—systems that inspect an agent’s actions but not its reasoning. An attacker repeatedly revises each edit using feedback from a monitor, while a separate language-model verifier checks whether the proposed attack is plausible and severe enough.\n\nIn tests on 94 transcripts, the team says it found attacks that evaded its own monitors and a held-out monitor from a frontier developer; human reviewers confirmed some of the latter. The authors see the clearest use as finding specific blind spots developers can patch, not measuring overall safety: the method searches only a limited part of the possible attack space, and finding no attack provides little assurance. They also describe a tradeoff: stronger monitors may catch harmful actions, but could also help developers advance capable systems without addressing underlying misalignment. The post says the pipeline does not yet handle reasoning traces and that its final attacks still require human review.","topicIds":["institutions-and-trust","intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The post also says human review confirmed 6 of 17 attacks that evaded the held-out monitor, while its earlier summary says six of 35 attacks evaded that monitor and were confirmed. The relationship between these denominators is not explained in the supplied text. The pipeline lacks access to the original environment, relies on an LLM verifier that can accept invalid attacks, and requires human review of resulting attacks. Screwtape currently generates transcripts without reasoning traces, limiting its applicability to monitors that inspect both actions and reasoning. The authors caution that the pipeline’s findings do not quantify real-world failure rates; failure to find an attack provides little information about safety.","speaker":{"name":"Gosha Kozhevnikov, Tyler","roleAtSource":"Article author"},"source":{"title":"Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming","publisher":"Redwood Research blog","format":"Article","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","publishedAt":"2026-10-16T16:01:26+00:00","dateBasis":"Date stated by the publisher."},"evidence":[{"text":"Screwtape edits real agent transcripts and iterates attacks against an action-only monitor, using a separate language-model verifier to assess whether edits are plausible and harmful.","excerpt":"a pipeline that turns agent transcripts into realistic attacks that evade action-only monitors","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","relationship":"Source-stated"},{"text":"The authors report that tests on 94 transcripts produced 35 attacks that evaded their monitors; six also evaded a held-out monitor and were confirmed through human review.","excerpt":"This produced 35 attacks that evaded our monitors. Six of these 35 attacks also evaded the frontier developer’s monitor, and we confirmed them as valid through human review.","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","relationship":"Source-stated"},{"text":"The authors say Screwtape is more useful for finding monitor weaknesses to patch than for estimating safety, because it cannot provide quantitative safety estimates and searches only a limited attack space.","excerpt":"In its current form, the pipeline can’t show how often a monitor fails in normal deployment or produce a numerical safety estimate.","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","relationship":"Source-stated"},{"text":"The authors are uncertain whether helping developers strengthen frontier monitors is beneficial, since they could improve short-term safety while allowing capability development to continue without addressing underlying misalignment.","excerpt":"Strong monitors increase short-term safety by catching malign actions. On the other hand, they might let developers keep improving capabilities without addressing the underlying misalignment","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","relationship":"Source-stated"}],"sourceMedia":{"status":"found","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","checkedAt":"2026-10-19T14:20:41.808Z","items":[{"type":"image","url":"https://substackcdn.com/image/fetch/$s_!Epxv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc311c042-341f-4cc2-8a87-de830f497cd9_2048x781.png","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","attribution":"Redwood Research blog","basis":"publisher-embed","availability":"verified","verifiedAt":"2026-10-19T14:20:41.808Z"}],"note":"Media declared by the original publisher; availability checked separately from the editorial reading."},"assumptions":[],"whatWouldChangeView":{"attribution":"Open question","text":"What additional methodology would be needed to turn attacks found by Screwtape into a credible estimate of how often monitors fail in deployment?"},"editorialComparisons":[],"live":{"title":"Redwood Research’s Screwtape finds monitor blind spots by editing real agent transcripts","summary":"Gosha Kozhevnikov and Tyler describe Screwtape, a pipeline that edits real AI-agent transcripts to create harmful actions that evade monitors—systems that inspect an agent’s actions but not its reasoning. An attacker repeatedly revises each edit using feedback from a monitor, while a separate language-model verifier checks whether the proposed attack is plausible and severe enough.\n\nIn tests on 94 transcripts, the team says it found attacks that evaded its own monitors and a held-out monitor from a frontier developer; human reviewers confirmed some of the latter. The authors see the clearest use as finding specific blind spots developers can patch, not measuring overall safety: the method searches only a limited part of the possible attack space, and finding no attack provides little assurance. They also describe a tradeoff: stronger monitors may catch harmful actions, but could also help developers advance capable systems without addressing underlying misalignment. The post says the pipeline does not yet handle reasoning traces and that its final attacks still require human review.","question":"What additional methodology would be needed to turn attacks found by Screwtape into a credible estimate of how often monitors fail in deployment?","claims":[{"text":"Screwtape edits real agent transcripts and iterates attacks against an action-only monitor, using a separate language-model verifier to assess whether edits are plausible and harmful.","excerpt":"a pipeline that turns agent transcripts into realistic attacks that evade action-only monitors","support":"source-stated"},{"text":"The authors report that tests on 94 transcripts produced 35 attacks that evaded their monitors; six also evaded a held-out monitor and were confirmed through human review.","excerpt":"This produced 35 attacks that evaded our monitors. Six of these 35 attacks also evaded the frontier developer’s monitor, and we confirmed them as valid through human review.","support":"source-stated"},{"text":"The authors say Screwtape is more useful for finding monitor weaknesses to patch than for estimating safety, because it cannot provide quantitative safety estimates and searches only a limited attack space.","excerpt":"In its current form, the pipeline can’t show how often a monitor fails in normal deployment or produce a numerical safety estimate.","support":"source-stated"},{"text":"The authors are uncertain whether helping developers strengthen frontier monitors is beneficial, since they could improve short-term safety while allowing capability development to continue without addressing underlying misalignment.","excerpt":"Strong monitors increase short-term safety by catching malign actions. On the other hand, they might let developers keep improving capabilities without addressing the underlying misalignment","support":"source-stated"}],"limitations":["The post also says human review confirmed 6 of 17 attacks that evaded the held-out monitor, while its earlier summary says six of 35 attacks evaded that monitor and were confirmed. The relationship between these denominators is not explained in the supplied text.","The pipeline lacks access to the original environment, relies on an LLM verifier that can accept invalid attacks, and requires human review of resulting attacks.","Screwtape currently generates transcripts without reasoning traces, limiting its applicability to monitors that inspect both actions and reasoning.","The authors caution that the pipeline’s findings do not quantify real-world failure rates; failure to find an attack provides little information about safety."],"mediaId":"69308e7ae62d6c8c27e929c3c4143abc9415e0fbb76cd9194884e8997c01d8c3","correctionReason":null,"id":"7f4ebc895c801d9abea2dca55cdd7af952980ffd944d20110d9f8d3147e86500","publicationId":"1a85fff9-d6a9-404c-8ea3-70f797d944c6","version":1,"sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","sourceTitle":"Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming","publisher":"Redwood Research","publishedAt":"2026-10-16T16:01:26+00:00","originalPublished":{"value":"2026-10-16T16:01:26+00:00","precision":"instant","basis":"publisher_page"},"firstObservedAt":"2026-10-18T02:34:54.300Z","createdAt":"2026-10-19T13:02:24.396Z","historical":false,"entityIds":["redwood-research"],"topicIds":["institutions-and-trust","intelligence-and-discovery"],"media":[{"id":"69308e7ae62d6c8c27e929c3c4143abc9415e0fbb76cd9194884e8997c01d8c3","kind":"article","url":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-10-19T13:01:29.093Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."},{"id":"941d337a739c86c6040468c70e45e4bdfcc5e096f6a5fb7f865145bc49f3aa92","kind":"image","url":"https://substackcdn.com/image/fetch/$s_!Epxv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc311c042-341f-4cc2-8a87-de830f497cd9_2048x781.png","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","label":"View source image","provenance":"publisher_page","verification":{"status":"verified","checkedAt":"2026-10-19T14:20:41.808Z","method":"publisher_association_and_head","contentType":"image/png","cacheUntil":1792419941808},"recommendationReason":"Observed on the acquired publisher source; availability checked."}],"sourceMedia":{"status":"found","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","checkedAt":"2026-10-19T14:20:41.808Z","items":[{"type":"image","url":"https://substackcdn.com/image/fetch/$s_!Epxv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc311c042-341f-4cc2-8a87-de830f497cd9_2048x781.png","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","attribution":"Redwood Research blog","basis":"publisher-embed","availability":"verified","verifiedAt":"2026-10-19T14:20:41.808Z"}],"note":"Media declared by the original publisher; availability checked separately from the editorial reading."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"69308e7ae62d6c8c27e929c3c4143abc9415e0fbb76cd9194884e8997c01d8c3","confidence":0.95,"provider":"vercel","model":"typesafe-ai/jev","inputHash":"7366281303dc193e38007b24b6a1d9537733ca9fd57893aa5b3795b9cf82230c","decidedAt":"2026-10-19T13:01:31.827Z"},"provenance":{"textHash":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","observationId":"731ec8394c162cbeb1068e6be6a500b6035a187988ac7fa9d1be7953052f4712","retrievedAt":"2026-10-19T13:01:28.795Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"c1199954-7844-43de-8bf1-c33fe4a33123"},"reviewer":{"model":"gpt-6-luna","runId":"e6c7bf88-3751-4343-81fe-16660cdd6dc9"}},"audit":{"status":"passed","reviewedAt":"2026-10-19T13:02:24.396Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Gosha Kozhevnikov","url":"https://substack.com/@baraltiva","kind":"Person"},{"name":"Tyler","url":"https://substack.com/@tylord321","kind":"Person"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Gosha Kozhevnikov","url":"https://substack.com/@baraltiva","kind":"Person","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gosha Kozhevnikov","status":"stated","provenance":{"basis":"article-jsonld","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","locator":"document/html[1]/body[2]/div[1]/div[1]/div[1]/script[0]@37513.author"},"assertionId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf:participation:0","sourceVersion":{"snapshotId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","contentSha256":"dfcb8ddbc3cb87d090be3591e64cc98a7869690900662273fc2b42bd1353f60d","textHash":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","observedAt":"2026-10-19T14:20:41.808Z"}},{"actor":{"name":"Tyler","url":"https://substack.com/@tylord321","kind":"Person","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Tyler","status":"stated","provenance":{"basis":"article-jsonld","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","locator":"document/html[1]/body[2]/div[1]/div[1]/div[1]/script[0]@37513.author"},"assertionId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf:participation:1","sourceVersion":{"snapshotId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","contentSha256":"dfcb8ddbc3cb87d090be3591e64cc98a7869690900662273fc2b42bd1353f60d","textHash":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","observedAt":"2026-10-19T14:20:41.808Z"}},{"actor":{"name":"Gosha Kozhevnikov","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Gosha Kozhevnikov","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","locator":"head/meta[name=author|citation_author]"},"assertionId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf:participation:2","sourceVersion":{"snapshotId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","contentSha256":"dfcb8ddbc3cb87d090be3591e64cc98a7869690900662273fc2b42bd1353f60d","textHash":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","observedAt":"2026-10-19T14:20:41.808Z"}},{"actor":{"name":"Redwood Research blog","url":"https://blog.redwoodresearch.org/","kind":"unknown","actorId":null},"role":"publisher","scope":{"kind":"publication","locator":null},"literalCredit":"Redwood Research blog","status":"stated","provenance":{"basis":"article-publisher","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","locator":"document/html[1]/body[2]/div[1]/div[1]/div[1]/script[0]@37513.publisher"},"assertionId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf:participation:3","sourceVersion":{"snapshotId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf","sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","contentSha256":"dfcb8ddbc3cb87d090be3591e64cc98a7869690900662273fc2b42bd1353f60d","textHash":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","observedAt":"2026-10-19T14:20:41.808Z"}}],"distribution":{"service":"Substack","basis":"registered-source-adapter","publicationUrl":"https://blog.redwoodresearch.org"},"publisher":{"name":"Redwood Research blog","url":"https://blog.redwoodresearch.org/"},"sourceUrl":"https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript","snapshotId":"04e9d4405aa5941d3eeb4d1a434dff49b7db8cd6013eda9d7126f43af0f6d1bf","contentSha256":"dfcb8ddbc3cb87d090be3591e64cc98a7869690900662273fc2b42bd1353f60d","textHash":"fdb3f4375abc89bb5e359b1266dc3bcb0c05031454ee1d2ffa14942c7fe1968a","observedAt":"2026-10-19T14:20:41.808Z","basis":"publisher-page-metadata","sourceLinks":27,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/7f4ebc895c801d9abea2dca55cdd7af952980ffd944d20110d9f8d3147e86500/source"}}}],"audits":[{"publicationId":"1a85fff9-d6a9-404c-8ea3-70f797d944c6","verdict":"passed","findings":[{"dimension":"attribution","status":"pass","explanation":"The draft correctly attributes the post to Gosha Kozhevnikov and Tyler, published by Redwood Research. It frames the results and judgments as the authors' reports and views.","excerpt":"Gosha Kozhevnikov and Tyler\nOct 16, 2026"},{"dimension":"summary","status":"pass","explanation":"The summary accurately describes the pipeline and its components: the attacker, the verifier and the monitor. It explains action-only monitors in plain language. It states the results and the authors' main lessons, uncertainties and limitations. It also flags the 6-of-17 versus 6-of-35 denominator ambiguity. The title and question are grounded in the source, which says that quantifying safety information requires additional methodology.","excerpt":"quantifying it requires additional methodology that carefully accounts for approximations and uncertainties"},{"dimension":"date","status":"pass","explanation":"The originalPublished value of 2026-10-16 matches the post's byline date and the feed date. No earlier submission is indicated.","excerpt":"Gosha Kozhevnikov and Tyler\nOct 16, 2026"},{"dimension":"support","status":"pass","explanation":"Each claim is directly supported by the source text. This covers the 94 transcripts, the 35 evading attacks and the 6 confirmed attacks. It also covers the inability to produce numerical safety estimates, the authors' uncertainty about strengthening monitors, the lack of reasoning-trace support and the need for human review. No benefits or costs are implied beyond what the source states.","excerpt":"We ran the pipeline 252 times on 94 unique transcripts. This produced 35 attacks that evaded our monitors."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified link to the original Redwood Research post. It is the direct and most relevant source.","excerpt":null}],"createdAt":"2026-10-19T13:02:24.396Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}