{"id":"a6a1d49319068db18e75dea548c0465eead9318ee671ab9452ea192a7f739257","title":"arXiv cs.AI: What Do Collusion Probes Read? Three Pitfalls in Evaluating Multi-Agent Collusion Monitors","argument":"Authors Mainul Hasan Mohsin and Mohammad Shahidur Rahman argue that activation probes can appear to detect secret collusion while mostly detecting that an agent was instructed to collude, rather than whether it chose to. Their benchmark includes agents that received the instruction but refused it, helping separate instruction from decision; published detectors barely distinguished those refusers from colluders. The authors report that their best probe’s agent AUROC fell from 1.00 to 0.61 when the instruction was removed. They identify two more evaluation pitfalls: scoring after a vote lets a rule that reads only votes reach AUROC 0.990, despite finding nothing before the vote; and in 57% of attempts, no colluder argues publicly for the target. The authors also report that a probe trained on agents’ own choices reads collusion decisions without the instruction, and that probes detect colluders whose goal was fine-tuned into the weights. They propose a protocol using these controls to test collusion monitors and adapt scan statistics into an online monitor for groups of any size. The abstract reports benchmark results, but does not give enough methodological detail to assess how well they generalize beyond those episodes.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the paper’s abstract; it does not provide methodological detail sufficient to assess how the benchmark episodes were constructed or how well the results generalize beyond them.","speaker":{"name":"Mohsin, Mainul Hasan, Rahman, Mohammad Shahidur","roleAtSource":"Article author"},"source":{"title":"What Do Collusion Probes Read? Three Pitfalls in Evaluating Multi-Agent Collusion Monitors","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.10620","publishedAt":"2026-11-07","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that activation probes mostly detect an instruction to collude, rather than an agent’s decision to do so.","excerpt":"show that they mostly detect that an agent was told to collude, not that it decided to.","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"The benchmark includes agents instructed to collude who refuse, allowing the authors to distinguish instruction from decision.","excerpt":"adds agents who receive the collusion instruction and refuse it, separating instruction from decision.","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"Without the collusion instruction, the authors’ best probe’s agent AUROC falls from 1.00 to 0.61.","excerpt":"without it, agent AUROC falls from 1.00 to 0.61.","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"A rule that reads only votes reaches AUROC 0.990 when scored after the vote, but does not detect collusion before it.","excerpt":"scored after the vote, a rule that only reads the votes reaches AUROC 0.990 but finds nothing before it","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"In 57% of attempts, no colluder argues for the target in public.","excerpt":"in 57% of attempts no colluder argues for the target in public.","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"The authors report that a probe trained on agents’ own choices reads collusion decisions without the instruction.","excerpt":"a probe trained on agents' own choices reads it (AUROC 0.73 without the instruction)","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"The authors report that probes detect colluders whose goal was fine-tuned into the weights.","excerpt":"probes find colluders whose goal was fine-tuned into the weights (AUROC 0.995; pre-registered)","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"The authors propose a protocol for testing collusion monitors and an online monitor adapted from scan statistics.","excerpt":"We turn the controls into a protocol for testing any collusion monitor, and adapt scan statistics into an online monitor for groups of any size.","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10620","checkedAt":"2026-11-12T11:50:21.070Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do the reported probe results and proposed controls hold beyond the benchmark episodes described in the abstract?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: What Do Collusion Probes Read? Three Pitfalls in Evaluating Multi-Agent Collusion Monitors","summary":"Authors Mainul Hasan Mohsin and Mohammad Shahidur Rahman argue that activation probes can appear to detect secret collusion while mostly detecting that an agent was instructed to collude, rather than whether it chose to. Their benchmark includes agents that received the instruction but refused it, helping separate instruction from decision; published detectors barely distinguished those refusers from colluders. The authors report that their best probe’s agent AUROC fell from 1.00 to 0.61 when the instruction was removed. They identify two more evaluation pitfalls: scoring after a vote lets a rule that reads only votes reach AUROC 0.990, despite finding nothing before the vote; and in 57% of attempts, no colluder argues publicly for the target. The authors also report that a probe trained on agents’ own choices reads collusion decisions without the instruction, and that probes detect colluders whose goal was fine-tuned into the weights. They propose a protocol using these controls to test collusion monitors and adapt scan statistics into an online monitor for groups of any size. The abstract reports benchmark results, but does not give enough methodological detail to assess how well they generalize beyond those episodes.","question":"How well do the reported probe results and proposed controls hold beyond the benchmark episodes described in the abstract?","claims":[{"text":"The authors argue that activation probes mostly detect an instruction to collude, rather than an agent’s decision to do so.","excerpt":"show that they mostly detect that an agent was told to collude, not that it decided to.","support":"source-stated"},{"text":"The benchmark includes agents instructed to collude who refuse, allowing the authors to distinguish instruction from decision.","excerpt":"adds agents who receive the collusion instruction and refuse it, separating instruction from decision.","support":"source-stated"},{"text":"Without the collusion instruction, the authors’ best probe’s agent AUROC falls from 1.00 to 0.61.","excerpt":"without it, agent AUROC falls from 1.00 to 0.61.","support":"source-stated"},{"text":"A rule that reads only votes reaches AUROC 0.990 when scored after the vote, but does not detect collusion before it.","excerpt":"scored after the vote, a rule that only reads the votes reaches AUROC 0.990 but finds nothing before it","support":"source-stated"},{"text":"In 57% of attempts, no colluder argues for the target in public.","excerpt":"in 57% of attempts no colluder argues for the target in public.","support":"source-stated"},{"text":"The authors report that a probe trained on agents’ own choices reads collusion decisions without the instruction.","excerpt":"a probe trained on agents' own choices reads it (AUROC 0.73 without the instruction)","support":"source-stated"},{"text":"The authors report that probes detect colluders whose goal was fine-tuned into the weights.","excerpt":"probes find colluders whose goal was fine-tuned into the weights (AUROC 0.995; pre-registered)","support":"source-stated"},{"text":"The authors propose a protocol for testing collusion monitors and an online monitor adapted from scan statistics.","excerpt":"We turn the controls into a protocol for testing any collusion monitor, and adapt scan statistics into an online monitor for groups of any size.","support":"source-stated"}],"limitations":["The evidence is the paper’s abstract; it does not provide methodological detail sufficient to assess how the benchmark episodes were constructed or how well the results generalize beyond them."],"mediaId":"7f5f5b4485fd1754f0a5ca3c3f7e20b68aa765e374a2e2dd6fc9c086c50b926d","correctionReason":null,"id":"a6a1d49319068db18e75dea548c0465eead9318ee671ab9452ea192a7f739257","publicationId":"8297d8ec-55a1-480e-82c0-b1a746e3e925","version":1,"sourceUrl":"https://arxiv.org/abs/2611.10620","sourceTitle":"What Do Collusion Probes Read? Three Pitfalls in Evaluating Multi-Agent Collusion Monitors","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-07","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:39:53.173Z","createdAt":"2026-11-12T11:18:26.904Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"7f5f5b4485fd1754f0a5ca3c3f7e20b68aa765e374a2e2dd6fc9c086c50b926d","kind":"article","url":"https://arxiv.org/abs/2611.10620","sourceUrl":"https://arxiv.org/abs/2611.10620","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T11:04:02.198Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10620","checkedAt":"2026-11-12T11:50:21.070Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"7f5f5b4485fd1754f0a5ca3c3f7e20b68aa765e374a2e2dd6fc9c086c50b926d","confidence":0.59,"provider":"typesafe","model":"jev-1.13.0","inputHash":"1c57ff3c9d54cea667a86fc9bacab82c2323adb34a64bfc8121a2f45616ed857","decidedAt":"2026-11-12T11:04:03.009Z"},"provenance":{"textHash":"ebf2530f7cc41b0904cbe08b4fb7003d29ba807cb65dbd357ced5c7be8771f82","observationId":"c1728229a87e90aa1977e313a7d8c058fcf7bc004c7c4ff2dc5f5746527179d8","retrievedAt":"2026-11-12T11:04:02.085Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"0a9f5995-33ff-49e8-8c22-e87db006d32b"},"reviewer":{"model":"gpt-6-luna","runId":"d57bacfb-622d-4511-8485-e37ff0a26635"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T11:18:26.904Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Mohsin, Mainul Hasan","url":null,"kind":"unknown"},{"name":"Rahman, Mohammad Shahidur","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Mohsin, Mainul Hasan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mohsin, Mainul Hasan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10620","locator":"head/meta[name=author|citation_author]"},"assertionId":"d3965b88a700b33722bc55cd51a20fc8cff7af4691b20d8cd9da843743b94757:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:50:21.070Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"d3965b88a700b33722bc55cd51a20fc8cff7af4691b20d8cd9da843743b94757","sourceUrl":"https://arxiv.org/abs/2611.10620","contentSha256":"582911045cdbd1a4e546fd48dfb8a1d2a3f8102f2ed75ab212e105a5dbc995df","textHash":"ebf2530f7cc41b0904cbe08b4fb7003d29ba807cb65dbd357ced5c7be8771f82","observedAt":"2026-11-12T11:50:21.070Z"}},{"actor":{"name":"Rahman, Mohammad Shahidur","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Rahman, Mohammad Shahidur","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10620","locator":"head/meta[name=author|citation_author]"},"assertionId":"d3965b88a700b33722bc55cd51a20fc8cff7af4691b20d8cd9da843743b94757:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:50:21.070Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"d3965b88a700b33722bc55cd51a20fc8cff7af4691b20d8cd9da843743b94757","sourceUrl":"https://arxiv.org/abs/2611.10620","contentSha256":"582911045cdbd1a4e546fd48dfb8a1d2a3f8102f2ed75ab212e105a5dbc995df","textHash":"ebf2530f7cc41b0904cbe08b4fb7003d29ba807cb65dbd357ced5c7be8771f82","observedAt":"2026-11-12T11:50:21.070Z"}}],"participationCoverage":{"total":2,"returned":2,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.10620","snapshotId":"d3965b88a700b33722bc55cd51a20fc8cff7af4691b20d8cd9da843743b94757","contentSha256":"582911045cdbd1a4e546fd48dfb8a1d2a3f8102f2ed75ab212e105a5dbc995df","textHash":"ebf2530f7cc41b0904cbe08b4fb7003d29ba807cb65dbd357ced5c7be8771f82","observedAt":"2026-11-12T11:50:21.070Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/a6a1d49319068db18e75dea548c0465eead9318ee671ab9452ea192a7f739257/source"}},"canonicalUrl":"https://escapemeridian.com/readings/arxiv-cs-ai-what-do-collusion-probes-read-three-pitfalls-in-a6a1d493","formats":{"html":"/readings/arxiv-cs-ai-what-do-collusion-probes-read-three-pitfalls-in-a6a1d493","markdown":"/readings/arxiv-cs-ai-what-do-collusion-probes-read-three-pitfalls-in-a6a1d493.md","json":"/readings/arxiv-cs-ai-what-do-collusion-probes-read-three-pitfalls-in-a6a1d493.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-benchmark","name":"Benchmark","explanation":"A defined set of tests used to compare systems. Results depend on the tasks and measures selected.","referenceUrl":"https://arxiv.org/abs/2211.09110"},{"id":"term-fine-tuning","name":"Fine-tuning","explanation":"Further training of an existing model to adapt it to a particular task or use.","referenceUrl":"https://developers.google.com/machine-learning/glossary#fine-tuning"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"a6a1d49319068db18e75dea548c0465eead9318ee671ab9452ea192a7f739257","revision":"8297d8ec-55a1-480e-82c0-b1a746e3e925","revisionKind":"publication-id","title":"arXiv cs.AI: What Do Collusion Probes Read? Three Pitfalls in Evaluating Multi-Agent Collusion Monitors","url":"https://escapemeridian.com/readings/arxiv-cs-ai-what-do-collusion-probes-read-three-pitfalls-in-a6a1d493"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.10620","revision":"ebf2530f7cc41b0904cbe08b4fb7003d29ba807cb65dbd357ced5c7be8771f82","revisionKind":"sha256-source-text","title":"What Do Collusion Probes Read? Three Pitfalls in Evaluating Multi-Agent Collusion Monitors","url":"https://arxiv.org/abs/2611.10620"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"a6a1d49319068db18e75dea548c0465eead9318ee671ab9452ea192a7f739257","title":"arXiv cs.AI: What Do Collusion Probes Read? Three Pitfalls in Evaluating Multi-Agent Collusion Monitors","argument":"Authors Mainul Hasan Mohsin and Mohammad Shahidur Rahman argue that activation probes can appear to detect secret collusion while mostly detecting that an agent was instructed to collude, rather than whether it chose to. Their benchmark includes agents that received the instruction but refused it, helping separate instruction from decision; published detectors barely distinguished those refusers from colluders. The authors report that their best probe’s agent AUROC fell from 1.00 to 0.61 when the instruction was removed. They identify two more evaluation pitfalls: scoring after a vote lets a rule that reads only votes reach AUROC 0.990, despite finding nothing before the vote; and in 57% of attempts, no colluder argues publicly for the target. The authors also report that a probe trained on agents’ own choices reads collusion decisions without the instruction, and that probes detect colluders whose goal was fine-tuned into the weights. They propose a protocol using these controls to test collusion monitors and adapt scan statistics into an online monitor for groups of any size. The abstract reports benchmark results, but does not give enough methodological detail to assess how well they generalize beyond those episodes.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The evidence is the paper’s abstract; it does not provide methodological detail sufficient to assess how the benchmark episodes were constructed or how well the results generalize beyond them.","speaker":{"name":"Mohsin, Mainul Hasan, Rahman, Mohammad Shahidur","roleAtSource":"Article author"},"source":{"title":"What Do Collusion Probes Read? Three Pitfalls in Evaluating Multi-Agent Collusion Monitors","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.10620","publishedAt":"2026-11-07","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors argue that activation probes mostly detect an instruction to collude, rather than an agent’s decision to do so.","excerpt":"show that they mostly detect that an agent was told to collude, not that it decided to.","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"The benchmark includes agents instructed to collude who refuse, allowing the authors to distinguish instruction from decision.","excerpt":"adds agents who receive the collusion instruction and refuse it, separating instruction from decision.","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"Without the collusion instruction, the authors’ best probe’s agent AUROC falls from 1.00 to 0.61.","excerpt":"without it, agent AUROC falls from 1.00 to 0.61.","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"A rule that reads only votes reaches AUROC 0.990 when scored after the vote, but does not detect collusion before it.","excerpt":"scored after the vote, a rule that only reads the votes reaches AUROC 0.990 but finds nothing before it","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"In 57% of attempts, no colluder argues for the target in public.","excerpt":"in 57% of attempts no colluder argues for the target in public.","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"The authors report that a probe trained on agents’ own choices reads collusion decisions without the instruction.","excerpt":"a probe trained on agents' own choices reads it (AUROC 0.73 without the instruction)","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"The authors report that probes detect colluders whose goal was fine-tuned into the weights.","excerpt":"probes find colluders whose goal was fine-tuned into the weights (AUROC 0.995; pre-registered)","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"},{"text":"The authors propose a protocol for testing collusion monitors and an online monitor adapted from scan statistics.","excerpt":"We turn the controls into a protocol for testing any collusion monitor, and adapt scan statistics into an online monitor for groups of any size.","url":"https://arxiv.org/abs/2611.10620","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10620","checkedAt":"2026-11-12T11:50:21.070Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How well do the reported probe results and proposed controls hold beyond the benchmark episodes described in the abstract?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"arXiv cs.AI: What Do Collusion Probes Read? Three Pitfalls in Evaluating Multi-Agent Collusion Monitors","summary":"Authors Mainul Hasan Mohsin and Mohammad Shahidur Rahman argue that activation probes can appear to detect secret collusion while mostly detecting that an agent was instructed to collude, rather than whether it chose to. Their benchmark includes agents that received the instruction but refused it, helping separate instruction from decision; published detectors barely distinguished those refusers from colluders. The authors report that their best probe’s agent AUROC fell from 1.00 to 0.61 when the instruction was removed. They identify two more evaluation pitfalls: scoring after a vote lets a rule that reads only votes reach AUROC 0.990, despite finding nothing before the vote; and in 57% of attempts, no colluder argues publicly for the target. The authors also report that a probe trained on agents’ own choices reads collusion decisions without the instruction, and that probes detect colluders whose goal was fine-tuned into the weights. They propose a protocol using these controls to test collusion monitors and adapt scan statistics into an online monitor for groups of any size. The abstract reports benchmark results, but does not give enough methodological detail to assess how well they generalize beyond those episodes.","question":"How well do the reported probe results and proposed controls hold beyond the benchmark episodes described in the abstract?","claims":[{"text":"The authors argue that activation probes mostly detect an instruction to collude, rather than an agent’s decision to do so.","excerpt":"show that they mostly detect that an agent was told to collude, not that it decided to.","support":"source-stated"},{"text":"The benchmark includes agents instructed to collude who refuse, allowing the authors to distinguish instruction from decision.","excerpt":"adds agents who receive the collusion instruction and refuse it, separating instruction from decision.","support":"source-stated"},{"text":"Without the collusion instruction, the authors’ best probe’s agent AUROC falls from 1.00 to 0.61.","excerpt":"without it, agent AUROC falls from 1.00 to 0.61.","support":"source-stated"},{"text":"A rule that reads only votes reaches AUROC 0.990 when scored after the vote, but does not detect collusion before it.","excerpt":"scored after the vote, a rule that only reads the votes reaches AUROC 0.990 but finds nothing before it","support":"source-stated"},{"text":"In 57% of attempts, no colluder argues for the target in public.","excerpt":"in 57% of attempts no colluder argues for the target in public.","support":"source-stated"},{"text":"The authors report that a probe trained on agents’ own choices reads collusion decisions without the instruction.","excerpt":"a probe trained on agents' own choices reads it (AUROC 0.73 without the instruction)","support":"source-stated"},{"text":"The authors report that probes detect colluders whose goal was fine-tuned into the weights.","excerpt":"probes find colluders whose goal was fine-tuned into the weights (AUROC 0.995; pre-registered)","support":"source-stated"},{"text":"The authors propose a protocol for testing collusion monitors and an online monitor adapted from scan statistics.","excerpt":"We turn the controls into a protocol for testing any collusion monitor, and adapt scan statistics into an online monitor for groups of any size.","support":"source-stated"}],"limitations":["The evidence is the paper’s abstract; it does not provide methodological detail sufficient to assess how the benchmark episodes were constructed or how well the results generalize beyond them."],"mediaId":"7f5f5b4485fd1754f0a5ca3c3f7e20b68aa765e374a2e2dd6fc9c086c50b926d","correctionReason":null,"id":"a6a1d49319068db18e75dea548c0465eead9318ee671ab9452ea192a7f739257","publicationId":"8297d8ec-55a1-480e-82c0-b1a746e3e925","version":1,"sourceUrl":"https://arxiv.org/abs/2611.10620","sourceTitle":"What Do Collusion Probes Read? Three Pitfalls in Evaluating Multi-Agent Collusion Monitors","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-07","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-10T08:39:53.173Z","createdAt":"2026-11-12T11:18:26.904Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"7f5f5b4485fd1754f0a5ca3c3f7e20b68aa765e374a2e2dd6fc9c086c50b926d","kind":"article","url":"https://arxiv.org/abs/2611.10620","sourceUrl":"https://arxiv.org/abs/2611.10620","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-12T11:04:02.198Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.10620","checkedAt":"2026-11-12T11:50:21.070Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v6","mediaId":"7f5f5b4485fd1754f0a5ca3c3f7e20b68aa765e374a2e2dd6fc9c086c50b926d","confidence":0.59,"provider":"typesafe","model":"jev-1.13.0","inputHash":"1c57ff3c9d54cea667a86fc9bacab82c2323adb34a64bfc8121a2f45616ed857","decidedAt":"2026-11-12T11:04:03.009Z"},"provenance":{"textHash":"ebf2530f7cc41b0904cbe08b4fb7003d29ba807cb65dbd357ced5c7be8771f82","observationId":"c1728229a87e90aa1977e313a7d8c058fcf7bc004c7c4ff2dc5f5746527179d8","retrievedAt":"2026-11-12T11:04:02.085Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"0a9f5995-33ff-49e8-8c22-e87db006d32b"},"reviewer":{"model":"gpt-6-luna","runId":"d57bacfb-622d-4511-8485-e37ff0a26635"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-12T11:18:26.904Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Mohsin, Mainul Hasan","url":null,"kind":"unknown"},{"name":"Rahman, Mohammad Shahidur","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Mohsin, Mainul Hasan","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Mohsin, Mainul Hasan","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10620","locator":"head/meta[name=author|citation_author]"},"assertionId":"d3965b88a700b33722bc55cd51a20fc8cff7af4691b20d8cd9da843743b94757:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:50:21.070Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"d3965b88a700b33722bc55cd51a20fc8cff7af4691b20d8cd9da843743b94757","sourceUrl":"https://arxiv.org/abs/2611.10620","contentSha256":"582911045cdbd1a4e546fd48dfb8a1d2a3f8102f2ed75ab212e105a5dbc995df","textHash":"ebf2530f7cc41b0904cbe08b4fb7003d29ba807cb65dbd357ced5c7be8771f82","observedAt":"2026-11-12T11:50:21.070Z"}},{"actor":{"name":"Rahman, Mohammad Shahidur","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Rahman, Mohammad Shahidur","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.10620","locator":"head/meta[name=author|citation_author]"},"assertionId":"d3965b88a700b33722bc55cd51a20fc8cff7af4691b20d8cd9da843743b94757:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-12T11:50:21.070Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"d3965b88a700b33722bc55cd51a20fc8cff7af4691b20d8cd9da843743b94757","sourceUrl":"https://arxiv.org/abs/2611.10620","contentSha256":"582911045cdbd1a4e546fd48dfb8a1d2a3f8102f2ed75ab212e105a5dbc995df","textHash":"ebf2530f7cc41b0904cbe08b4fb7003d29ba807cb65dbd357ced5c7be8771f82","observedAt":"2026-11-12T11:50:21.070Z"}}],"participationCoverage":{"total":2,"returned":2,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.10620","snapshotId":"d3965b88a700b33722bc55cd51a20fc8cff7af4691b20d8cd9da843743b94757","contentSha256":"582911045cdbd1a4e546fd48dfb8a1d2a3f8102f2ed75ab212e105a5dbc995df","textHash":"ebf2530f7cc41b0904cbe08b4fb7003d29ba807cb65dbd357ced5c7be8771f82","observedAt":"2026-11-12T11:50:21.070Z","basis":"publisher-page-metadata","sourceLinks":44,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/a6a1d49319068db18e75dea548c0465eead9318ee671ab9452ea192a7f739257/source"}}}],"audits":[{"publicationId":"8297d8ec-55a1-480e-82c0-b1a746e3e925","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The reading correctly attributes the paper and its findings to authors Mainul Hasan Mohsin and Mohammad Shahidur Rahman. It keeps the results framed as the authors' reports rather than established fact.","excerpt":"Authors:Mainul Hasan Mohsin, Mohammad Shahidur Rahman"},{"dimension":"summary","status":"uncertain","explanation":"The reading follows the abstract faithfully, but a general reader cannot interpret it as written. Central terms are left unexplained: activation probes (tools that read a model's internal signals), AUROC (a 0.5–1.0 measure of how well a detector separates two groups), fine-tuned into the weights, and scan statistics. Without them, the 1.00-to-0.61 drop is opaque. The reading also omits the AUROC 0.73 figure for the choice-trained probe. Without it, 'reads collusion decisions' sounds stronger than the moderate result reported. Smallest repairs: add the 0.73 figure, gloss AUROC and activation probes, and say briefly why inflated monitor scores matter for overseeing groups of AI agents.","excerpt":"a probe trained on agents' own choices reads it (AUROC 0.73 without the instruction)"},{"dimension":"date","status":"pass","explanation":"The originalPublished date of 2026-11-07 matches the v1 submission date shown on the arXiv page.","excerpt":"[Submitted on 7 Nov 2026]"},{"dimension":"support","status":"pass","explanation":"Each claim is backed by a verbatim abstract excerpt. The limitation about generalization is labeled as an evidence gap, not asserted as a finding. The question follows from the evidence. No unsupported benefits or costs are implied.","excerpt":"Published detectors barely tell colluders from these refusers"},{"dimension":"media","status":"pass","explanation":"The selected media is the verified arXiv abstract page itself, so it gives direct access to the original source.","excerpt":null}],"createdAt":"2026-11-12T11:18:26.904Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}