{"id":"e04cb86cebbe2585a8ce32927ebd149fb06d43126eb3ab6770097b8bdd786bba","title":"What the Middleman Delivers: How LLM API relays compare","argument":"Authors Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li and Jianbing Ni examine third-party API relays, services that provide access to large language models. They ask whether relays deliver expected model behavior, respect tool permissions and bill accurately—not just whether their advertised models and prices look attractive.\n\nThe authors compare answers from 20 relays offering GPT-6 Astra and Claude Opus 5 with batch-matched official responses. In the first-round sample, their analysis flags behavioral differences at 7/10 low-price relays and 2/10 same- or higher-price relays. Retesting changes some flags, so a single comparison may not give a stable picture.\n\nThey report no confirmed injection in 3,332 screened responses to a fixed, non-adversarial code and setup workload. But four relays return tool calls despite instructions disabling them. For identical requests, five relays report 4,200 to 60,000 more input tokens than the official API. Where balances are available, charges follow reported usage at listed prices, including extra reported input tokens at one service.\n\nThe authors argue that advertised models and prices are insufficient to judge relay reliability. They identify unexplained billed token quantities and tool-permission violations as risks, and say service evaluations need repeated measurements.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported tests cover 20 relays and a fixed, non-adversarial code and setup workload for the injection screening; the abstract does not establish how results generalize beyond the tested services and conditions. The abstract reports retest changes in behavioral flags, so the initial comparison does not alone establish that those differences persist across measurements.","speaker":{"name":"Wu, Xiaodong, Zhao, Zhimin, Li, Qi, Cheong, Chaklam, Li, Xiangman, Ni, Jianbing","roleAtSource":"Article author"},"source":{"title":"What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.07650","publishedAt":"2026-11-05","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors compared 20 relays offering GPT-6 Astra and Claude Opus 5 against batch-matched official responses.","excerpt":"We test 20 relays offering GPT-6 Astra and Claude Opus 5. First, we compare answers to knowledge questions with batch-matched official responses.","url":"https://arxiv.org/abs/2611.07650","relationship":"Source-stated"},{"text":"In the first-round sample, the analysis flagged behavioral differences at 7/10 low-price relays and 2/10 same- or higher-price relays; retesting changed some flags.","excerpt":"In our first-round sample, the analysis flags 7/10 low-price and 2/10 same- or higher-price relays for behavioral differences. On retesting 35 pairs, seven gain or lose a flag under the original rules","url":"https://arxiv.org/abs/2611.07650","relationship":"Source-stated"},{"text":"The authors report no confirmed injection in the screened responses, but four relays returned tool calls despite instructions disabling them.","excerpt":"No injection is confirmed in 3,332 screened responses to a fixed, non-adversarial code and setup workload, yet four relays return tool calls despite instructions disabling them in the first round.","url":"https://arxiv.org/abs/2611.07650","relationship":"Source-stated"},{"text":"For identical requests, five relays reported substantially more input tokens than the official API; where balances were available, charges followed reported usage at listed prices.","excerpt":"For identical requests, five relays report 4,200 to 60,000 more input tokens than the official API. Where balances are available (six services), charges follow reported usage at listed prices","url":"https://arxiv.org/abs/2611.07650","relationship":"Source-stated"},{"text":"The authors argue that advertised models and prices alone are insufficient to judge relay reliability.","excerpt":"Our study shows that advertised models and prices are insufficient to judge relay reliability.","url":"https://arxiv.org/abs/2611.07650","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07650","checkedAt":"2026-11-08T04:22:07.156Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How consistently do the reported behavioral differences appear across repeated measurements?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"What the Middleman Delivers: How LLM API relays compare","summary":"Authors Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li and Jianbing Ni examine third-party API relays, services that provide access to large language models. They ask whether relays deliver expected model behavior, respect tool permissions and bill accurately—not just whether their advertised models and prices look attractive.\n\nThe authors compare answers from 20 relays offering GPT-6 Astra and Claude Opus 5 with batch-matched official responses. In the first-round sample, their analysis flags behavioral differences at 7/10 low-price relays and 2/10 same- or higher-price relays. Retesting changes some flags, so a single comparison may not give a stable picture.\n\nThey report no confirmed injection in 3,332 screened responses to a fixed, non-adversarial code and setup workload. But four relays return tool calls despite instructions disabling them. For identical requests, five relays report 4,200 to 60,000 more input tokens than the official API. Where balances are available, charges follow reported usage at listed prices, including extra reported input tokens at one service.\n\nThe authors argue that advertised models and prices are insufficient to judge relay reliability. They identify unexplained billed token quantities and tool-permission violations as risks, and say service evaluations need repeated measurements.","question":"How consistently do the reported behavioral differences appear across repeated measurements?","claims":[{"text":"The authors compared 20 relays offering GPT-6 Astra and Claude Opus 5 against batch-matched official responses.","excerpt":"We test 20 relays offering GPT-6 Astra and Claude Opus 5. First, we compare answers to knowledge questions with batch-matched official responses.","support":"source-stated"},{"text":"In the first-round sample, the analysis flagged behavioral differences at 7/10 low-price relays and 2/10 same- or higher-price relays; retesting changed some flags.","excerpt":"In our first-round sample, the analysis flags 7/10 low-price and 2/10 same- or higher-price relays for behavioral differences. On retesting 35 pairs, seven gain or lose a flag under the original rules","support":"source-stated"},{"text":"The authors report no confirmed injection in the screened responses, but four relays returned tool calls despite instructions disabling them.","excerpt":"No injection is confirmed in 3,332 screened responses to a fixed, non-adversarial code and setup workload, yet four relays return tool calls despite instructions disabling them in the first round.","support":"source-stated"},{"text":"For identical requests, five relays reported substantially more input tokens than the official API; where balances were available, charges followed reported usage at listed prices.","excerpt":"For identical requests, five relays report 4,200 to 60,000 more input tokens than the official API. Where balances are available (six services), charges follow reported usage at listed prices","support":"source-stated"},{"text":"The authors argue that advertised models and prices alone are insufficient to judge relay reliability.","excerpt":"Our study shows that advertised models and prices are insufficient to judge relay reliability.","support":"source-stated"}],"limitations":["The reported tests cover 20 relays and a fixed, non-adversarial code and setup workload for the injection screening; the abstract does not establish how results generalize beyond the tested services and conditions.","The abstract reports retest changes in behavioral flags, so the initial comparison does not alone establish that those differences persist across measurements."],"mediaId":"343b5bd40bb89519fbe2aa8d7318e176913cc73c0907e6df851f129cbc2dd61f","correctionReason":null,"id":"e04cb86cebbe2585a8ce32927ebd149fb06d43126eb3ab6770097b8bdd786bba","publicationId":"a3ad766d-5d81-4fef-8040-3e7af776f2ef","version":1,"sourceUrl":"https://arxiv.org/abs/2611.07650","sourceTitle":"What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-05","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:10:45.324Z","createdAt":"2026-11-08T03:18:44.244Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"343b5bd40bb89519fbe2aa8d7318e176913cc73c0907e6df851f129cbc2dd61f","kind":"article","url":"https://arxiv.org/abs/2611.07650","sourceUrl":"https://arxiv.org/abs/2611.07650","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T03:17:37.037Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07650","checkedAt":"2026-11-08T04:22:07.156Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"343b5bd40bb89519fbe2aa8d7318e176913cc73c0907e6df851f129cbc2dd61f","confidence":0.85,"provider":"typesafe","model":"jev-1.13.0","inputHash":"2a313bb95a43e98f3008f1f3d9351267abe85174a5824c860432ecba1636da24","decidedAt":"2026-11-08T03:17:37.904Z"},"provenance":{"textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observationId":"b6d105ef882e58d32525648583bb35522574dff2c57b5385287094095275174e","retrievedAt":"2026-11-08T03:17:36.884Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"95c4666e-d59a-4ddc-8e1e-1d2914c79886"},"reviewer":{"model":"gpt-6-luna","runId":"d256c0c6-4872-4a55-8067-2ebb0eeb611e"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-08T03:18:44.244Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Wu, Xiaodong","url":null,"kind":"unknown"},{"name":"Zhao, Zhimin","url":null,"kind":"unknown"},{"name":"Li, Qi","url":null,"kind":"unknown"},{"name":"Cheong, Chaklam","url":null,"kind":"unknown"},{"name":"Li, Xiangman","url":null,"kind":"unknown"},{"name":"Ni, Jianbing","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Wu, Xiaodong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Xiaodong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07650","locator":"head/meta[name=author|citation_author]"},"assertionId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:22:07.156Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","sourceUrl":"https://arxiv.org/abs/2611.07650","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z"}},{"actor":{"name":"Zhao, Zhimin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhao, Zhimin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07650","locator":"head/meta[name=author|citation_author]"},"assertionId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:22:07.156Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","sourceUrl":"https://arxiv.org/abs/2611.07650","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z"}},{"actor":{"name":"Li, Qi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Qi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07650","locator":"head/meta[name=author|citation_author]"},"assertionId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:22:07.156Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","sourceUrl":"https://arxiv.org/abs/2611.07650","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z"}},{"actor":{"name":"Cheong, Chaklam","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Cheong, Chaklam","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07650","locator":"head/meta[name=author|citation_author]"},"assertionId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:22:07.156Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","sourceUrl":"https://arxiv.org/abs/2611.07650","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z"}},{"actor":{"name":"Li, Xiangman","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Xiangman","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07650","locator":"head/meta[name=author|citation_author]"},"assertionId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:22:07.156Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","sourceUrl":"https://arxiv.org/abs/2611.07650","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z"}},{"actor":{"name":"Ni, Jianbing","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ni, Jianbing","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07650","locator":"head/meta[name=author|citation_author]"},"assertionId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:22:07.156Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","sourceUrl":"https://arxiv.org/abs/2611.07650","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.07650","snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z","basis":"publisher-page-metadata","sourceLinks":66,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/e04cb86cebbe2585a8ce32927ebd149fb06d43126eb3ab6770097b8bdd786bba/source"}},"canonicalUrl":"https://escapemeridian.com/readings/what-the-middleman-delivers-how-llm-api-relays-compare-e04cb86c","formats":{"html":"/readings/what-the-middleman-delivers-how-llm-api-relays-compare-e04cb86c","markdown":"/readings/what-the-middleman-delivers-how-llm-api-relays-compare-e04cb86c.md","json":"/readings/what-the-middleman-delivers-how-llm-api-relays-compare-e04cb86c.json"},"attribution":"Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.","maintainedGuideLinks":[],"readerContext":{"version":"reader-context-v1","attribution":"Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.","terms":[{"id":"term-llm","name":"Large language model","explanation":"A language model with many learned numerical settings, used to process and generate language.","referenceUrl":"https://developers.google.com/machine-learning/glossary#large-language-model"},{"id":"term-api","name":"API","explanation":"An application programming interface: a defined way for software to request data or actions from another system.","referenceUrl":"https://developer.mozilla.org/en-US/docs/Glossary/API"},{"id":"term-tokens","name":"Model tokens","explanation":"Small chunks of text, often parts of words, that a language model reads and writes; usage and prices are often counted in tokens.","referenceUrl":"https://developers.google.com/machine-learning/glossary#token"}],"entities":[{"id":"arxiv","name":"arXiv","kind":"organization","context":"arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.","referenceUrls":["https://arxiv.org/","https://info.arxiv.org/help/api/index.html"],"path":"/entities/arxiv","json":"/entities/arxiv.json","markdown":"/entities/arxiv.md"},{"id":"claude","name":"Claude","kind":"model","context":"Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.","referenceUrls":["https://claude.com/product/overview","https://www.anthropic.com/news","https://www.anthropic.com/research/team/economics"],"path":"/entities/claude","json":"/entities/claude.json","markdown":"/entities/claude.md"}]},"researchQuestions":[],"withdrawn":false,"references":{"schemaVersion":1,"target":{"type":"reading","id":"e04cb86cebbe2585a8ce32927ebd149fb06d43126eb3ab6770097b8bdd786bba","revision":"a3ad766d-5d81-4fef-8040-3e7af776f2ef","revisionKind":"publication-id","title":"What the Middleman Delivers: How LLM API relays compare","url":"https://escapemeridian.com/readings/what-the-middleman-delivers-how-llm-api-relays-compare-e04cb86c"},"source":{"type":"source","id":"https://arxiv.org/abs/2611.07650","revision":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","revisionKind":"sha256-source-text","title":"What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays","url":"https://arxiv.org/abs/2611.07650"},"status":"available","items":[],"next":null,"scope":"Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter."},"revisions":[{"id":"e04cb86cebbe2585a8ce32927ebd149fb06d43126eb3ab6770097b8bdd786bba","title":"What the Middleman Delivers: How LLM API relays compare","argument":"Authors Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li and Jianbing Ni examine third-party API relays, services that provide access to large language models. They ask whether relays deliver expected model behavior, respect tool permissions and bill accurately—not just whether their advertised models and prices look attractive.\n\nThe authors compare answers from 20 relays offering GPT-6 Astra and Claude Opus 5 with batch-matched official responses. In the first-round sample, their analysis flags behavioral differences at 7/10 low-price relays and 2/10 same- or higher-price relays. Retesting changes some flags, so a single comparison may not give a stable picture.\n\nThey report no confirmed injection in 3,332 screened responses to a fixed, non-adversarial code and setup workload. But four relays return tool calls despite instructions disabling them. For identical requests, five relays report 4,200 to 60,000 more input tokens than the official API. Where balances are available, charges follow reported usage at listed prices, including extra reported input tokens at one service.\n\nThe authors argue that advertised models and prices are insufficient to judge relay reliability. They identify unexplained billed token quantities and tool-permission violations as risks, and say service evaluations need repeated measurements.","topicIds":["intelligence-and-discovery"],"domain":"science","claimType":"attributed reading","limits":"The reported tests cover 20 relays and a fixed, non-adversarial code and setup workload for the injection screening; the abstract does not establish how results generalize beyond the tested services and conditions. The abstract reports retest changes in behavioral flags, so the initial comparison does not alone establish that those differences persist across measurements.","speaker":{"name":"Wu, Xiaodong, Zhao, Zhimin, Li, Qi, Cheong, Chaklam, Li, Xiangman, Ni, Jianbing","roleAtSource":"Article author"},"source":{"title":"What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays","publisher":"arXiv cs.AI","format":"Article","url":"https://arxiv.org/abs/2611.07650","publishedAt":"2026-11-05","dateBasis":"Date basis not provided."},"evidence":[{"text":"The authors compared 20 relays offering GPT-6 Astra and Claude Opus 5 against batch-matched official responses.","excerpt":"We test 20 relays offering GPT-6 Astra and Claude Opus 5. First, we compare answers to knowledge questions with batch-matched official responses.","url":"https://arxiv.org/abs/2611.07650","relationship":"Source-stated"},{"text":"In the first-round sample, the analysis flagged behavioral differences at 7/10 low-price relays and 2/10 same- or higher-price relays; retesting changed some flags.","excerpt":"In our first-round sample, the analysis flags 7/10 low-price and 2/10 same- or higher-price relays for behavioral differences. On retesting 35 pairs, seven gain or lose a flag under the original rules","url":"https://arxiv.org/abs/2611.07650","relationship":"Source-stated"},{"text":"The authors report no confirmed injection in the screened responses, but four relays returned tool calls despite instructions disabling them.","excerpt":"No injection is confirmed in 3,332 screened responses to a fixed, non-adversarial code and setup workload, yet four relays return tool calls despite instructions disabling them in the first round.","url":"https://arxiv.org/abs/2611.07650","relationship":"Source-stated"},{"text":"For identical requests, five relays reported substantially more input tokens than the official API; where balances were available, charges followed reported usage at listed prices.","excerpt":"For identical requests, five relays report 4,200 to 60,000 more input tokens than the official API. Where balances are available (six services), charges follow reported usage at listed prices","url":"https://arxiv.org/abs/2611.07650","relationship":"Source-stated"},{"text":"The authors argue that advertised models and prices alone are insufficient to judge relay reliability.","excerpt":"Our study shows that advertised models and prices are insufficient to judge relay reliability.","url":"https://arxiv.org/abs/2611.07650","relationship":"Source-stated"}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07650","checkedAt":"2026-11-08T04:22:07.156Z","items":[],"note":"No suitable media was found by the supported page extraction."},"assumptions":[],"openQuestion":{"attribution":"Open question","text":"How consistently do the reported behavioral differences appear across repeated measurements?"},"whatWouldChangeView":{"attribution":"Change condition not recorded","text":""},"editorialComparisons":[],"live":{"title":"What the Middleman Delivers: How LLM API relays compare","summary":"Authors Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li and Jianbing Ni examine third-party API relays, services that provide access to large language models. They ask whether relays deliver expected model behavior, respect tool permissions and bill accurately—not just whether their advertised models and prices look attractive.\n\nThe authors compare answers from 20 relays offering GPT-6 Astra and Claude Opus 5 with batch-matched official responses. In the first-round sample, their analysis flags behavioral differences at 7/10 low-price relays and 2/10 same- or higher-price relays. Retesting changes some flags, so a single comparison may not give a stable picture.\n\nThey report no confirmed injection in 3,332 screened responses to a fixed, non-adversarial code and setup workload. But four relays return tool calls despite instructions disabling them. For identical requests, five relays report 4,200 to 60,000 more input tokens than the official API. Where balances are available, charges follow reported usage at listed prices, including extra reported input tokens at one service.\n\nThe authors argue that advertised models and prices are insufficient to judge relay reliability. They identify unexplained billed token quantities and tool-permission violations as risks, and say service evaluations need repeated measurements.","question":"How consistently do the reported behavioral differences appear across repeated measurements?","claims":[{"text":"The authors compared 20 relays offering GPT-6 Astra and Claude Opus 5 against batch-matched official responses.","excerpt":"We test 20 relays offering GPT-6 Astra and Claude Opus 5. First, we compare answers to knowledge questions with batch-matched official responses.","support":"source-stated"},{"text":"In the first-round sample, the analysis flagged behavioral differences at 7/10 low-price relays and 2/10 same- or higher-price relays; retesting changed some flags.","excerpt":"In our first-round sample, the analysis flags 7/10 low-price and 2/10 same- or higher-price relays for behavioral differences. On retesting 35 pairs, seven gain or lose a flag under the original rules","support":"source-stated"},{"text":"The authors report no confirmed injection in the screened responses, but four relays returned tool calls despite instructions disabling them.","excerpt":"No injection is confirmed in 3,332 screened responses to a fixed, non-adversarial code and setup workload, yet four relays return tool calls despite instructions disabling them in the first round.","support":"source-stated"},{"text":"For identical requests, five relays reported substantially more input tokens than the official API; where balances were available, charges followed reported usage at listed prices.","excerpt":"For identical requests, five relays report 4,200 to 60,000 more input tokens than the official API. Where balances are available (six services), charges follow reported usage at listed prices","support":"source-stated"},{"text":"The authors argue that advertised models and prices alone are insufficient to judge relay reliability.","excerpt":"Our study shows that advertised models and prices are insufficient to judge relay reliability.","support":"source-stated"}],"limitations":["The reported tests cover 20 relays and a fixed, non-adversarial code and setup workload for the injection screening; the abstract does not establish how results generalize beyond the tested services and conditions.","The abstract reports retest changes in behavioral flags, so the initial comparison does not alone establish that those differences persist across measurements."],"mediaId":"343b5bd40bb89519fbe2aa8d7318e176913cc73c0907e6df851f129cbc2dd61f","correctionReason":null,"id":"e04cb86cebbe2585a8ce32927ebd149fb06d43126eb3ab6770097b8bdd786bba","publicationId":"a3ad766d-5d81-4fef-8040-3e7af776f2ef","version":1,"sourceUrl":"https://arxiv.org/abs/2611.07650","sourceTitle":"What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays","publisher":"arXiv cs.AI","publishedAt":null,"originalPublished":{"value":"2026-11-05","precision":"day","basis":"publisher_citation_date"},"firstObservedAt":"2026-11-06T05:10:45.324Z","createdAt":"2026-11-08T03:18:44.244Z","historical":true,"entityIds":["arxiv"],"topicIds":["intelligence-and-discovery"],"media":[{"id":"343b5bd40bb89519fbe2aa8d7318e176913cc73c0907e6df851f129cbc2dd61f","kind":"article","url":"https://arxiv.org/abs/2611.07650","sourceUrl":"https://arxiv.org/abs/2611.07650","label":"Read original","provenance":"source_link","verification":{"status":"verified","checkedAt":"2026-11-08T03:17:37.037Z","method":"acquired_source","contentType":"text/html"},"recommendationReason":"Recommended by Jev for understanding this source; checked by Luna."}],"sourceMedia":{"status":"not_found","sourceUrl":"https://arxiv.org/abs/2611.07650","checkedAt":"2026-11-08T04:22:07.156Z","items":[],"note":"No suitable media was found by the supported page extraction."},"editorialDecision":{"outcome":"accept","reasonCodes":["material_development"],"rubricVersion":"stream-admission-v5","mediaId":"343b5bd40bb89519fbe2aa8d7318e176913cc73c0907e6df851f129cbc2dd61f","confidence":0.85,"provider":"typesafe","model":"jev-1.13.0","inputHash":"2a313bb95a43e98f3008f1f3d9351267abe85174a5824c860432ecba1636da24","decidedAt":"2026-11-08T03:17:37.904Z"},"provenance":{"textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observationId":"b6d105ef882e58d32525648583bb35522574dff2c57b5385287094095275174e","retrievedAt":"2026-11-08T03:17:36.884Z","kind":"publisher-page","preparer":{"model":"gpt-6-luna","runId":"95c4666e-d59a-4ddc-8e1e-1d2914c79886"},"reviewer":{"model":"gpt-6-luna","runId":"d256c0c6-4872-4a55-8067-2ebb0eeb611e"}},"audit":{"status":"uncertain","reviewedAt":"2026-11-08T03:18:44.244Z","model":"anthropic/claude-opus-5.5"},"sourceAttribution":{"authors":[{"name":"Wu, Xiaodong","url":null,"kind":"unknown"},{"name":"Zhao, Zhimin","url":null,"kind":"unknown"},{"name":"Li, Qi","url":null,"kind":"unknown"},{"name":"Cheong, Chaklam","url":null,"kind":"unknown"},{"name":"Li, Xiangman","url":null,"kind":"unknown"},{"name":"Ni, Jianbing","url":null,"kind":"unknown"}],"authorRole":"article-author","hosts":[],"status":"stated","authorRoster":{"truncated":false,"omitted":0,"assertionGroupsOmitted":0},"participation":[{"actor":{"name":"Wu, Xiaodong","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Wu, Xiaodong","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07650","locator":"head/meta[name=author|citation_author]"},"assertionId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040:participation:0","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:22:07.156Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","sourceUrl":"https://arxiv.org/abs/2611.07650","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z"}},{"actor":{"name":"Zhao, Zhimin","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Zhao, Zhimin","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07650","locator":"head/meta[name=author|citation_author]"},"assertionId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040:participation:1","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:22:07.156Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","sourceUrl":"https://arxiv.org/abs/2611.07650","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z"}},{"actor":{"name":"Li, Qi","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Qi","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07650","locator":"head/meta[name=author|citation_author]"},"assertionId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040:participation:2","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:22:07.156Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","sourceUrl":"https://arxiv.org/abs/2611.07650","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z"}},{"actor":{"name":"Cheong, Chaklam","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Cheong, Chaklam","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07650","locator":"head/meta[name=author|citation_author]"},"assertionId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040:participation:3","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:22:07.156Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","sourceUrl":"https://arxiv.org/abs/2611.07650","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z"}},{"actor":{"name":"Li, Xiangman","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Li, Xiangman","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07650","locator":"head/meta[name=author|citation_author]"},"assertionId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040:participation:4","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:22:07.156Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","sourceUrl":"https://arxiv.org/abs/2611.07650","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z"}},{"actor":{"name":"Ni, Jianbing","url":null,"kind":"unknown","actorId":null},"role":"author","scope":{"kind":"work-credit","locator":null},"literalCredit":"Ni, Jianbing","status":"stated","provenance":{"basis":"head-meta","sourceUrl":"https://arxiv.org/abs/2611.07650","locator":"head/meta[name=author|citation_author]"},"assertionId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040:participation:5","identity":{"version":"entity-identity-v1","status":"unresolved","actorId":null,"canonicalActorId":null,"candidateActorIds":[],"assertions":[],"sourceObservedAt":"2026-11-08T04:22:07.156Z","knowledgeAsOf":null},"sourceVersion":{"snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","sourceUrl":"https://arxiv.org/abs/2611.07650","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z"}}],"participationCoverage":{"total":6,"returned":6,"complete":true,"fullAssertions":"source-metadata-endpoint"},"distribution":null,"publisher":null,"sourceUrl":"https://arxiv.org/abs/2611.07650","snapshotId":"67b11f2b050e6119384dbd4566e076abfe788049def5b3e881055facc44f6040","contentSha256":"b883ecb5260ca3aee462d8e4dcef6314f2bee3fbf3bc0b9c8a787cffca7ba959","textHash":"f25e3d1d8f37bd664414f6b06ffb8449a21f0a292010ea468fa132a693891db7","observedAt":"2026-11-08T04:22:07.156Z","basis":"publisher-page-metadata","sourceLinks":66,"linksTruncated":false,"metadataUrl":"/api/pipeline/stream/e04cb86cebbe2585a8ce32927ebd149fb06d43126eb3ab6770097b8bdd786bba/source"}}}],"audits":[{"publicationId":"a3ad766d-5d81-4fef-8040-3e7af776f2ef","verdict":"uncertain","findings":[{"dimension":"attribution","status":"pass","explanation":"The draft correctly attributes the study to Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li and Jianbing Ni. It keeps the findings and conclusions framed as the authors' reported results and arguments.","excerpt":"Authors:Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li, Jianbing Ni"},{"dimension":"summary","status":"uncertain","explanation":"The structure is faithful and the question follows from the evidence. A non-specialist, however, meets central terms without explanation. 'Injection' is never explained, so 'no confirmed injection' is opaque; 'tool calls', 'input tokens' and 'batch-matched' are also left bare. The definition of relays also drops 'often at lower prices', which drives the low-price versus same/higher-price comparison. Smallest repair: restore the lower-price context and add term-library tooltips for injection, tool calls, input tokens and batch-matched, keeping the terms themselves.","excerpt":"Third-party application programming interface (API) relays offer access to large language models (LLMs), often at lower prices."},{"dimension":"date","status":"pass","explanation":"originalPublished 2026-11-05 matches the v1 submission date on arXiv. The feed date of Nov 6 is later and correctly not used.","excerpt":"[Submitted on 5 Nov 2026]"},{"dimension":"support","status":"uncertain","explanation":"Most claims match the abstract. However, the summary and claim 3 drop the scope condition 'in the first round' from the tool-call finding, presenting it as unqualified. This matters because the paper stresses that retesting changes results. Smallest repair: add 'in the first round' to that sentence. Retest figures (35 pairs; seven flag changes under the original rules, six under matched corrections) are also compressed to 'changes some flags'. That compression is acceptable but less precise.","excerpt":"yet four relays return tool calls despite instructions disabling them in the first round."},{"dimension":"media","status":"pass","explanation":"The selected media is the verified arXiv abstract page for this paper. It is the direct original source and the most relevant access point.","excerpt":null}],"createdAt":"2026-11-08T03:18:44.244Z","model":"anthropic/claude-opus-5.5"}],"correctionUrl":"https://escapemeridian.com/corrections/"}