Kolluru, Sailendra, Duvvarapu, Maheshkumar · Nov 12, 2026
arXiv cs.AI: Small Language Models for Inference-Capability Routing
Markdown · JSON · Original source
Authors Sailendra Kolluru and Maheshkumar Duvvarapu investigate whether prompt optimization can help small language models decide when to activate reasoning, retrieval, web search, or . Activating capabilities indiscriminately can waste computation, while missing a capability can reduce answer quality. Their framework makes four independent capability decisions, allowing multiple capabilities per query, and compares Qwen2.5 routers with different capacities and prompting strategies. Under minimal prompting, greater router capacity does not consistently improve classification. Prompt optimization narrows the capacity gap for reasoning and calculator routing, while and web search retain larger benefits from higher capacity. In an evaluation using GPT-OSS-120B on 127 questions, the 14B router’s mean BERTScore F1 was 0.8408, compared with 0.8418 when all capabilities were enabled. The authors report 68.7% lower average downstream token consumption and 80.5% lower median downstream with the router. These savings exclude router overhead, which was measured separately in a different inference environment.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 12, 2026 (2026-11-12).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T11:16:44.345Z. First observed: 2026-11-13T07:42:51.723Z. Retrieved: 2026-11-13T11:02:46.916Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported downstream evaluation uses one fixed model, GPT-OSS-120B, and 127 questions; the evidence does not establish how results generalize to other models or query sets. The reported token and savings exclude router overhead. The authors measured that overhead separately in a different inference environment, so the evidence does not give a same-environment net comparison.
Open question
How would including router overhead in the same inference environment change the reported efficiency comparison between router configurations?
Source-stated
The authors investigate prompt-driven small language models that make independent decisions about whether to activate four capabilities, with multiple capabilities allowed per query.
The proposed framework uses prompt-driven small language models to make four independent capability decisions, allowing multiple capabilities to be selected per query.
Source-stated
Under minimal prompting, increasing router capacity does not consistently improve classification; prompt optimization's effect varies by capability.
Under minimal prompting, increasing router capacity does not consistently improve classification performance. Prompt optimization narrows the capacity gap for reasoning and calculator routing, whereas RAG and web search retain larger benefits from higher capacity.
Source-stated
The authors report that similar classification F1 scores can still produce different downstream behavior: false positives consume computation, while missed capabilities can reduce answer quality.
Similar classification F1 scores do not necessarily imply equivalent downstream behavior: false-positive activations increase unnecessary computation, while missed capabilities can reduce answer quality.
Source-stated
In the reported evaluation, the 14B router's mean BERTScore F1 was close to the all-capabilities configuration, with lower downstream token consumption and latency; the savings exclude router overhead.
The 14B router achieves a mean BERTScore F1 of 0.8408, compared with 0.8418 when all capabilities are enabled, while reducing average downstream token consumption by 68.7\% (8,837 to 2,764 tokens) and median downstream latency by 80.5\% (9.79 to 1.91 seconds). These savings exclude router overhead
Source and contributor context
Kolluru, Sailendra, Duvvarapu, Maheshkumar · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Tool use
A model's ability to operate other software, such as a search engine, a calculator or a code runner, while it works.
Retrieval-augmented generation
A setup in which a model first looks up relevant documents and then answers using them.
Latency
The delay between sending a request and receiving the result.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: f504401e-a416-4720-8a6a-a4ccb4452a37.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-13T11:16:44.345Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 8701619f725c999fb88d7c0ea9b9e63ec7c5946392aa3517e21bc2c0c1ae5b4d. Source text hash: 5b0a7f7e180e34b2a3c73168aa28e574e8824ceda6eaabe9951173e156ce7d98.
Revision and source-check history
Version 1 · 2026-11-13T11:16:44.345Z
arXiv cs.AI: Small Language Models for Inference-Capability Routing
Authors Sailendra Kolluru and Maheshkumar Duvvarapu investigate whether prompt optimization can help small language models decide when to activate reasoning, retrieval, web search, or . Activating capabilities indiscriminately can waste computation, while missing a capability can reduce answer quality. Their framework makes four independent capability decisions, allowing multiple capabilities per query, and compares Qwen2.5 routers with different capacities and prompting strategies. Under minimal prompting, greater router capacity does not consistently improve classification. Prompt optimization narrows the capacity gap for reasoning and calculator routing, while and web search retain larger benefits from higher capacity. In an evaluation using GPT-OSS-120B on 127 questions, the 14B router’s mean BERTScore F1 was 0.8408, compared with 0.8418 when all capabilities were enabled. The authors report 68.7% lower average downstream token consumption and 80.5% lower median downstream with the router. These savings exclude router overhead, which was measured separately in a different inference environment.
Original source
cs.AI · Article
Source details
Original source date: Nov 12, 2026 (2026-11-12).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-13T11:16:44.345Z. First observed: 2026-11-13T07:42:51.723Z. Retrieved: 2026-11-13T11:02:46.916Z.
Limits
The reported downstream evaluation uses one fixed model, GPT-OSS-120B, and 127 questions; the evidence does not establish how results generalize to other models or query sets. The reported token and savings exclude router overhead. The authors measured that overhead separately in a different inference environment, so the evidence does not give a same-environment net comparison.
Open question
How would including router overhead in the same inference environment change the reported efficiency comparison between router configurations?
Source-stated
The authors investigate prompt-driven small language models that make independent decisions about whether to activate four capabilities, with multiple capabilities allowed per query.
The proposed framework uses prompt-driven small language models to make four independent capability decisions, allowing multiple capabilities to be selected per query.
Source-stated
Under minimal prompting, increasing router capacity does not consistently improve classification; prompt optimization's effect varies by capability.
Under minimal prompting, increasing router capacity does not consistently improve classification performance. Prompt optimization narrows the capacity gap for reasoning and calculator routing, whereas RAG and web search retain larger benefits from higher capacity.
Source-stated
The authors report that similar classification F1 scores can still produce different downstream behavior: false positives consume computation, while missed capabilities can reduce answer quality.
Similar classification F1 scores do not necessarily imply equivalent downstream behavior: false-positive activations increase unnecessary computation, while missed capabilities can reduce answer quality.
Source-stated
In the reported evaluation, the 14B router's mean BERTScore F1 was close to the all-capabilities configuration, with lower downstream token consumption and latency; the savings exclude router overhead.
The 14B router achieves a mean BERTScore F1 of 0.8408, compared with 0.8418 when all capabilities are enabled, while reducing average downstream token consumption by 68.7\% (8,837 to 2,764 tokens) and median downstream latency by 80.5\% (9.79 to 1.91 seconds). These savings exclude router overhead
Source and contributor context
Kolluru, Sailendra, Duvvarapu, Maheshkumar · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Tool use
A model's ability to operate other software, such as a search engine, a calculator or a code runner, while it works.
Retrieval-augmented generation
A setup in which a model first looks up relevant documents and then answers using them.
Latency
The delay between sending a request and receiving the result.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: f504401e-a416-4720-8a6a-a4ccb4452a37.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-13T11:16:44.345Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 8701619f725c999fb88d7c0ea9b9e63ec7c5946392aa3517e21bc2c0c1ae5b4d. Source text hash: 5b0a7f7e180e34b2a3c73168aa28e574e8824ceda6eaabe9951173e156ce7d98.
uncertain · 2026-11-13T11:16:44.345Z · anthropic/claude-opus-5.5
Publication: f504401e-a416-4720-8a6a-a4ccb4452a37
attribution — pass: The reading correctly attributes the study to authors Sailendra Kolluru and Maheshkumar Duvvarapu. It frames the findings as the authors' investigation and reported results rather than established fact.
Authors:Sailendra Kolluru, Maheshkumar Duvvarapu
summary — uncertain: The summary omits a condition that affects how the headline numbers should be read: router overhead increases with router capacity, and the reported savings come from the largest (14B) router. It also leaves out the authors' conclusion that router selection must weigh error profiles and overhead, not just classification scores. Several central terms are never explained in plain language: router, BERTScore F1, RAG and downstream tokens. Smallest repair: add that overhead grows with router capacity, add the authors' conclusion, and add plain glosses or tooltips for those terms.
These savings exclude router overhead, which is measured separately in a different inference environment and increases with router capacity.
date — pass: The originalPublished date of 2026-11-12 matches the v1 submission date on arXiv. The later feed date of 2026-11-13 was correctly not used.
[Submitted on 12 Nov 2026]
support — pass: Each claim, the question and the limitations are grounded in the abstract. The numbers match the source: 0.8408 vs 0.8418, 68.7%, 80.5% and 127 questions. The savings are correctly attributed as excluding router overhead. No benefits or costs are invented. The question follows from the stated limitation about measuring overhead in a different environment.
The 14B router achieves a mean BERTScore F1 of 0.8408, compared with 0.8418 when all capabilities are enabled
media — pass: The selected media is the verified direct link to the arXiv abstract page, which is the original source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
A model's ability to operate other software, such as a search engine, a calculator or a code runner, while it works.
Definition referenceA setup in which a model first looks up relevant documents and then answers using them.
Definition referenceThe delay between sending a request and receiving the result.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivThe delay between sending a request and receiving the result.
Definition referenceA model's ability to operate other software, such as a search engine, a calculator or a code runner, while it works.
Definition referenceA setup in which a model first looks up relevant documents and then answers using them.
Definition referenceThe delay between sending a request and receiving the result.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivThe delay between sending a request and receiving the result.
Definition reference