Authors test how candidate order shapes LLM recommendations

Authors test how candidate order shapes LLM recommendations

Carlos Peláez-González, Francisco Herrera and Andrés Herrera-Poyatos study whether an LLM-based recommender’s top recommendation changes when the same candidates appear in a different order. They call this positional bias: the recommended item depends systematically on candidate order in the prompt. In a controlled product-recommendation testbed, the authors used a real product catalog, a fixed prompt template and generated user queries. Across the evaluated models, their two statistical methods rejected the hypotheses that recommendations were independent of position and uniformly distributed across positions. Their measures also distinguish ordinary sensitivity to order from cases where position pushes a recommendation toward a clearly less relevant item. The authors report that the effect persisted across changes to candidate-set size and decoding temperature, and describe it as structural and model-dependent in their test setting. That matters for systems that retrieve several options and ask an LLM to choose: the order of those options may affect what users are recommended, even when the candidates are unchanged. The paper proposes order randomization and standardized prompting as governance controls, but its evidence comes from a controlled product-recommendation testbed; how well those controls work in deployed systems remains unknown.

Publication: a5e92e9f-62f4-49d4-87ed-8815b4af7919

attribution — pass: The draft correctly names the three listed authors, Carlos Peláez-González, Francisco Herrera and Andrés Herrera-Poyatos. It keeps the findings and recommendations attributed to them rather than presenting them as established fact.

Authors:Carlos Peláez-González, Francisco Herrera, Andrés Herrera-Poyatos

summary — pass: The title, summary and question match the abstract's scope: top-1, prompt-order positional bias, a controlled product testbed, two statistical tests, a strong-bias metric, robustness checks, and proposed controls. The central idea is explained in plain language. Decoding temperature and null hypothesis are not explained in the text but are not essential to the main point; tooltips could cover them. The phrase 'proposes' is a slight rewording of 'motivate', but it stays faithful. The question is a genuine open issue that follows from the evidence.

systematic dependence of the recommended item on the order in which candidates are presented in the prompt.

date — pass: The displayed original publication date, 2026-10-27, matches the v1 submission date. The feed date of October 31 is later and is correctly not used.

[Submitted on 27 Oct 2026]

support — pass: All claims trace to the abstract. The significance sentence is conditional ('may affect') and is grounded in the permutation design and the authors' framing about deployed systems. The limitations correctly note two things: the evidence comes from a controlled testbed, and the abstract does not quantify how effective the proposed controls are. The draft does not invent any benefits or costs.

motivate routine auditing and governance controls (e.g., order randomization, standardized prompting) for deployed LLM-based RS

media — pass: The selected media is the verified direct link to the arXiv abstract page, which is the original source itself. No other supplied media exists.