# arXiv eess.SY: Evaluating Rollout-Verified Reinforcement Fine-Tuning for Building Control [Canonical reading](https://escapemeridian.com/readings/arxiv-eess-sy-evaluating-rollout-verified-reinforcement-fine-2a49d9a6) Takumi Shioda, Kohei Terashima and Tatsuo Nagai investigate whether reasoning models can control multi-zone variable-air-volume systems without building-specific training, and whether rollout-verified reinforcement fine-tuning can improve a smaller model. In a physics-based four-zone emulator, GPT-5 used less electricity than a Guideline 36-based baseline, but also used less outdoor air and had lower CO₂ compliance. A trained reinforcement learning controller reduced electricity use while improving temperature and CO₂ compliance. Yet a critic whose estimates closely tracked rollout returns often failed to rank actions correctly within the same state. Fine-tuning gpt-oss-20b with rollout returns produced no sustained improvement in sampled-action returns or closed-loop performance, and the authors found unreliable next-state predictions before and after fine-tuning. The results show that energy use, air quality and temperature can move in different directions, and that aggregate agreement between a verifier and rollout returns did not ensure useful action choices. The study used an emulator over five summer days; whether the results carry over to real buildings remains unknown. The authors propose testing supervised fine-tuning on next-state prediction before subsequent rollout-verified reinforcement fine-tuning. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source arXiv eess.SY · Article Original source date: Jul 30, 2026 (2026-07-30). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T10:40:43.849Z. First observed: 2026-11-02T05:06:47.910Z. Retrieved: 2026-11-02T07:46:56.225Z. - [Evaluating Rollout-Verified Reinforcement Fine-Tuning of a Reasoning Model for Multi-Zone Variable-Air-Volume Control](https://arxiv.org/abs/2607.27914) ## Limits The evaluation was conducted in a physics-based four-zone emulator over five summer days; the abstract does not report tests in real buildings. The abstract does not specify how the observed energy, outdoor-air, temperature and CO₂ outcomes would transfer to other buildings or operating conditions. The abstract describes one 200-step fine-tuning run, so it does not establish whether other runs or training setups would produce different results. ## Open question Would supervised fine-tuning on next-state prediction improve subsequent rollout-verified reinforcement fine-tuning, as the authors propose testing? ## Source-stated The authors evaluate model-based control and fine-tuning in a physics-based four-zone emulator over five summer days. > Five controllers were evaluated over five summer days in a physics-based four-zone emulator. - [Supporting source](https://arxiv.org/abs/2607.27914) ## Source-stated GPT-5 used less electricity than the baseline but also used less outdoor air and had lower CO₂ compliance. > GPT-5 used 6.8-7.4% less electricity than a Guideline 36-based baseline, but used less outdoor air and had lower CO$\_2$ compliance. - [Supporting source](https://arxiv.org/abs/2607.27914) ## Source-stated A trained reinforcement learning controller used less electricity while improving temperature and CO₂ compliance relative to the baseline. > A trained reinforcement learning controller used 4.5% less electricity than the baseline while improving temperature and CO$\_2$ compliance. - [Supporting source](https://arxiv.org/abs/2607.27914) ## Source-stated The critic's high correlation with rollout returns did not reliably identify the best action within a state. > Across 470 candidates, their correlation was 0.9995, yet the critic selected the best action in only 7 of 30 states. - [Supporting source](https://arxiv.org/abs/2607.27914) ## Source-stated The fine-tuning run did not improve sampled-action returns or closed-loop performance, and next-state predictions remained unreliable. > One 200-step run produced no sustained improvement in the returns of sampled actions and no improvement in closed-loop performance. Analysis of the model's reasoning traces and quantitative tests showed that its next-state predictions of temperature and CO$\_2$ were unreliable both before and after - [Supporting source](https://arxiv.org/abs/2607.27914) ## Source-stated The authors identify supervised fine-tuning on next-state prediction as a direction to test before subsequent rollout-verified reinforcement fine-tuning. > these findings motivate testing whether supervised fine-tuning on next-state prediction improves subsequent rollout-verified reinforcement fine-tuning. - [Supporting source](https://arxiv.org/abs/2607.27914) ## Source and contributor context Shioda, Takumi, Terashima, Kohei, Nagai, Tatsuo · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Fine-tuning Further training of an existing model to adapt it to a particular task or use. - [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning) ## Reasoning model A language model trained to work through a problem in written steps before it answers. - [Definition reference](https://en.wikipedia.org/wiki/Reasoning_model) ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 40468c9f-c9ca-41cd-8534-9fb73360ca89. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T10:40:43.849Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: c722943255a532e3bc2500196f873ed5feddbbf6b6fe40b49627e0a0f22f12c4. Source text hash: d1eff08c869938785076002b257cdfab51a3f218e3f36ab7133bc55ea553a6db. ## Revision and source-check history ### Version 1 — 2026-11-10T10:40:43.849Z Takumi Shioda, Kohei Terashima and Tatsuo Nagai investigate whether reasoning models can control multi-zone variable-air-volume systems without building-specific training, and whether rollout-verified reinforcement fine-tuning can improve a smaller model. In a physics-based four-zone emulator, GPT-5 used less electricity than a Guideline 36-based baseline, but also used less outdoor air and had lower CO₂ compliance. A trained reinforcement learning controller reduced electricity use while improving temperature and CO₂ compliance. Yet a critic whose estimates closely tracked rollout returns often failed to rank actions correctly within the same state. Fine-tuning gpt-oss-20b with rollout returns produced no sustained improvement in sampled-action returns or closed-loop performance, and the authors found unreliable next-state predictions before and after fine-tuning. The results show that energy use, air quality and temperature can move in different directions, and that aggregate agreement between a verifier and rollout returns did not ensure useful action choices. The study used an emulator over five summer days; whether the results carry over to real buildings remains unknown. The authors propose testing supervised fine-tuning on next-state prediction before subsequent rollout-verified reinforcement fine-tuning. ## Original source arXiv eess.SY · Article Original source date: Jul 30, 2026 (2026-07-30). Date basis: Date basis not provided. Added to Escape Meridian: 2026-11-10T10:40:43.849Z. First observed: 2026-11-02T05:06:47.910Z. Retrieved: 2026-11-02T07:46:56.225Z. - [Evaluating Rollout-Verified Reinforcement Fine-Tuning of a Reasoning Model for Multi-Zone Variable-Air-Volume Control](https://arxiv.org/abs/2607.27914) ## Limits The evaluation was conducted in a physics-based four-zone emulator over five summer days; the abstract does not report tests in real buildings. The abstract does not specify how the observed energy, outdoor-air, temperature and CO₂ outcomes would transfer to other buildings or operating conditions. The abstract describes one 200-step fine-tuning run, so it does not establish whether other runs or training setups would produce different results. ## Open question Would supervised fine-tuning on next-state prediction improve subsequent rollout-verified reinforcement fine-tuning, as the authors propose testing? ## Source-stated The authors evaluate model-based control and fine-tuning in a physics-based four-zone emulator over five summer days. > Five controllers were evaluated over five summer days in a physics-based four-zone emulator. - [Supporting source](https://arxiv.org/abs/2607.27914) ## Source-stated GPT-5 used less electricity than the baseline but also used less outdoor air and had lower CO₂ compliance. > GPT-5 used 6.8-7.4% less electricity than a Guideline 36-based baseline, but used less outdoor air and had lower CO$\_2$ compliance. - [Supporting source](https://arxiv.org/abs/2607.27914) ## Source-stated A trained reinforcement learning controller used less electricity while improving temperature and CO₂ compliance relative to the baseline. > A trained reinforcement learning controller used 4.5% less electricity than the baseline while improving temperature and CO$\_2$ compliance. - [Supporting source](https://arxiv.org/abs/2607.27914) ## Source-stated The critic's high correlation with rollout returns did not reliably identify the best action within a state. > Across 470 candidates, their correlation was 0.9995, yet the critic selected the best action in only 7 of 30 states. - [Supporting source](https://arxiv.org/abs/2607.27914) ## Source-stated The fine-tuning run did not improve sampled-action returns or closed-loop performance, and next-state predictions remained unreliable. > One 200-step run produced no sustained improvement in the returns of sampled actions and no improvement in closed-loop performance. Analysis of the model's reasoning traces and quantitative tests showed that its next-state predictions of temperature and CO$\_2$ were unreliable both before and after - [Supporting source](https://arxiv.org/abs/2607.27914) ## Source-stated The authors identify supervised fine-tuning on next-state prediction as a direction to test before subsequent rollout-verified reinforcement fine-tuning. > these findings motivate testing whether supervised fine-tuning on next-state prediction improves subsequent rollout-verified reinforcement fine-tuning. - [Supporting source](https://arxiv.org/abs/2607.27914) ## Source and contributor context Shioda, Takumi, Terashima, Kohei, Nagai, Tatsuo · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media No suitable media was found by the supported page extraction. ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Fine-tuning Further training of an existing model to adapt it to a particular task or use. - [Definition reference](https://developers.google.com/machine-learning/glossary#fine-tuning) ## Reasoning model A language model trained to work through a problem in written steps before it answers. - [Definition reference](https://en.wikipedia.org/wiki/Reasoning_model) ## Reinforcement learning Training by trial and error, in which a system is rewarded for the outcomes its designers want. - [Definition reference](https://developers.google.com/machine-learning/glossary#reinforcement-learning-rl) ## arXiv arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed. - [Explore this context](/entities/arxiv) - [Reference](https://arxiv.org/) - [Reference](https://info.arxiv.org/help/api/index.html) ## Editorial provenance Version 1. Publication ID: 40468c9f-c9ca-41cd-8534-9fb73360ca89. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T10:40:43.849Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: c722943255a532e3bc2500196f873ed5feddbbf6b6fe40b49627e0a0f22f12c4. Source text hash: d1eff08c869938785076002b257cdfab51a3f218e3f36ab7133bc55ea553a6db. ### uncertain — 2026-11-10T10:40:43.849Z — anthropic/claude-opus-5.5 Publication: 40468c9f-c9ca-41cd-8534-9fb73360ca89 attribution — pass: The reading correctly credits Takumi Shioda, Kohei Terashima and Tatsuo Nagai and keeps the findings framed as their study results. Naming GPT-5 and gpt-oss-20b matches the abstract. > Authors:Takumi Shioda, Kohei Terashima, Tatsuo Nagai summary — uncertain: The reading is faithful, but a non-specialist could not follow its core. It never says these are building heating, ventilation and air-conditioning (HVAC) systems, the abstract's own framing. Central terms are unexplained: variable-air-volume, rollout-verified reinforcement fine-tuning, rollout returns, critic, closed-loop, Guideline 36 and next-state prediction. Smallest repair: open with the HVAC energy context. Add term-library tooltips for these terms. State that the critic came from the trained RL controller and was tested as a lower-cost verifier. 'Reasoning models' should be 'a frontier reasoning model'. > To assess this controller's critic as a lower-cost verifier, we compared its estimates with deterministic rollout returns. date — pass: The displayed original publication date of 2026-07-30 matches the v1 submission date. It is not the 30 Oct 2026 v2 revision or the November feed date. > Submitted on 30 Jul 2026 (v1), last revised 30 Oct 2026 (this version, v2) support — pass: The claims match the abstract: the GPT-5 tradeoff, the RL controller's gains, the critic's 0.9995 correlation but only 7 of 30 best-action picks, the failed 200-step fine-tuning run, unreliable next-state predictions and the proposed next test. Two lines are interpretive but grounded. One says energy, air quality and temperature can diverge; the other says aggregate agreement did not ensure good action choices. The stated limitations do not overreach. > High overall correlation did not ensure reliable ranking at the same state. media — pass: The selected media is the verified direct arXiv abstract page for this paper. That is the most direct way to access the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/2a49d9a6d667099c491d8c31a67877aa55e0a907ec1d26d62ce6f63a47f75f5f/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/arxiv-eess-sy-evaluating-rollout-verified-reinforcement-fine-2a49d9a6.json)