How I Train an HVAC Controller
BUILDING ARDUIX
Why Arduix begins with simulation, bounded zone-setpoint control, and a system designed to learn from failure safely
John Succo | Building Arduix | August 2026
Arduix did not begin with a single breakthrough. It grew from three kinds of work that normally live in separate lanes: constructing energy models, programming PLC ladder logic, and analyzing operational data from SCADA systems.
Those experiences gave me different views of the same underlying problem. Large HVAC and manufacturing systems are dynamic thermodynamic processes. A control decision can change temperatures, flows, equipment loads, and future operating conditions. The effect may not be obvious for minutes or hours, and the best action can change with weather, occupancy, time of day, and the state of the rest of the system.
I began to see that there was no easy or practical way to manually explore all of the possible control policies. That observation became the starting point for Arduix: build a company around safely exploring better control strategies, managing the risks of applying them, and turning the successful strategies into measurable reductions in energy use.
01 / THE CONTROL PROBLEM
The control problem is bigger than a schedule
Conventional building controls are good at executing known sequences reliably. Schedules, setpoints, deadbands, reset strategies, and PID loops have operated buildings for decades. The difficulty is not that these tools are useless. The difficulty is that an engineer must decide in advance which rules to write and how those rules should respond across a large number of interacting conditions.
That challenge grows quickly in a multizone building. A setpoint that is reasonable for a sun-exposed perimeter zone may be wasteful for a stable core zone. A change that reduces cooling energy may increase airflow and fan energy somewhere else. A strategy that works in mild weather may fail during a hot afternoon. There are too many combinations to evaluate one reset sequence at a time.
The opportunity is not merely to automate an existing schedule. It is to search a much larger control space while keeping every action bounded, observable, and reversible.
02 / THE APPROACH
Why deep reinforcement learning
My hypothesis was that deep reinforcement learning could explore the feedback loop between an action and the environment's response faster and more systematically than I could by manually programming and comparing thousands of control rules. In reinforcement learning, the controller observes the current state, takes an action, sees how the environment changes, and receives feedback based on objectives such as energy use, comfort, and control stability.
That does not make the controller intelligent in a human sense, and it does not remove the need for engineering judgment. The observation space, actions, reward, operating bounds, validation tests, and fallback behavior all have to be designed. The learning algorithm can search within that structure, but the structure determines what a safe and useful result means.
For the first Arduix controller, I chose zone cooling setpoints as the entry point. They are commonly exposed through building automation systems, they can be changed gradually, and their authority can be limited. Compared with directly controlling safety-critical industrial equipment, bounded supervisory setpoint adjustments offer a lower-risk way to test the product and its deployment process.
I do not believe zone setpoints alone are dynamic enough to support the entire long-term business. They are a deliberate first control surface, not the finish line. The larger Arduix vision is more holistic supervisory control: coordinating zone behavior with supply-air temperature and other system-level processes where the interactions are harder to capture in static reset logic.
03 / THE DATA
Why not begin with terabytes of building data?
Some approaches to building optimization begin by collecting very large historical datasets. For a solo founder, that was not a practical starting point. Even if I could obtain terabytes of BAS data, the data would still reflect the policies that had already been used. It would not automatically reveal what would have happened if the building had taken a different action.
Historical BAS data is also messy. Points may be renamed, unavailable, overridden, sampled at different intervals, or affected by maintenance events that were never recorded. Most important, a live building is not a safe place for an unproven policy to explore both good and bad decisions.
Simulation changes the role of real data. Instead of waiting for a perfect dataset before building anything, I can begin with a model that is good enough to capture the important thermal and HVAC relationships. Real telemetry can later be used to calibrate that model, quantify the sim-to-real gap, and determine how much authority the policy should receive in a pilot.
04 / THE TRAINING GROUND
EnergyPlus as a digital training ground
For that training environment, I chose EnergyPlus, the U.S. Department of Energy's open-source building energy simulation engine. EnergyPlus has been developed and used for decades to model building envelopes, internal loads, weather, zones, and HVAC systems. The official project is available at energyplus.net.
EnergyPlus is not a perfect representation of a specific real building, and I do not describe the current model as a calibrated digital twin. The goal is a useful digital training ground: close enough to reproduce the important physical relationships, deterministic enough to compare policies, and flexible enough to be improved later with site data.
The current Arduix environment uses a roughly 50,000-square-foot, three-story Denver prototype medium-office model. It contains 15 occupied zones distributed across the bottom, middle, and top floors, with a separate air system serving each level. Weather files provide the external conditions under which the policy has to operate.
05 / THE CONTROLLER
What the controller sees and changes
Every 15 minutes of simulated time, the controller assembles an observation of the building. Depending on the experiment, that observation can include per-zone temperatures and setpoints, recent temperature trends, previous actions, occupancy, time, outdoor conditions, and selected forecast or system-level features.
The policy then produces one normalized cooling-setpoint action for each controlled zone. The action is not a direct compressor or fan command. It is a bounded supervisory adjustment that the simulated building's existing HVAC controls must respond to. The policy might slightly pre-cool a perimeter zone that is heating rapidly while leaving a stable core zone closer to its normal setpoint.
The reward combines interval HVAC electricity with comfort and control-behavior terms. This is important because the mathematically easiest way to save cooling energy is to stop cooling. A usable policy must reduce energy without achieving that reduction by making occupied zones uncomfortable or by producing unstable setpoint movement.
A policy that saves energy by making the building uncomfortable is not a successful policy.
06 / THE TRAINING LOOP
The training loop
Generate experience
A Python orchestrator launches parallel EnergyPlus workers in Docker containers. The current configuration can keep 16 CPU-bound workers active, each simulating a randomized five-day window from the training season. At startup, each worker reads the latest available policy.
Record the consequences
During the simulation, the worker records observations, actions, rewards, next observations, energy, comfort, and diagnostic information. When the window finishes, it writes the trajectory to a shared location and marks it complete.
Update the policy
The orchestrator ingests completed trajectories into replay memory. A centralized TD3 learner uses the GPU to update the policy and publishes a new snapshot. Workers already running finish with the policy they loaded; newly launched workers receive the newer version.
Validate before evaluation
At fixed intervals, retained checkpoints are tested over early-, middle-, and late-season validation windows. Candidates are classified as feasible or infeasible using comfort and hard temperature limits. An energy improvement cannot become the selected policy if it fails those constraints.
Compare against the same baseline
The selected candidate is evaluated against the baseline using the same model, weather, dates, zones, and observation configuration. The analysis compares HVAC electricity, demand, comfort severity, fan and cooling components, and zone-level behavior from the exact evaluation traces.
Track and promote
Training, evaluation, and analysis metrics are logged through MLflow into Azure Machine Learning. Azure ML is the tracking and model-governance layer; it does not run the current training cycle. A policy bundle is registered only after it passes the configured evaluation gate.
07 / THE ARCHITECTURE
Why the architecture is deliberately simple
The current system runs on one Azure VM. The orchestrator and learner run on the host, and the EnergyPlus simulations run in Docker containers. Policies and trajectories move through a shared directory mounted into every container. Azure Machine Learning stores experiment history, artifacts, lineage, and accepted model bundles.
This is not an infinitely scalable cloud platform, and it is not intended to be one yet. For v1, the important problem is not maximum infrastructure scale. It is whether the controller can learn a repeatable energy-and-comfort advantage, whether failures can be diagnosed, and whether a policy can be promoted through a transparent gate. The architecture is complex enough to answer those questions without hiding the experiment inside unnecessary infrastructure.
08 / THE REALITY CHECK
What simulation can and cannot prove
Simulation can show whether the learning loop works. It can expose bad reward design, missing observations, unstable control behavior, and energy savings that are really comfort failures. It can also compare policies under identical weather and operating conditions in a way that is impossible in two separate real-world weeks.
It cannot prove that the same percentage of savings will appear in a real building. The current model does not contain every sensor error, actuator delay, override, maintenance issue, occupant complaint, or undocumented sequence that exists in an operating BAS. It is also not a substitute for calibration and measurement and verification.
The eventual deployment path has to acknowledge that gap. A real pilot should begin with point mapping and read-only validation, progress through shadow-mode recommendations, and introduce limited supervisory control only inside explicit bounds. Every action should be logged, reversible, and subordinate to the building's existing safety controls and fallback sequence.
09 / WHAT COMES NEXT
A first control surface, not the final destination
Zone-setpoint control is a useful place to begin because the integration is understandable and the authority can be constrained. It is also a demanding enough problem to reveal whether the training method is credible. But the long-term opportunity is larger: coordinating zone decisions with supply-air temperature and other dynamic system processes that are difficult to optimize as independent reset sequences.
That is the direction I want Arduix to grow toward: a bounded supervisory optimization layer that works through the existing BAS, respects the operating envelope, and keeps searching for better ways to run the whole system as conditions change.
The goal is not to replace operators with an autonomous black box. It is to give buildings a transparent, bounded way to keep searching for better control policies as conditions change.
The next article in this series will cover the first major lesson from training: the controller learned to save energy, but some early policies found those savings by allowing particular zones to get too warm. Fixing that behavior required a better definition of comfort and better information about which zones were approaching risk.
