Multi-agent reinforcement learning for modeling and control of thermostatically controlled loads. (15th March 2019)

• Record Type: Journal Article
• Authors: Kazmi, Hussain; Suykens, Johan; Balint, Attila; Driesen, Johan
• Is Part Of: Applied energy. Volume 238(2019)
• Page Start: 1022
• Page End: 1035
• Publication Date: 2019-03-15
• Subjects: Multi agent reinforcement learning -- Distributed learning -- Optimal control -- Thermostatically controlled loads -- Domestic hot water storage vessel -- Heat pumps; Power (Mechanics) -- Periodicals; Energy conservation -- Periodicals; Energy conversion -- Periodicals; 621.042
• DOI: 10.1016/j.apenergy.2019.01.140
• Languages: English
• ISSNs: 0306-2619

Abstract

Highlights: A multi-agent reinforcement learning framework for black-box modelling is presented. Agency can be used interchangeably with increased sensing or domain knowledge. The framework accelerates modelling performance linearly with increasing agents. Multi-agent systems learn faster and better than a comparable single agent system. Efficiency gains of 20% were realized in a real world pilot running for a year. Abstract: Increasing energy efficiency of thermostatically controlled loads has the potential to substantially reduce domestic energy demand. However, optimizing the efficiency of thermostatically controlled loads requires either an existing model or detailed data from sensors to learn it online. Often, neither is practical because of real-world constraints. In this paper, we demonstrate that this problem can benefit greatly from multi-agent learning and collaboration. Starting with no thermostatically controlled load specific information, the multi-agent modelling and control framework is evaluated over an entire year of operation in a large scale pilot in The Netherlands, constituting over 50 houses, resulting in energy savings of almost 200 kW h per household (or 20% of the energy required for hot water production). Theoretically, these savings can be even higher, a result also validated using simulations. In these experiments, model accuracy in the multi-agent frameworks scales linearly with the number of agents and provides compelling evidence for increased agency as an alternative to additional sensing, domain knowledge or data gathering time. In fact, multi-agent systems can accelerate learning of a thermostatically controlled load's behaviour by multiple orders of magnitude over single-agent systems, enabling active control faster. These findings hold even when learning is carried out in a distributed manner to address privacy issues arising from multi-agent cooperation.