A deep recurrent Q network towards self‐adapting distributed microservice architecture. (28th November 2019)

• Record Type: Journal Article
• Authors: Magableh, Basel; Almiani, Muder
• Is Part Of: Software, practice & experience. Volume 50:Number 2(2020)
• Page Start: 116
• Page End: 135
• Publication Date: 2019-11-28
• Subjects: deep Q‐learning networks -- multiagent environment -- policy approximation -- Q‐learning algorithms -- recurrent Q‐learning networks -- reinforcement learning -- self‐adaptive architectures -- service‐oriented architecture; Computer software -- Periodicals; Computer programming -- Periodicals; Computer programs -- Periodicals; 005.3
• DOI: 10.1002/spe.2778
• Languages: English
• ISSNs: 0038-0644

Abstract

Summary: One desired aspect of microservice architecture is the ability to self‐adapt its own architecture and behavior in response to changes in the operational environment. To achieve the desired high levels of self‐adaptability, this research implements distributed microservice architecture model running a swarm cluster, as informed by the Monitor, Analyze, Plan, and Execute over a shared Knowledge (MAPE‐K) model. The proposed architecture employs multiadaptation agents supported by a centralized controller, which can observe the environment and execute a suitable adaptation action. The adaptation planning is managed by a deep recurrent Q‐learning network (DRQN). It is argued that such integration between DRQN and Markov decision process (MDP) agents in a MAPE‐K model offers distributed microservice architecture with self‐adaptability and high levels of availability and scalability. Integrating DRQN into the adaptation process improves the effectiveness of the adaptation and reduces any adaptation risks, including resource overprovisioning and thrashing. The performance of DRQN is evaluated against deep Q‐learning and policy gradient algorithms, including (1) a deep Q‐learning network (DQN), (2) a dueling DQN (DDQN), (3) a policy gradient neural network, and (4) deep deterministic policy gradient. The DRQN implementation in this paper manages to outperform the aforementioned algorithms in terms of total reward, less adaptation time, lower error rates, plus faster convergence and training time. We strongly believe that DRQN is more suitable for driving the adaptation in distributed services‐oriented architecture and offers better performance than other dynamic decision‐making algorithms.