Impact of etcd deployment on Kubernetes, Istio, and application performance. (7th August 2020)

• Record Type: Journal Article
• Authors: Larsson, Lars; Tärneberg, William; Klein, Cristian; Elmroth, Erik; Kihl, Maria
• Is Part Of: Software, practice & experience. Volume 50:Number 10(2020)
• Page Start: 1986
• Page End: 2007
• Publication Date: 2020-08-07
• Subjects: cloud computing -- distributed systems -- etcd -- Kubernetes -- performance; Computer software -- Periodicals; Computer programming -- Periodicals; Computer programs -- Periodicals; 005.3
• DOI: 10.1002/spe.2885
• Languages: English
• ISSNs: 0038-0644

Abstract

Summary: This experience article describes lessons learned as we conducted experiments in a Kubernetes‐based environment, the most notable of which was that the performance of both the Kubernetes control plane and the deployed application depends strongly and in unexpected ways on the performance of the etcd database. The article contains (a) detailed descriptions of how networking with and without Istio works in Kubernetes, based on the Flannel Container Networking Interface (CNI) provider in VXLAN mode with IP Virtual Server (IPVS)‐backed Kubernetes Services, (b) a comprehensive discussion about how to conduct load and performance testing using a closed‐loop workload generator, and (c) an open source experiment framework useful for executing experiments in a shared cloud environment and exploring the resulting data. It also shows that statistical analysis may reveal the data resulting from such experiments to be misleading even when careful preparations are made, and that nondeterministic behavior stemming from etcd can affect both the platform as a whole and the deployed application. Finally, it is demonstrated that using high‐performance backing storage for etcd can reduce the occurrence of such nondeterministic behaviors by a statistically significant ( P < .05) margin. The implication of this experience article is that systems researchers studying the performance of applications deployed on Kubernetes cannot simply consider their specific application to be under test. Instead, the particularities of the underlying Kubernetes and cloud platform must be taken into account, in particular because their performance can impact that of etcd.