Local Epochs Inefficiency Caused by Device Heterogeneity in Federated Learning. (6th January 2022)

• Record Type: Journal Article
• Authors: Zeng, Yan; Wang, Xin; Yuan, Junfeng; Zhang, Jilin; Wan, Jian
• Other Names: Xiong Jinbo Academic Editor.
• Is Part Of: Wireless communications and mobile computing. Volume 2022(2022)
• Publication Date: 2022-01-06
• Subjects: Wireless communication systems -- Periodicals; Mobile communication systems -- Periodicals; 621.38205
• DOI: 10.1155/2022/6887040
• Languages: English
• ISSNs: 1530-8669

Abstract

Abstract : Federated learning is a new framework of machine learning, it trains models locally on multiple clients and then uploads local models to the server for model aggregation iteratively until the model converges. In most cases, the local epochs of all clients are set to the same value in federated learning. In practice, the clients are usually heterogeneous, which leads to the inconsistent training speed of clients. The faster clients will remain idle for a long time to wait for the slower clients, which prolongs the model training time. As the time cost of clients' local training can reflect the clients' training speed, and it can be used to guide the dynamic setting of local epochs, we propose a method based on deep learning to predict the training time of models on heterogeneous clients. First, a neural network is designed to extract the influence of different model features on training time. Second, we propose a dimensionality reduction rule to extract the key features which have a great impact on training time based on the influence of model features. Finally, we use the key features extracted by the dimensionality reduction rule to train the time prediction model. Our experiments show that, compared with the current prediction method, our method reduces 30% of model features and 25% of training data for the convolutional layer, 20% of model features and 20% of training data for the dense layer, while maintaining the same level of prediction error.