Federated Learning is a privacy-preserving, machine learning technique that generates a globally shared model with in-situ model training on distributed devices. These systems are often comprised of millions of user devices and only a subset of available devices can be used for training in each epoch. Designing a device selection strategy is challenging, given that devices are highly heterogeneous in both their system resources and training data. This heterogeneity makes device selection very crucial for timely model convergence and sufficient model accuracy. Existing approaches have addressed system heterogeneity for device selection but have largely ignored the data heterogeneity. In this work, we analyze the impact of data heterogeneity on device selection, model convergence, model accuracy, and fault tolerance in a federated learning setting. Based on our analysis, we propose that clustering devices with similar data distributions followed by selecting the devices with the best processing capacity from each cluster can significantly improve the model convergence without compromising model accuracy. This clustering also guides us in designing policies for fault tolerance in the system. We propose three methods for identifying groups of devices with similar data distributions. We also identify and discuss rich trade-offs between privacy, bandwidth consumption, and computation overhead for each of these proposed methods. Our preliminary experiments show that the proposed methods can provide a 46%-58% reduction in training time compared to existing approaches in reaching the same accuracy.
|Original language||English (US)|
|Title of host publication||EdgeSys 2021 - Proceedings of the 4th International Workshop on Edge Systems, Analytics and Networking, Part of EuroSys 2021|
|Publisher||Association for Computing Machinery, Inc|
|Number of pages||6|
|State||Published - Apr 26 2021|
|Event||4th International Workshop on Edge Systems, Analytics and Networking, EdgeSys 2021, in conjunction with ACM EuroSys 2021 - Virtual, Online, United Kingdom|
Duration: Apr 26 2021 → …
|Name||EdgeSys 2021 - Proceedings of the 4th International Workshop on Edge Systems, Analytics and Networking, Part of EuroSys 2021|
|Conference||4th International Workshop on Edge Systems, Analytics and Networking, EdgeSys 2021, in conjunction with ACM EuroSys 2021|
|Period||4/26/21 → …|
Bibliographical noteFunding Information:
This research was supported in part by the NSF under grant CNS-1717834.
© 2021 ACM.