GNSD: A Gradient-Tracking Based Nonconvex Stochastic Algorithm for Decentralized Optimization

Songtao Lu, Xinwei Zhang, Haoran Sun, Mingyi Hong

Research output: Chapter in Book/Report/Conference proceedingConference contribution

13 Scopus citations

Abstract

In the era of big data, it is challenging to train a machine learning model on a single machine or over a distributed system with a central controller over a large-scale dataset. In this paper, we propose a gradient-tracking based nonconvex stochastic decentralized (GNSD) algorithm for solving nonconvex optimization problems, where the data is partitioned into multiple parts and processed by the local computational resource. Through exchanging the parameters at each node over a network, GNSD is able to find the first-order stationary points (FOSP) efficiently. From the theoretical analysis, it is guaranteed that the convergence rate of GNSD to FOSPs matches the well-known convergence rate \mathcal{O}\left( {1/\sqrt T } \right) of stochastic gradient descent by shrinking the step-size. Finally, we perform extensive numerical experiments on computational clusters to demonstrate the advantage of GNSD compared with other state-of-the-art methods.

Original languageEnglish (US)
Title of host publication2019 IEEE Data Science Workshop, DSW 2019 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages315-321
Number of pages7
ISBN (Electronic)9781728107080
DOIs
StatePublished - Jun 2019
Event2019 IEEE Data Science Workshop, DSW 2019 - Minneapolis, United States
Duration: Jun 2 2019Jun 5 2019

Publication series

Name2019 IEEE Data Science Workshop, DSW 2019 - Proceedings

Conference

Conference2019 IEEE Data Science Workshop, DSW 2019
CountryUnited States
CityMinneapolis
Period6/2/196/5/19

Bibliographical note

Funding Information:
†equal contribution. The authors of this paper have been supported by NSF grants CMMI-1727757, CCF-1526078, an AFOSR grant 15RT0767, and a Digital TechnologyInitiative Seed Grant from the Digital Technology Center at University of Minnesota.

Keywords

  • Stochastic
  • decentralized
  • gradient tracking
  • neural networks
  • nonconvex optimization

Fingerprint Dive into the research topics of 'GNSD: A Gradient-Tracking Based Nonconvex Stochastic Algorithm for Decentralized Optimization'. Together they form a unique fingerprint.

Cite this