Communication-Efficient distributed optimization for sparse learning via two-way truncation

We propose a communication and computation efficient algorithm for high-dimensional distributed sparse learning, motivated by the approach of (Wang et al., 2016). At each iteration, local machines compute local gradients on their own local data and using these, a master machine solves a shifted l\ regularized minimization problem. Here, our contribution reduces the communication cost per transmission from the order of the parameter dimension to the order of the number of nonzero entries in the parameter via a Two-Way Truncation procedure. Theoretically, we prove that the estimation error of the proposed algorithm decreases exponentially and matches that of the centralized method under mild conditions. Extensive experiments on both simulated data and real data support that the proposed algorithm is efficient and has statistical performance comparable with the centralized method.