Large-scale neural modeling in MapReduce and Giraph

Shuo Yang, Nicholas D. Spielman, Jadin C. Jackson, Brad S. Rubin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations


One of the most crucial challenges in scientific computing is scalability. Hadoop, an open-source implementation of the MapReduce parallel programming model developed by Google, has emerged as a powerful platform for performing large-scale scientific computing at very low costs. In this paper, we explore the use of Hadoop to model large-scale neural networks. A neural network is most naturally modeled by a graph structure with iterative processing. In this paper, we first present an improved graph algorithm design pattern in MapReduce called Mapper-side Schimmy. Experiments show that the application of our design pattern, combined with the current best practices, can reduce the running time of the neural network simulation on a neural network with 100,000 neurons and 2.3 billion edges by 64%. MapReduce, however, is inherently not efficient for iterative graph processing. To address the limitation of the MapReduce model, we then explore the use of Giraph, an open source large-scale graph processing framework that sits on top of Hadoop to implement graph algorithms with a vertex-centric approach. We show that our Giraph implementation boosted performance by 91% compared to a basic MapReduce implementation and by 60% compared to our improved Mapper-side Schimmy algorithm.

Original languageEnglish (US)
Title of host publication2014 IEEE International Conference on Electro/Information Technology, EIT 2014
PublisherIEEE Computer Society
Number of pages6
ISBN (Print)9781479947744
StatePublished - 2014
Event2014 IEEE International Conference on Electro/Information Technology, EIT 2014 - Milwaukee, WI, United States
Duration: Jun 5 2014Jun 7 2014

Publication series

NameIEEE International Conference on Electro Information Technology
ISSN (Print)2154-0357
ISSN (Electronic)2154-0373


Other2014 IEEE International Conference on Electro/Information Technology, EIT 2014
CountryUnited States
CityMilwaukee, WI


Dive into the research topics of 'Large-scale neural modeling in MapReduce and Giraph'. Together they form a unique fingerprint.

Cite this