Eliminating the middleman: Peer-to-peer dataflow

Adam Barker, Jon B. Weissman, Jano Van Hemert

Research output: Chapter in Book/Report/Conference proceedingConference contribution

22 Scopus citations

Abstract

Efficiently executing large-scale, data-intensive workflows such as Montage must take into account the volume and pattern of communication. When orchestrating data-centric workflows, centralised servers common to standard workflow systems can become a bottleneck to performance. However, standards-based workflow systems that rely on centralisation, e.g., Web service based frameworks, have many other benefits such as a wide user base and sustained support. This paper presents and evaluates a light-weight hybrid architecture which maintains the robustness and simplicity of centralised orchestration, but facilitates choreography by allowing services to exchange data directly with one another. Furthermore our architecture is standards compliment, flexible and is a non-disruptive solution; service definitions do not have to be altered prior to enactment. Our architecture could be realised within any existing workflow framework, in this paper, we focus on a Web service based framework. Taking inspiration from Montage, a number of common workflow patterns (sequence, fan-in and fan-out), input to output data size relationships and network configurations are identified and evaluated. The performance analysis concludes that a substantial reduction in communication overhead results in a 2-4 fold performance benefit across all patterns. An end-to-end pattern through the Montage workflow results in an 8 fold performance benefit and demonstrates how the advantage of using our hybrid architecture increases as the complexity of a workflow grows.

Original languageEnglish (US)
Title of host publicationProceedings of the 17th International Symposium on High Performance Distributed Computing 2008, HPDC'08
Pages55-64
Number of pages10
DOIs
StatePublished - Dec 12 2008
Event17th International Symposium on High Performance Distributed Computing 2008, HPDC'08 - Boston, MA, United States
Duration: Jun 23 2008Jun 27 2008

Publication series

NameProceedings of the 17th International Symposium on High Performance Distributed Computing 2008, HPDC'08

Other

Other17th International Symposium on High Performance Distributed Computing 2008, HPDC'08
CountryUnited States
CityBoston, MA
Period6/23/086/27/08

Keywords

  • Decentralised orchestration
  • Workflow optimisation

Fingerprint Dive into the research topics of 'Eliminating the middleman: Peer-to-peer dataflow'. Together they form a unique fingerprint.

Cite this