BDS: A Centralized Near-Optimal Overlay Network for Inter-Datacenter Data Replication

Yuchao Zhang; Junchen Jiang; Ke Xu; Xiaohui Nie; Martin J. Reed; Haiyang Wang; Guang Yao; Miao Zhang; Kai Chen

doi:10.1145/3190508.3190519

BDS: A Centralized Near-Optimal Overlay Network for Inter-Datacenter Data Replication

Yuchao Zhang, Junchen Jiang, Ke Xu, Xiaohui Nie, Martin J. Reed, Haiyang Wang, Guang Yao, Miao Zhang, Kai Chen

Computer Science (Duluth)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

22 Scopus citations

Abstract

Many important cloud services require replicating massive data from one datacenter (DC) to multiple DCs. While the performance of pair-wise inter-DC data transfers has been much improved, prior solutions are insufficient to optimize bulk-data multicast, as they fail to explore the capability of servers to store-and-forward data, as well as the rich inter-DC overlay paths that exist in geo-distributed DCs. To take advantage of these opportunities, we present BDS, an application-level multicast overlay network for large-scale inter-DC data replication. At the core of BDS is a fully centralized architecture, allowing a central controller to maintain an up-to-date global view of data delivery status of intermediate servers, in order to fully utilize the available overlay paths. To quickly react to network dynamics and workload churns, BDS speeds up the control algorithm by decoupling it into selection of overlay paths and scheduling of data transfers, each can be optimized efficiently. This enables BDS to update overlay routing decisions in near realtime (e.g., every other second) at the scale of multicasting hundreds of TB data over tens of thousands of overlay paths. A pilot deployment in one of the largest online service providers shows that BDS can achieve 3-5× speedup over the provider’s existing system and several well-known overlay routing baselines.

Original language	English (US)
Title of host publication	Proceedings of the 13th EuroSys Conference, EuroSys 2018
Publisher	Association for Computing Machinery, Inc
ISBN (Electronic)	9781450355841
DOIs	https://doi.org/10.1145/3190508.3190519
State	Published - Apr 23 2018
Event	13th EuroSys Conference, EuroSys 2018 - Porto, Portugal Duration: Apr 23 2018 → Apr 26 2018

Publication series

Name	Proceedings of the 13th EuroSys Conference, EuroSys 2018
Volume	2018-January

Conference

Conference	13th EuroSys Conference, EuroSys 2018
Country/Territory	Portugal
City	Porto
Period	4/23/18 → 4/26/18

Bibliographical note

Funding Information:
This work is supported in part by the HK RGC ECS-26200014, CRF-C703615G, HKUST PDF fund, the China 973 Program (2014CB340300), the National Natural Foundation of China (61472212), EU Marie Curie Actions CROWN (FP7-PEOPLE-2013-IRSES-610524). We would like to thank our shepherd, Paolo Romano, and the anonymous EuroSys reviewers for their valuable feedback.

Publisher Copyright:
© 2018 Association for Computing Machinery.

Access

10.1145/3190508.3190519

OpenUrl availability

Full text

Cite this

Zhang, Y., Jiang, J., Xu, K., Nie, X., Reed, M. J., Wang, H., Yao, G., Zhang, M., & Chen, K. (2018). BDS: A Centralized Near-Optimal Overlay Network for Inter-Datacenter Data Replication. In Proceedings of the 13th EuroSys Conference, EuroSys 2018 (Proceedings of the 13th EuroSys Conference, EuroSys 2018; Vol. 2018-January). Association for Computing Machinery, Inc. https://doi.org/10.1145/3190508.3190519

BDS: A Centralized Near-Optimal Overlay Network for Inter-Datacenter Data Replication. / Zhang, Yuchao; Jiang, Junchen; Xu, Ke et al.
Proceedings of the 13th EuroSys Conference, EuroSys 2018. Association for Computing Machinery, Inc, 2018. (Proceedings of the 13th EuroSys Conference, EuroSys 2018; Vol. 2018-January).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Zhang, Y, Jiang, J, Xu, K, Nie, X, Reed, MJ, Wang, H, Yao, G, Zhang, M & Chen, K 2018, BDS: A Centralized Near-Optimal Overlay Network for Inter-Datacenter Data Replication. in Proceedings of the 13th EuroSys Conference, EuroSys 2018. Proceedings of the 13th EuroSys Conference, EuroSys 2018, vol. 2018-January, Association for Computing Machinery, Inc, 13th EuroSys Conference, EuroSys 2018, Porto, Portugal, 4/23/18. https://doi.org/10.1145/3190508.3190519

@inproceedings{abf1db0f46e74e8187cd38ef2ce8cb0b,

title = "BDS: A Centralized Near-Optimal Overlay Network for Inter-Datacenter Data Replication",

abstract = "Many important cloud services require replicating massive data from one datacenter (DC) to multiple DCs. While the performance of pair-wise inter-DC data transfers has been much improved, prior solutions are insufficient to optimize bulk-data multicast, as they fail to explore the capability of servers to store-and-forward data, as well as the rich inter-DC overlay paths that exist in geo-distributed DCs. To take advantage of these opportunities, we present BDS, an application-level multicast overlay network for large-scale inter-DC data replication. At the core of BDS is a fully centralized architecture, allowing a central controller to maintain an up-to-date global view of data delivery status of intermediate servers, in order to fully utilize the available overlay paths. To quickly react to network dynamics and workload churns, BDS speeds up the control algorithm by decoupling it into selection of overlay paths and scheduling of data transfers, each can be optimized efficiently. This enables BDS to update overlay routing decisions in near realtime (e.g., every other second) at the scale of multicasting hundreds of TB data over tens of thousands of overlay paths. A pilot deployment in one of the largest online service providers shows that BDS can achieve 3-5× speedup over the provider{\textquoteright}s existing system and several well-known overlay routing baselines.",

author = "Yuchao Zhang and Junchen Jiang and Ke Xu and Xiaohui Nie and Reed, {Martin J.} and Haiyang Wang and Guang Yao and Miao Zhang and Kai Chen",

note = "Funding Information: This work is supported in part by the HK RGC ECS-26200014, CRF-C703615G, HKUST PDF fund, the China 973 Program (2014CB340300), the National Natural Foundation of China (61472212), EU Marie Curie Actions CROWN (FP7-PEOPLE-2013-IRSES-610524). We would like to thank our shepherd, Paolo Romano, and the anonymous EuroSys reviewers for their valuable feedback. Publisher Copyright: {\textcopyright} 2018 Association for Computing Machinery.; 13th EuroSys Conference, EuroSys 2018 ; Conference date: 23-04-2018 Through 26-04-2018",

year = "2018",

month = apr,

day = "23",

doi = "10.1145/3190508.3190519",

language = "English (US)",

series = "Proceedings of the 13th EuroSys Conference, EuroSys 2018",

publisher = "Association for Computing Machinery, Inc",

booktitle = "Proceedings of the 13th EuroSys Conference, EuroSys 2018",

}

TY - GEN

T1 - BDS

T2 - 13th EuroSys Conference, EuroSys 2018

AU - Zhang, Yuchao

AU - Jiang, Junchen

AU - Xu, Ke

AU - Nie, Xiaohui

AU - Reed, Martin J.

AU - Wang, Haiyang

AU - Yao, Guang

AU - Zhang, Miao

AU - Chen, Kai

N1 - Funding Information: This work is supported in part by the HK RGC ECS-26200014, CRF-C703615G, HKUST PDF fund, the China 973 Program (2014CB340300), the National Natural Foundation of China (61472212), EU Marie Curie Actions CROWN (FP7-PEOPLE-2013-IRSES-610524). We would like to thank our shepherd, Paolo Romano, and the anonymous EuroSys reviewers for their valuable feedback. Publisher Copyright: © 2018 Association for Computing Machinery.

PY - 2018/4/23

Y1 - 2018/4/23

N2 - Many important cloud services require replicating massive data from one datacenter (DC) to multiple DCs. While the performance of pair-wise inter-DC data transfers has been much improved, prior solutions are insufficient to optimize bulk-data multicast, as they fail to explore the capability of servers to store-and-forward data, as well as the rich inter-DC overlay paths that exist in geo-distributed DCs. To take advantage of these opportunities, we present BDS, an application-level multicast overlay network for large-scale inter-DC data replication. At the core of BDS is a fully centralized architecture, allowing a central controller to maintain an up-to-date global view of data delivery status of intermediate servers, in order to fully utilize the available overlay paths. To quickly react to network dynamics and workload churns, BDS speeds up the control algorithm by decoupling it into selection of overlay paths and scheduling of data transfers, each can be optimized efficiently. This enables BDS to update overlay routing decisions in near realtime (e.g., every other second) at the scale of multicasting hundreds of TB data over tens of thousands of overlay paths. A pilot deployment in one of the largest online service providers shows that BDS can achieve 3-5× speedup over the provider’s existing system and several well-known overlay routing baselines.

AB - Many important cloud services require replicating massive data from one datacenter (DC) to multiple DCs. While the performance of pair-wise inter-DC data transfers has been much improved, prior solutions are insufficient to optimize bulk-data multicast, as they fail to explore the capability of servers to store-and-forward data, as well as the rich inter-DC overlay paths that exist in geo-distributed DCs. To take advantage of these opportunities, we present BDS, an application-level multicast overlay network for large-scale inter-DC data replication. At the core of BDS is a fully centralized architecture, allowing a central controller to maintain an up-to-date global view of data delivery status of intermediate servers, in order to fully utilize the available overlay paths. To quickly react to network dynamics and workload churns, BDS speeds up the control algorithm by decoupling it into selection of overlay paths and scheduling of data transfers, each can be optimized efficiently. This enables BDS to update overlay routing decisions in near realtime (e.g., every other second) at the scale of multicasting hundreds of TB data over tens of thousands of overlay paths. A pilot deployment in one of the largest online service providers shows that BDS can achieve 3-5× speedup over the provider’s existing system and several well-known overlay routing baselines.

UR - http://www.scopus.com/inward/record.url?scp=85050862389&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050862389&partnerID=8YFLogxK

U2 - 10.1145/3190508.3190519

DO - 10.1145/3190508.3190519

M3 - Conference contribution

AN - SCOPUS:85050862389

T3 - Proceedings of the 13th EuroSys Conference, EuroSys 2018

BT - Proceedings of the 13th EuroSys Conference, EuroSys 2018

PB - Association for Computing Machinery, Inc

Y2 - 23 April 2018 through 26 April 2018

ER -

BDS: A Centralized Near-Optimal Overlay Network for Inter-Datacenter Data Replication

Abstract

Publication series

Conference

Bibliographical note

Access

OpenUrl availability

Other files and links

Fingerprint

Cite this