Observing the web by understanding the past: Archival internet research

Matthew S. Weber

doi:10.1145/2567948.2579213

Observing the web by understanding the past: Archival internet research

Matthew S. Weber

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Scopus citations

Abstract

This paper discusses the challenges and opportunities for using archival Internet data in order to observe a host of social science phenomena. Specifically, this paper introduces HistoryTracker, a new tool for accessing and extracting archived data from the Internet Archive, the largest repository of archived Web data in existence. The HistoryTracker tool serves to create a Web observatory that allows scholars to study the history of the Web. HistoryTracker takes advantages of Hadoop processing capacity, and allows researchers to extract large swaths of archived data into a link list format that can be easily transferred to a number of other analytical tools. A brief illustration of the use of HistoryTracker is presented demonstrating the use of the tool. Finally, a number of continuing research challenges are discussed, and future research opportunities are outlined.

Original language	English (US)
Title of host publication	WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web
Publisher	Association for Computing Machinery, Inc
Pages	1031-1036
Number of pages	6
ISBN (Electronic)	9781450327459
DOIs	https://doi.org/10.1145/2567948.2579213
State	Published - Apr 7 2014
Externally published	Yes
Event	23rd International Conference on World Wide Web, WWW 2014 - Seoul, Korea, Republic of Duration: Apr 7 2014 → Apr 11 2014

Publication series

Name	WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web

Other

Other	23rd International Conference on World Wide Web, WWW 2014
Country/Territory	Korea, Republic of
City	Seoul
Period	4/7/14 → 4/11/14

Bibliographical note

Funding Information:
The author acknowledges support from the National Science Foundation (NSF Award 1244727), as well as the support of a number of collaborators including Kris Carpenter, David Lazer, Katherine Ognyanova, Vinay Goel, Luan Nguyen, Hai Nguyen and Allie Kosterich.

Publisher Copyright:
© Copyright 2014 by the International World Wide Web Conferences Steering Committee.

Keywords

Archived data
Data extraction
Network analysis
Occupy wall street
Social sciences
Web observatory

Access

10.1145/2567948.2579213

Cite this

Weber, M. S. (2014). Observing the web by understanding the past: Archival internet research. In WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web (pp. 1031-1036). (WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web). Association for Computing Machinery, Inc. https://doi.org/10.1145/2567948.2579213

Observing the web by understanding the past : Archival internet research. / Weber, Matthew S.

WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web. Association for Computing Machinery, Inc, 2014. p. 1031-1036 (WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Weber, MS 2014, Observing the web by understanding the past: Archival internet research. in WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web. WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web, Association for Computing Machinery, Inc, pp. 1031-1036, 23rd International Conference on World Wide Web, WWW 2014, Seoul, Korea, Republic of, 4/7/14. https://doi.org/10.1145/2567948.2579213

@inproceedings{ea23657cea9c4de08bab35ef8b6f9381,

title = "Observing the web by understanding the past: Archival internet research",

abstract = "This paper discusses the challenges and opportunities for using archival Internet data in order to observe a host of social science phenomena. Specifically, this paper introduces HistoryTracker, a new tool for accessing and extracting archived data from the Internet Archive, the largest repository of archived Web data in existence. The HistoryTracker tool serves to create a Web observatory that allows scholars to study the history of the Web. HistoryTracker takes advantages of Hadoop processing capacity, and allows researchers to extract large swaths of archived data into a link list format that can be easily transferred to a number of other analytical tools. A brief illustration of the use of HistoryTracker is presented demonstrating the use of the tool. Finally, a number of continuing research challenges are discussed, and future research opportunities are outlined.",

keywords = "Archived data, Data extraction, Network analysis, Occupy wall street, Social sciences, Web observatory",

author = "Weber, {Matthew S.}",

note = "Funding Information: The author acknowledges support from the National Science Foundation (NSF Award 1244727), as well as the support of a number of collaborators including Kris Carpenter, David Lazer, Katherine Ognyanova, Vinay Goel, Luan Nguyen, Hai Nguyen and Allie Kosterich. Publisher Copyright: {\textcopyright} Copyright 2014 by the International World Wide Web Conferences Steering Committee.; 23rd International Conference on World Wide Web, WWW 2014 ; Conference date: 07-04-2014 Through 11-04-2014",

year = "2014",

month = apr,

day = "7",

doi = "10.1145/2567948.2579213",

language = "English (US)",

series = "WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web",

publisher = "Association for Computing Machinery, Inc",

pages = "1031--1036",

booktitle = "WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web",

}

TY - GEN

T1 - Observing the web by understanding the past

T2 - 23rd International Conference on World Wide Web, WWW 2014

AU - Weber, Matthew S.

N1 - Funding Information: The author acknowledges support from the National Science Foundation (NSF Award 1244727), as well as the support of a number of collaborators including Kris Carpenter, David Lazer, Katherine Ognyanova, Vinay Goel, Luan Nguyen, Hai Nguyen and Allie Kosterich. Publisher Copyright: © Copyright 2014 by the International World Wide Web Conferences Steering Committee.

PY - 2014/4/7

Y1 - 2014/4/7

N2 - This paper discusses the challenges and opportunities for using archival Internet data in order to observe a host of social science phenomena. Specifically, this paper introduces HistoryTracker, a new tool for accessing and extracting archived data from the Internet Archive, the largest repository of archived Web data in existence. The HistoryTracker tool serves to create a Web observatory that allows scholars to study the history of the Web. HistoryTracker takes advantages of Hadoop processing capacity, and allows researchers to extract large swaths of archived data into a link list format that can be easily transferred to a number of other analytical tools. A brief illustration of the use of HistoryTracker is presented demonstrating the use of the tool. Finally, a number of continuing research challenges are discussed, and future research opportunities are outlined.

AB - This paper discusses the challenges and opportunities for using archival Internet data in order to observe a host of social science phenomena. Specifically, this paper introduces HistoryTracker, a new tool for accessing and extracting archived data from the Internet Archive, the largest repository of archived Web data in existence. The HistoryTracker tool serves to create a Web observatory that allows scholars to study the history of the Web. HistoryTracker takes advantages of Hadoop processing capacity, and allows researchers to extract large swaths of archived data into a link list format that can be easily transferred to a number of other analytical tools. A brief illustration of the use of HistoryTracker is presented demonstrating the use of the tool. Finally, a number of continuing research challenges are discussed, and future research opportunities are outlined.

KW - Archived data

KW - Data extraction

KW - Network analysis

KW - Occupy wall street

KW - Social sciences

KW - Web observatory

UR - http://www.scopus.com/inward/record.url?scp=84990902037&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84990902037&partnerID=8YFLogxK

U2 - 10.1145/2567948.2579213

DO - 10.1145/2567948.2579213

M3 - Conference contribution

AN - SCOPUS:84990902037

T3 - WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web

SP - 1031

EP - 1036

BT - WWW 2014 Companion - Proceedings of the 23rd International Conference on World Wide Web

PB - Association for Computing Machinery, Inc

Y2 - 7 April 2014 through 11 April 2014

ER -

Observing the web by understanding the past: Archival internet research

Abstract

Publication series

Other

Bibliographical note

Keywords

Access

Other files and links

Fingerprint

Cite this