Keeping Community in the Loop: Understanding Wikipedia Stakeholder Values for Machine Learning-Based Systems

C. Estelle Smith, Bowen Yu, Anjali Srivastava, Aaron L Halfaker, Loren Terveen, Haiyi Zhu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

On Wikipedia, sophisticated algorithmic tools are used to assess the quality of edits and take corrective actions. However, algorithms can fail to solve the problems they were designed for if they conflict with the values of communities who use them. In this study, we take a Value-Sensitive Algorithm Design approach to understanding a community-created and -maintained machine learning-based algorithm called the Objective Revision Evaluation System (ORES) - -A quality prediction system used in numerous Wikipedia applications and contexts. Five major values converged across stakeholder groups that ORES (and its dependent applications) should: (1) reduce the effort of community maintenance, (2) maintain human judgement as the final authority, (3) support differing peoples' differing workflows, (4) encourage positive engagement with diverse editor groups, and (5) establish trustworthiness of people and algorithms within the community. We reveal tensions between these values and discuss implications for future research to improve algorithms like ORES.

Original languageEnglish (US)
Title of host publicationCHI 2020 - Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450367080
DOIs
StatePublished - Apr 21 2020
Event2020 ACM CHI Conference on Human Factors in Computing Systems, CHI 2020 - Honolulu, United States
Duration: Apr 25 2020Apr 30 2020

Publication series

NameConference on Human Factors in Computing Systems - Proceedings

Conference

Conference2020 ACM CHI Conference on Human Factors in Computing Systems, CHI 2020
Country/TerritoryUnited States
CityHonolulu
Period4/25/204/30/20

Bibliographical note

Funding Information:
We thank our anonymous reviewers, Zhiwei Steven Wu, colleagues at GroupLens Research at the University of Minnesota, and the HCI Institute at Carnegie Mellon University for their feedback. This work was supported by the National Science Foundation (NSF) under Award No. IIS-2001851 and IIS-2000782, and the NSF Program on Fairness in AI in collaboration with Amazon under Award No. IIS-1939606.

Publisher Copyright:
© 2020 Owner/Author.

Keywords

  • ORES
  • community values
  • machine learning
  • peer production
  • value sensitive algorithm design
  • wikipedia

Fingerprint

Dive into the research topics of 'Keeping Community in the Loop: Understanding Wikipedia Stakeholder Values for Machine Learning-Based Systems'. Together they form a unique fingerprint.

Cite this