The relationship between nonresponse strategies and measurement error: Comparing online panel surveys to traditional surveys

One of the most daunting challenges facing survey researchers is ensuring high response rates, which have been declining due to societal, political, and technological changes. This chapter analyzes whether investing resources into successfully interviewing difficult-to-obtain and reluctant respondents via nonresponse strategies (e.g., multiple call-back attempts and refusal conversions)-thereby reducing nonresponse bias-is counterproductive because the respondents obtained via these methods may not be fully engaged with the survey, and may therefore be inclined to satisfice. As such, nonresponse strategies may bring respondents into the sample who introduce more systematic and/or random measurement error than their easier-to-reach counterparts. We examine the relationship between reluctance and satisficing in three Internet surveys, and then examine whether survey mode moderates the impact of reluctance on satisficing by comparing Internet, telephone, and face-to-face surveys. We find that difficult-to-obtain Internet respondents are more likely to indicate a DK/NO response and are more likely to select the first reasonable response than their easy-to-obtain counterparts. In contrast, across the five indicators of satisficing examined in the face-to-face and phone modes (midpoint responding, DK/NO responding, non-differentiation, mental coin-flipping, and selecting the first reasonable response), the only one for which difficult-to-reach respondents were more likely to evidence satisficing (compared to their easy-to-reach counterparts) was middle responding (in the phone mode). The difference in effects between the Internet and phone mode varies in a theoretically predictable manner. Whereas reluctant respondents are more likely to select the middle option in telephone surveys when an interviewer is present (thus responding without expending much cognitive effort while maintaining the façade of compliance or competence), they are more likely to provide "don't know" responses when interviewed over the Internet (when the lack of an interviewer reduced concerns about appearing compliant or competent). Given that some forms of satisficing occurred more among reluctant respondents in the Internet mode, these new forms of survey interviewing present future challenges.