Total Survey Error in Practice

Featuring a timely presentation of total survey error (TSE), this edited volume introduces valuable tools for understanding and improving survey data quality in the context of evolving large–scale data sets

This book provides an overview of the TSE framework and current TSE research as related to survey design, data collection, estimation, and analysis. It recognizes that survey data affects many public policy and business decisions and thus focuses on the framework for understanding and improving survey data quality. The book also addresses issues with data quality in official statistics and in social, opinion, and market research as these fields continue to evolve, leading to larger and messier data sets. This perspective challenges survey organizations to find ways to collect and process data more efficiently without sacrificing quality. The volume consists of the most up–to–date research and reporting from over 70 contributors representing the best academics and researchers from a range of fields. The chapters are broken out into five main sections: The Concept of TSE and the TSE Paradigm, Implications for Survey Design, Data Collection and Data Processing Applications, Evaluation and Improvement, and Estimation and Analysis. Each chapter introduces and examines multiple error sources, such as sampling error, measurement error, and nonresponse error, which often offer the greatest risks to data quality, while also encouraging readers not to lose sight of the less commonly studied error sources, such as coverage error, processing error, and specification error. The book also notes the relationships between errors and the ways in which efforts to reduce one type can increase another, resulting in an estimate with larger total error.

This book:

Features various error sources, and the complex relationships between them, in 25 high–quality chapters on the most up–to–date research in the field of TSE

Provides comprehensive reviews of the literature on error sources as well as data collection approaches and estimation methods to reduce their effects

Presents examples of recent international events that demonstrate the effects of data error, the importance of survey data quality, and the real–world issues that arise from these errors

Spans the four pillars of the total survey error paradigm (design, data collection, evaluation and analysis) to address key data quality issues in official statistics and survey research

Total Survey Error in Practice is a reference for survey researchers and data scientists in research areas that include social science, public opinion, public policy, and business. It can also be used as a textbook or supplementary material for a graduate–level course in survey research methods.

Paul P. Biemer, PhD, is distinguished fellow at RTI International and associate director of Survey Research and Development at the Odum Institute, University of North Carolina, USA.

Edith de Leeuw, PhD, is professor of survey methodology in the Department of Methodology and Statistics at Utrecht University, the Netherlands.

Stephanie Eckman, PhD, is fellow at RTI International, USA.

Brad Edwards is vice president, director of Field Services, and deputy area director at Westat, USA.

Frauke Kreuter, PhD, is professor and director of the Joint Program in Survey Methodology, University of Maryland, USA; professor of statistics and methodology at the University of Mannheim, Germany; and head of the Statistical Methods Research Department at the Institute for Employment Research, Germany.

Lars E. Lyberg, PhD, is senior advisor at Inizio, Sweden.

N. Clyde Tucker, PhD, is principal survey methodologist at the American Institutes for Research, USA.

Brady T. West, PhD, is research associate professor in the Survey Research Center, located within the Institute for Social Research at the University of Michigan (U–M), and also serves as statistical consultant on the Consulting for Statistics, Computing and Analytics Research (CSCAR) team at U–M, USA.

Contributors

Preface

SECTION 1 – The Concept of TSE and the TSE Paradigm

1. The Roots and Evolution of the Total Survey Error Concept
Lars Lyberg and Diana Stukel

1.1 Introduction and historical backdrop

1.2 Specific error sources and their control

1.3 Survey models and total survey design

1.4 The advent of more systematic approaches toward survey quality

1.5 What the future will bring

References

2. Total Twitter Error: Public Opinion Measurement on Twitter from a Total Survey Error Perspective
Patrick Hsieh and Joe Murphy

2.1 Introduction

2.2 Social media: an evolving online public sphere

2.3 Components of Twitter error

2.4 Studying public opinion on the Twittersphere and the potential error sources of Twitter data: two case studies

2.5 Discussion

2.6 Conclusion

References

3. Big Data: A Survey Research Perspective
Reg Baker

3.1 Introduction

3.2 Definitions

3.3 The analytic challenge: From database marketing to Big Data and data science

3.4 Assessing data quality

3.5 Applications in market, opinion, and social research

3.6 The ethics of research using Big Data

3.7 The future of surveys in a data rich environment

References

4. The Role of Statistical Disclosure Limitation in Total Survey Error
Alan Karr

4.1 Introduction

4.2 Primer on SDL

4.3 TSE–aware SDL

4.4 Edit–Respecting SDL

4.5 SDL–aware TSE

4.6 Full unification of edit, imputation and SDL

4.7 Big Data issues

4.8 Conclusion

Acknowledgements

References

SECTION 2 – Implications for Survey Design

5. The Undercoverage–Nonresponse Tradeoff
Stephanie Eckman and Frauke Kreuter

5.1 Introduction

5.2 Examples of the tradeoff

5.3 Simple demonstration of the tradeoff

5.4 Coverage and response propensities and bias

5.5 Simulation study of rates and bias

5.6 Costs

5.7 Lessons for survey practice

References

6. Mixing Modes: Tradeoffs among Coverage, Nonresponse, and Measurement Error
Roger Tourangeau

6.1 Introduction

6.2 The effect of offering a choice of modes

6.3 Getting people to respond online

6.4 Sequencing different modes of data collection

6.5 Separating the effects of mode on selection and reporting

6.6 Maximizing comparability versus minimizing error

6.7 Conclusions

References

7. Mobile Web Surveys: A Total Survey Error Perspective
Mick P. Couper, Christopher Antoun, and Aigul Mavletova

7.1 Introduction

7.2 Coverage

7.3 Nonresponse

7.4 Measurement error

7.5 Links between different error sources

7.6 The future of mobile web surveys

References

8. The Effects of a Mid–Data Collection Change in Financial Incentives on Total Survey Error in the National Survey of Family Growth: Results from a Randomized Experiment
James Wagner, Brady T. West, Heidi Guyer, Paul Burton, Jennifer Kelley, Mick P. Couper, and William D. Mosher

8.1 Introduction

8.2 Literature review: Incentives in face–to–face surveys

8.3 Data and methods

8.4 Results

8.5 Conclusion

References

9. A Total Survey Error Perspective on Surveys in Multinational, Multiregional, and Multicultural Contexts
Beth–Ellen Pennell, Kristen Cibelli Hibben, Lars Lyberg, Peter Ph. Mohler, and Gelaye Worku

9.1 Introduction

9.2 Total survey error in multinational, multiregional, and multicultural surveys

9.3 Challenges related to representation and measurement components in comparative surveys

9.4 Quality assurance and quality control in 3MC surveys

References

10. Smartphone Participation in Web surveys: Choosing between the Potential for Coverage, Nonresponse, and Measurement Error
Gregg Peterson, Jamie Griffin, John LaFrance, and JiaoJiao Li

10.1 Introduction

10.2 Prevalence of smartphone participation in web surveys

10.3 Smartphone participation choices

10.4 Instrument design choices

10.5 device and design treatment choices

10.6 Conclusion

10.7 Future challenges and research needs

References

Survey Research and the Quality of Survey Data among Ethnic Minorities
Joost Kappelhof

11.1 Introduction

11.2 On the use of the terms ethnicity and ethnic minorities

11.3 On the representation of ethnic minorities in surveys

11.4 Measurement issues

11.5 Comparability, timeliness and cost concerns

11.6 Conclusions

References

SECTION 3 – Data Collection and Data Processing Applications

12. Measurement Error in Survey Operations: Detection, Quantification, Visualization, and Reduction
Brad Edwards, Aaron Maitland, and Sue Connor

12.1 TSE background on survey operations

12.2 Better and better: Using behavior coding (CARICODE) and paradata to evaluate and improve question (specification) error and interviewer error

12.3 Field–centered design: Mobile app for rapid reporting and management

12.4 Faster and cheaper: Detecting falsification with GIS tools

12.5 Putting it all together: Field supervisor dashboards

12.6 Discussion

References

13. Total Survey Error for Longitudinal Surveys
Peter Lynn and Peter J. Lugtig

13.1 Introduction

13.2 Distinctive aspects of longitudinal surveys

13.3 Total survey error components in longitudinal surveys

13.4 Design of longitudinal surveys from a Total Survey Error perspective

13.5 Examples of trade–offs in three longitudinal surveys

13.6 Discussion

References

14. Text Interviews on Mobile Devices
Frederick G. Conrad, Michael F. Schober, Christopher Antoun, Andrew L. Hupp, and H. Yanna Yan

14.1 Texting as a way of interacting

14.2 Contacting and inviting potential respondents through text

14.3 Texting as an interview mode

14.4 Costs and efficiency of text interviewing

14.5 Discussion

References

15. Quantifying Measurement Errors in Partially Edited Business Survey Data
Thomas Laitila, Karin Lindgren, Anders Norberg, and Can Tongur

15.1 Introduction

15.2 Selective editing

15.3 Effects of errors remaining after SE

15.4 Case study: Foreign trade in goods within the European Union

15.5 Editing Big Data

15.6 Conclusions

References

SECTION 4 – Evaluation and Improvement

16. Estimating Error Rates in an Administrative Register and Survey Questions Using a Latent Class Model
Daniel Oberski

16.1 Introduction

16.2 Administrative and survey measures of neighborhood

16.3 A latent class model for neighborhood of residence

16.4 Results

16.5 Discussion and Conclusion

References

17. ASPIRE – An Approach for Evaluating and Reducing the Total Error in Statistical Products with Application to Registers and the National Accounts
Paul Biemer, Dennis Trewin, Heather Bergdahl, and Xie Yingfu

17.1 Introduction and background

17.2 Overview of ASPIRE

17.3 The ASPIRE model

17.4 Evaluation of registers

17.5 National accounts

17.6 A sensitivity analysis of GDP error sources

17.7 Concluding remarks

References

Appendix 17.A

18. Classification Error in Crime Victimization Surveys: A Markov Latent Class Analysis
Marcus Berzofsky and Paul Biemer

18.1. Introduction

18.2 Background

18.3 Analytic approach

18.4 Model selection

18.5 Results

18.6 Discussion and summary of findings

18.7 Conclusions

Appendix 18.1

Appendix 18.2

Appendix 18.3

References

19. Using Doorstep Concerns Data to Evaluate and Correct for Nonresponse Error in Longitudinal Survey
Ting Yan

19.1 Introduction

19.2 Data and methods

19.3 Results

19.4 Discussion

Acknowledgement

References

20. Total Survey Error Assessment for Socio–Demographic Subgroups in the 2012 National Immunization Survey
Kirk M. Wolter, Vicki J. Pineau, Benjamin Skalland, Wei Zeng, James A. Singleton, Meena Khare, Zhen Zhao, David Yankey, and Philip J. Smith

20.1 Introduction

20.2 Total survey error model framework

20.3 Overview of the National Immunization Survey

20.4 National Immunization Survey: Inputs for TSE model

20.5 National Immunization Survey total survey error analysis

20.6 Summary

References

21. Establishing Infrastructure for the Use of Big Data to Understand Total Survey Error: Examples from Four Survey Research Organizations Overview
Brady T. West

21.1 Big Data infrastructure at the Institute for Employment Research (IAB)
Antje Kirchner, Daniela Hochfellner, and Stefan Bender

References

21.2 Using administrative records data at the U.S. Census Bureau: Lessons learned from two research projects evaluating survey data
Elizabeth M. Nichols, Mary H. Mulry, and Jennifer Hunter Childs

Acknowledgements and disclaimers

References

21.3 Statistics New Zealand s approach to making use of alternative data sources in a new era of integrated data
Anders Holmberg and Christine Bycroft

References

21.4 Big Data serving survey research: Experiences at the University of Michigan Survey Research Center
Grant Benson and Frost Hubbard

Acknowledgements and disclaimers

References

SECTION 5 – Estimation and Analysis

22 Analytic Error as an Important Component of Total Survey Error: Results from a Meta–Analysis
Brady West, Joe Sakshaug, and Yumi Kim

22.1 Overview22.2 Analytic error as a component of total survey error

22.3 Appropriate analytic methods for survey data

22.4 Methods

22.5 Results

22.6 Discussion

Acknowledgements

References

23 Mixed Mode Research: Issues in Design and Analysis
Joop Hox, Edith de Leeuw, and Thomas Klausch

23.1 Introduction

23.2 Designing mixed mode surveys

23.3 Literature overview

23.4 Diagnosing sources of error in mixed mode surveys

23.5 Adjusting for mode measurement effects

23.6 Conclusions

References

24 The Effect of Nonresponse and Measurement Error on Wage Regression Across Survey Modes: A Validation Study
Antje Kirchner and Barbara Felderer

24.1 Introduction

24.2 Nonresponse and response bias in survey statistics

24.3 Data and methods

24.4 Results

24.5 Summary and conclusion

Acknowledgements

References

25 Errors in Linking Survey and Administrative Data
Joe Sakshaug and Manfred Antoni

25.1 Introduction

25.2 Conceptual framework of linkage and error sources

25.3 Errors due to linkage consent

25.4 Erroneous linkage with unique identifiers

25.5 Erroneous linkage with nonunique identifiers

25.6 Applications and practical guidance

25.7 Conclusions and tale–home points

References

Paul P. Biemer, PhD, is distinguished fellow at RTI International and associate director of Survey Research and Development at the Odum Institute, University of North Carolina, USA.

Edith de Leeuw, PhD, is professor of survey methodology in the Department of Methodology and Statistics at Utrecht University, the Netherlands.

Stephanie Eckman, PhD, is fellow at RTI International, USA.

Brad Edwards is vice president, director of Field Services, and deputy area director at Westat, USA.

Frauke Kreuter, PhD, is professor and director of the Joint Program in Survey Methodology, University of Maryland, USA; professor of statistics and methodology at the University of Mannheim, Germany; and head of the Statistical Methods Research Department at the Institute for Employment Research, Germany.

Lars E. Lyberg, PhD, is senior advisor at Inizio, Sweden.

N. Clyde Tucker, PhD, is principal survey methodologist at the American Institutes for Research, USA.

Brady T. West, PhD, is research associate professor in the Survey Research Center, located within the Institute for Social Research at the University of Michigan (U–M), and also serves as statistical consultant on the Consulting for Statistics, Computing and Analytics Research (CSCAR) team at U–M, USA.