Expert review or expert evaluation is a method to evaluate survey questions from the perspective of one or more experts.
An expert review has two primary goals: [1]
- Identify potential problems related to data quality and data collection so they can be mitigated.
- Group survey items by how likely they are to result in measurement errors, such as to classify interviewer effects.
An individual expert or a panel of experts such as survey methodologists, subject matter experts (such as sociologists) or other people familiar with questionnaire design review the survey, and they may be asked to suggest revisions. [2] Because variation is found in the problems identified by each expert, a panel rather than one single expert is recommended. [1]
The experts document their assessment and typically report the results in open-ended comments. [3] They can also conduct a systematic review using a coding tool such as the Questionnaire Appraisal System that is supported by an item taxonomy of the cognitive demands of a question and a detailed set of potential problem codes document the features that may lead to response error. [4] [5]
When applied to translated surveys, the expert panel also includes translators and they usually produce individual reports and then convene a resolution meeting that generates an improved version of the survey translation. [6]
Compared to empirical pretesting methods, an expert review is more time-efficient and less costly to complete. [7] An expert review can also be combined with other methods to enable multimode pretesting. [8] [9] [10]
See also
References
- ^ a b Olson, Kristen (2010). "An Examination of Questionnaire Evaluation by Expert Reviewers". Field Methods. 22 (4): 295–318. doi: 10.1177/1525822X10379795.
- ^ DeMaio, Terry; Landreth, Ashley (2004), Prüfer, Peter; Rexroth, Margrit; Fowler, Floyd Jackson Jr. (eds.), "Examining expert reviews as a pretest method", QUEST 2003: proceedings of the 4th Conference on Questionnaire Evaluation Standards, 21–23 October 2003, ZUMA-Nachrichten Spezial, vol. 9, Mannheim: Zentrum für Umfragen, Methoden und Analysen -ZUMA-, pp. 60–73, ISBN 978-3-924220-27-3, retrieved 2024-02-21
- ^ Yan, T.; Kreuter, F.; Tourangeau, R (December 2012). "Evaluating Survey Questions: A Comparison of Methods". Journal of Official Statistics. 28 (4): 503–529.
- ^ Willis, G; Lessler, J (1999). "Question Appraisal System (QAS-99)" (PDF). Centers for Disease Control and Prevention Website.
- ^ Maitland, Aaron; Presser, Stanley (2020-01-02), Beatty, Paul; Collins, Debbie; Kaye, Lyn; Padilla, Jose Luis (eds.), "A Comparison of Five Question Evaluation Methods in Predicting the Validity of Respondent Answers to Factual Items", Advances in Questionnaire Design, Development, Evaluation and Testing (1 ed.), Wiley, pp. 75–90, doi: 10.1002/9781119263685.ch4, ISBN 978-1-119-26362-3, retrieved 2024-02-21
- ^ Pan, Yuling; Sha, Mandy (2019-07-09). The Sociolinguistics of Survey Translation. London: Routledge. pp. 118–124. doi: 10.4324/9780429294914. ISBN 978-0-429-29491-4. S2CID 198632812.
- ^ Presser, S; Blair, J. "Survey pretesting: Do different methods produce different results?". Sociological Methodology. 24: 73–104.
- ^ Ridolfo, Heather; Thompson, Ashley (2023-10-05). "Conducting Small-Scale Multimethod Questionnaire Evaluation". Survey Practice. 16 (1). doi: 10.29115/SP-2023-0017.
- ^ Sha, Mandy (March 2012). "Translation Review and Cognitive Testing of ACS Language Assistance Guides in Multiple Languages". United States Census Bureau. Retrieved 2024-01-08.
- ^ Aizpurua, Eva (2020). "Pretesting methods in cross-cultural research (Chapter 7)". In Sha, Mandy; Gabel, Tim (eds.). The essential role of language in survey research. RTI Press. pp. 129–150. doi: 10.3768/rtipress.bk.0023.2004. ISBN 978-1-934831-23-6.
Expert review or expert evaluation is a method to evaluate survey questions from the perspective of one or more experts.
An expert review has two primary goals: [1]
- Identify potential problems related to data quality and data collection so they can be mitigated.
- Group survey items by how likely they are to result in measurement errors, such as to classify interviewer effects.
An individual expert or a panel of experts such as survey methodologists, subject matter experts (such as sociologists) or other people familiar with questionnaire design review the survey, and they may be asked to suggest revisions. [2] Because variation is found in the problems identified by each expert, a panel rather than one single expert is recommended. [1]
The experts document their assessment and typically report the results in open-ended comments. [3] They can also conduct a systematic review using a coding tool such as the Questionnaire Appraisal System that is supported by an item taxonomy of the cognitive demands of a question and a detailed set of potential problem codes document the features that may lead to response error. [4] [5]
When applied to translated surveys, the expert panel also includes translators and they usually produce individual reports and then convene a resolution meeting that generates an improved version of the survey translation. [6]
Compared to empirical pretesting methods, an expert review is more time-efficient and less costly to complete. [7] An expert review can also be combined with other methods to enable multimode pretesting. [8] [9] [10]
See also
References
- ^ a b Olson, Kristen (2010). "An Examination of Questionnaire Evaluation by Expert Reviewers". Field Methods. 22 (4): 295–318. doi: 10.1177/1525822X10379795.
- ^ DeMaio, Terry; Landreth, Ashley (2004), Prüfer, Peter; Rexroth, Margrit; Fowler, Floyd Jackson Jr. (eds.), "Examining expert reviews as a pretest method", QUEST 2003: proceedings of the 4th Conference on Questionnaire Evaluation Standards, 21–23 October 2003, ZUMA-Nachrichten Spezial, vol. 9, Mannheim: Zentrum für Umfragen, Methoden und Analysen -ZUMA-, pp. 60–73, ISBN 978-3-924220-27-3, retrieved 2024-02-21
- ^ Yan, T.; Kreuter, F.; Tourangeau, R (December 2012). "Evaluating Survey Questions: A Comparison of Methods". Journal of Official Statistics. 28 (4): 503–529.
- ^ Willis, G; Lessler, J (1999). "Question Appraisal System (QAS-99)" (PDF). Centers for Disease Control and Prevention Website.
- ^ Maitland, Aaron; Presser, Stanley (2020-01-02), Beatty, Paul; Collins, Debbie; Kaye, Lyn; Padilla, Jose Luis (eds.), "A Comparison of Five Question Evaluation Methods in Predicting the Validity of Respondent Answers to Factual Items", Advances in Questionnaire Design, Development, Evaluation and Testing (1 ed.), Wiley, pp. 75–90, doi: 10.1002/9781119263685.ch4, ISBN 978-1-119-26362-3, retrieved 2024-02-21
- ^ Pan, Yuling; Sha, Mandy (2019-07-09). The Sociolinguistics of Survey Translation. London: Routledge. pp. 118–124. doi: 10.4324/9780429294914. ISBN 978-0-429-29491-4. S2CID 198632812.
- ^ Presser, S; Blair, J. "Survey pretesting: Do different methods produce different results?". Sociological Methodology. 24: 73–104.
- ^ Ridolfo, Heather; Thompson, Ashley (2023-10-05). "Conducting Small-Scale Multimethod Questionnaire Evaluation". Survey Practice. 16 (1). doi: 10.29115/SP-2023-0017.
- ^ Sha, Mandy (March 2012). "Translation Review and Cognitive Testing of ACS Language Assistance Guides in Multiple Languages". United States Census Bureau. Retrieved 2024-01-08.
- ^ Aizpurua, Eva (2020). "Pretesting methods in cross-cultural research (Chapter 7)". In Sha, Mandy; Gabel, Tim (eds.). The essential role of language in survey research. RTI Press. pp. 129–150. doi: 10.3768/rtipress.bk.0023.2004. ISBN 978-1-934831-23-6.