Are they answering what we think we're asking? The importance of pre-testing survey questions using cognitive interviews

Evaluation often involves conducting surveys. To create a survey, a typical process consists of: drafting questions using our experience and understanding of basic principles of good question writing to guide us, and obtaining feedback from colleagues. Sometimes a focus group might be included, or conducting a small scale trial. However this process misses the vital pre-testing stage of cognitive interviews. Cognitive interviews are widely used in the development of surveys for health and social sciences, but appear to be underutilised to inform the development of instruments in other fields. In the process of reviewing its unit and teaching evaluation instruments, the University of New England (UNE) investigated both existing and proposed questions with cognitive interviews using a retrospective think aloud protocol. Over 120 interviews were conducted with students. Each student was asked three or four questions from the full set, resulting in about 20 responses for each question. Interviews were recorded, transcribed, and analysed for variation in understanding. The cognitive interviews identified problems with conceptualisation and phrasing in the question wording, as well as problems with variation of interpretation of the items on the response scale. These problems were not at all clear to the panel developing the survey prior to the interviews being conducted.