Open-Ended Questions in Survey Research

Controversy initially erupted over the use of open-ended (or free response) and close-ended questions in surveys and interviews (Converse, 1984) in the 1940’s. While close-ended questions rose to dominance due in large part to the expense of processing and analyzing free response data (Converse, 1984), interest in mixed methods in survey research has re-emerged in the last several years. Driving this resurgence is the value open-ended questions can add to the interpretation of responses; including open-ended prompts in surveys populated with otherwise close-ended questions provides insight into the considerations, concerns, and thought processes of survey participants. Specifically, open-ended comments are forwarded as  useful in understanding the replies to closed questions (Driscoll, Appiah-Yeboah, Salib, & Rupert, 2007; Garcia et al., 2004), providing more depth to the topics discussed in the survey (Garcia et al., 2004), identifying new research issues (Garcia et al., 2004), obtaining feedback on the research process (Garcia et al., 2004), and avoiding bias that may result from suggesting responses to participants (Reja, Manfreda, Hlebec, & Vehovar, 2003).

Despite the many advantages of using open-ended comments in survey research, such questions introduce a unique set of challenges and concerns. Most notably, open-ended questions require extensive coding and are often associated with higher levels of non-response (Reja et al., 2003). How, then, can open-ended comments be employed most productively in research efforts? We provide an outline below of best practices in the use of open-ended questions as suggested by research in this area.

1. Target open-ended question prompts toward specific topics. Responses to general (e.g., “if you have any additional comments, please provide them below”) prompts are more likely to vary in relevance and scope (Evans et al., 2005; Garcia et al., 2004) and may not provide the level of detail or kind of information desired.
2. To increase response rate on open-ended questions, include targeted questions throughout the survey rather than only using a general open-ended question at the end of the survey. Open-ended questions asked at the end of a survey may elicit shorter answers than open-ended questions asked earlier in a survey (Galesic & Bosnjak, 2009).
3. Assess response bias to open-ended comments. Certain people, including those with education (Garcia et al., 2004), more interest in the survey topic (Geer, 1988; 1991), higher perceptions of survey value (Rogelberg, Fisher, Maynard, Hakel, & Horvath, 2001), and more negative experiences (Evans et al., 2005; Garcia et al., 2004; Poncheri, Lindberg, Thompson, & Surface, 2008) may be more likely to respond to open-ended questions. Consequently, responses to open-ended comments may not be representative of all respondents’ opinions.
4. To reduce negativity bias in – and potentially boost response rate to – responses to open-ended questions, provide more detailed, motivating instructions in open-ended item stems (Smyth, Dillman, Christian, & McBride, 2009). Including explanations or instructions in the stem (e.g., emphasizing the importance of open-ended responses to the project) may improve open-ended item response length, elaboration on themes, and item response rate (Smyth et al., 2009). However, it is unclear whether these instructions would substantively affect the negativity of responses in addition to the response rate.

Using open-ended questions in survey research can illuminate responses to close-ended questions, providing researchers with richer data. To minimize the potential weaknesses of open-ended questions, the literature suggests that researchers should use targeted open-ended questions with motivating instructions throughout their surveys rather than only using a general open-ended question at the end of the survey. Additionally, researchers should assess how generalizable the content of open-ended responses is by comparing the characteristics and attitudes of participants who responded and did not respond to open-ended questions.

References

Converse, J. M. (1984). Strong arguments and weak evidence: The open/closed questioning controversy of the 1940’s. Public Opinion Quarterly, 48, 267-282.

Driscoll, D. L., Appiah-Yeboah, A., Salib, P., & Rupert, D. J. (2007). Merging qualitative and quantitative data in mixed methods research: How to an why not. Ecological and Environmental Anthropology, 3, 19-28.

Evans, J., Lambert, T. W., & Goldacre, M. J. (2005). Postal surveys of doctors’ careers: Who writes comments and what do they write about? Quality & Quantity, 39, 217-239.

Galesic, M. & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73, 349-360.

Garcia, J., Evans, J., & Reshaw, M. (2004). “Is there anything else you would like to tell us” – Methodological issues in the use of free-text comments from postal surveys. Quality & Quantity, 38, 113-125.

Geer, J. G. (1991). Do open-ended questions measure “salient” issues? Public Opinion Quarterly, 55, 360-370.

Geer, J. G. (1988). What do open-ended questions measure? Public Opinion Quarterly, 52, 365-371.

Poncheri, R. M., Lingberg, J. T., Thompson, L. F. & Surface, E. A. (2008). A comment on employee surveys: Negativity bias in open-ended responses. Organizational Research Methods, 11, 614-630.

Reja, U., Manfreda, K. L., Hlebec, V., & Vehovar, V. (2003). Open-ended vs. close-ended questions in web questionnaires. Developments in Applied Statistics, 19, 159-177.

Rogelberg, S., G., Fisher, G. G., Maynard, D. C., Hakel, M. D., & Hovath, M. (2001). Attitudes toward surveys: Development of a measure and its relationship to respondent behavior. Organizational Research Methods, 4, 3-25.

Smyth, J. D., Dillman, D. A., Christian, L. M., & McBride, M. (2009). Open-ended questions in web surveys: Can increasing the size of answer boxes and providing extra verbal instructions improve response quality? Public Opinion Quarterly, 73, 325-337.