By Dr Keith Meadows BA, PhD, CMRS, AFBPsS, CPsychol
Introduction
Are focus groups a valid method for cognitive interviewing in developing and evaluating clinical outcome assessment measures (COAs)? To answer this question it will be instructive to go back to a time in 1983 when the early work of cognition and survey research resulted in a movement which has become known as The Cognitive Aspects of Survey Methodology or CASM.
The impact of this movement resulted in a paradigm shift from the behavioural to a cognitive paradigm which postulated that respondents typically, follow a logical sequence of comprehension and retrieval in which the response process was divided into four key components (Tourangeau 1984) which are:
1. Comprehension of the item – Identify question focus and required information
2. Retrieval of relent information – Conduct an information retrieval strategy
3. Judgement – Assess the relevance and completeness of the retrieved information
4. Response – Map judgement onto response category
Each of the four components can give rise to response effects such as misinterpreting the question or forgetting crucial information and which form the structure of the cognitive interview to understand respondents’ thought process as they interpret survey items and determine their answers.
This process inspired two kinds of CASM that are rarely if ever defined in the reporting of studies conducting cognitive interviews which are:
1. The Reparative Approach (Applied CASM)
Willis (2005, 2015) defines the reparative approach as the “inspect and repair” model or as finding and fixing focusing on improving survey questions by reducing response errors (Miller, 2011; Willis, 2005, 2015). For example, the SF-36 item “Would you consider your health in general is excellent, very good, good, fair, or poor?” (Ware & Sherbourne, 1992). On the basis of cognitive interview participants may report they fail to agree on the interpretation of quantifiers such as “very good.” Therefore, a set of more specific identifiers may provide better information.
2. The Descriptive Approach (Basic CASM)
In the descriptive approach, using the SF-36 item again as an example, researchers may conclude that the item functions sufficiently for the intended purpose of the investigation so there are no repairs or modifications needed. However, if the researcher needs a better understanding of the concept the item is measuring may need more description. “
Although we learn a lot from cognitive interviews regarding semantic, recall and computational problems that occur during the response process, often little insight is gained about the type of information the question elicits particularly when no response errors are found” (Ridolfo & Schoua-Glasberg, 2011). This suggests a shift from discovering problem questions and repairing
them to one of gaining a better understanding of how the question measures a particular concept. The reparative and descriptive approaches are not dichotomous but endpoints on a continuum.
Cognitive interviewing is a technique used in which participants are encouraged to either concurrently or retrospectively think aloud and the interviewer probes about specific items to ensure that the questions are easily understood and valid. It is now the most widely use tool for questionnaire development (Conrad & Blair1996) with its central underlying thesis being the exploration of the individual’s mental processes in the understanding of and response to the questionnaire’s content. (Note here the emphasis of the process being the ‘individual respondent’).
Focus groups as a tool for cognitive interviewing
Focus groups are increasingly seen as a valid method for cognitive interviewing that can provide insights into a target population’s cognitive processes and context and promote open-ended dialogue. See Farmer et al (2022). However, their use in the development of clinical outcome assessment measures (COAs) should not be seen as representing an alternative to the traditional; method of cognitive interviews.
Operating in the manner of a mixed-methods approach, focus groups can encourage open-ended dialogue among participants, building on each other’s ideas where the moderator can interact with the group to ask follow-up questions. For example, cognitive-based focus groups based on the think-aloud interviews can result in a rich source of information that can complement that obtained through the cognitive interviews. But, it is not an alternative to the data obtained via the cognitive interview. With the cognitive interview we have the respondent’s self-report untarnished by the reports of others.
Conclusion
Cognitive interviews are now a common method in the development and evaluating COAs and their cultural adaptation. Nevertheless, there is a growing movement in the application of focus groups as an alternative to the traditional cognitive interview of how individual respondents react to survey questions.
While focus groups can be more cost effective and quicker to conduct than cognitive interviews, they should not be seen as an alternative method to the traditional cognitive interview as they are based on a different theoretical grounding of group interaction which can produce insights and data that would be difficult to access without the group setting.
Cognitive-based focus groups however, can be part of a mixed-methods approach in which the findings from the cognitive interviews form the basis for further inquiry
References
Conrad, F. F., & Blair, J. (1996). From impressions to data: Increasing the objectivity of cognitive interviews. Paper presented at the 1996 Joint Statistical Meeting, Chicago, August 4,1996.
Farmer, N., M., T., Middleton, K. R., Brooks, A. T., Mitchell, V., Troncoso, M., Ceasar, J., Claudel, S. E., Andrews, M. R., Kazmi, N., Johnson, A., & Wallen, G. R. (2022). Use of a focus group-based cognitive interview methodology to validate a cooking behavior survey among African-American adults. Frontiers in Nutrition, 9, 1000258. https://doi.org/10.3389/fnut.2022.1000258
Ridolfo H., Schoua-Glusberg A. (2011). Analyzing cognitive interview data using the constant comparative method of analysis to understand cross-cultural patterns in survey data. Field Methods, 23(4): 379–396.
Tourangeau R. (1984). Cognitive sciences and survey methods. In Jabine T., Straf M., Tanur J., Tourangeau R. (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73–100). National Academy Press.
Ware J. E. Jr., Sherbourne C. D. (1992). The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection. Medical Care, 30, 473–483.
Willis G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage.
Willis G. B. (2015). Analysis of the cognitive interview in questionnaire design. Oxford University Press.
Duc DM, Bergstrom A, Eriksson L, Selling K, Thi Thu Ha B, Wallin L. Response process and test-retest reliability of the Context Assessment for Community Health tool in Vietnam. Glob Health Action. (2016) 9:31572. doi: 10.3402/gha.v9.31572 52.
Duc, D. M., Bergström, A., Eriksson, L., Selling, K., Thi Thu Ha, B., & Wallin, L. (2016). Response process and test–retest reliability of the Context Assessment for Community Health tool in Vietnam. Global Health Action, 9. https://doi.org/10.3402/gha.v9.31572
Huang KT, Owino C, Vreeman RC, Hagembe M, Njuguna F, Strother RM, et al. Assessment of the face validity of two pain scales in Kenya: a validation study using cognitive interviewing. BMC Palliat Care. (2012) 11:5. doi: 10.1186/1472- 684X-11-5 53.
Napoles-Springer AM, Santoyo-Olsson J, O’Brien H, Stewart AL. Using cognitive interviews to develop surveys in diverse populations. Med Care. (2006) 44(11 Suppl. 3):S21–30. doi: 10.1097/01.mlr.0000245425.65905.1d 54.
Napoles-Springer, A. M., Santoyo-Olsson, J., O??Brien, H., & Stewart, A. L. (2006). Using Cognitive Interviews to Develop Surveys in Diverse Populations. Medical Care, 44(Suppl 3), S21–S30. https://doi.org/10.1097/01.mlr.0000245425.65905.1d
Huang KT, Owino C, Vreeman RC, Hagembe M, Njuguna F, Strother RM, et al. Assessment of the face validity of two pain scales in Kenya: a validation study using cognitive interviewing. BMC Palliat Care. (2012) 11:5. doi: 10.1186/1472- 684X-11-5 53.