Qualitative Testing of Questionnaires on Existential, Spiritual, and Religious Constructs: Epistemological Reflections

Abstract

Quantitative methodologies using questionnaires are widely utilised across the health and social sciences, including when assessing difficult-to-measure constructs such as existential, spiritual, or religious constructs. The validity of the questionnaire is crucial to secure good research. Qualitative methods are essential to assess the validity of questionnaires, including a range of measurement properties such as comprehensiveness, comprehensibility, and content validity—properties that quantitative measures have difficulty addressing. Cognitive and semi-structured interviews are the most common qualitative methods to evaluate these properties when testing questionnaires. However, the choice of interview method and subsequent analysis also reflects a negotiation between epistemologies, influencing how the data are interpreted and applied. Based on experience with cognitive and semi-structured interviews to test a questionnaire on existential, spiritual, and religious constructs, this paper critically discusses the opposing epistemological premises for cognitive and semi-structured interviews. It is a relevant methodological discussion for researchers working with questionnaires in health sciences, not the least in relation to religious studies broadly speaking.

1. Introduction

When designing or adapting questionnaires, researchers have long advised the inclusion of qualitative methods, such as cognitive interviews (CIs) and semi-structured interviews (SSIs) (de Vet et al. 2011; Terwee et al. 2018). They argue that understanding how future respondents conceptualise and experience the construct being measured is essential for examining both comprehensiveness and comprehensibility, which, in turn, is crucial for establishing content validity (de Vet et al. 2011; Terwee et al. 2018). This is true in health sciences broadly but perhaps even more so when using psychometric instruments to measure such ungraspable constructs as religiosity, spirituality, or existentiality. In this paper, spirituality is considered the umbrella term for these constructs, which can, by drawing on experience from palliative medicine, be defined as the “dynamic and intrinsic aspect of humanity through which persons seek ultimate meaning, purpose, and transcendence, and experience relationship to self, family, others, community, society, nature, and the significant or sacred. Spirituality is expressed through beliefs, values, traditions, and practices” (Puchalski et al. 2014, p. 646). This definition is inclusive of the continuum between theistic (i.e., religious) orientations and atheist convictions (i.e., more secular-oriented meaning-making).

There are multiple questionnaires available today that measure different (overlapping) aspects of religion, spirituality, and meaning in life. While this is not the place to review and compare all of these, a few pointers and places for further reading will be made. The most prominent instruments are reviewed in the Handbook of Religion and Health (Koenig et al. 2012). A catalogue and review of instruments on spiritual care was made by Damberg Nissen et al. (2020, 2021). A recent review of instruments measuring such constructs in children can be found in the Archive for the Psychology of Religion (Bloch et al. 2025). By the combined efforts of brilliant scholars around the world, we now have access to a range of well-validated measures of these constructs. Multiple instruments, such as the Meaning in Life Questionnaire by Steger et al. (2006), the Duke University Religion Index (DUREL) by Koenig and Büssing (2010), and the Spiritual Needs Questionnaire by Büssing et al. (2010), have undergone rigorous testing to ensure validity. However, sometimes instruments are developed based entirely on theoretical reflections by the researcher. While an expert perspective is undoubtedly crucial in instrument development, such a development process risks the instrument not being valid for the intended responder since they were not asked about how they understand the construct. A risk that a few of the instruments available still suffer from is the failure to differentiate between states of mental well-being and an orientation towards something transcendent, grouping, e.g., feeling good and turning towards God in the same measure of, e.g., spirituality. As noted by Koenig, this is something to be aware of as it may lead to tautological inferences regarding the relationship between religiosity, spirituality, and (mental) health (Koenig 2008). What I shall try to argue below is that when attempting to measure anything related to such difficult constructs as mentioned above, qualitative testing focusing on really understanding the responder’s perspective is crucial to secure validity. Second, clear definitions (at the bare minimum internally within the study) and transparency are needed to mitigate the difficulties in reaching consensus definitions on these subjects.

The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) initiative was founded to enhance the quality of studies focused on measurement properties. Since then, the COSMIN guidelines have been the gold standard for survey and questionnaire taxonomy and terminology (Mokkink et al. 2018; Prinsen et al. 2018). COSMIN argues that qualitative methods, e.g., CIs and SSIs, are necessary to establish measurement properties (Terwee et al. 2018). It further argues that researchers should interview individuals representing the target population. Despite these recommendations, qualitative interviews are often not employed for different reasons, such as unawareness of these recommendations, time limitations in the project, lack of funding, etc.

The author of this paper is positioned within both the positivistic and interpretative/constructivist paradigms, employing both quantitative and qualitative methods in their research. This paper aims to enhance the methodological and epistemological discussion of the qualitative testing of questionnaires used to measure constructs such as religiosity, spirituality, and existentialism. The paper starts by establishing the rationale for this approach by shortly outlining the key characteristics of CIs and SSIs, respectively, for the unacquainted reader. Subsequently, the epistemological assumptions inherent in each method are examined. Drawing on experience from the qualitative testing of a questionnaire on spiritual needs as a point of departure, the author discusses the strengths and limitations of specific aspects of using CIs and SSIs either as standalones or in combination.

2. Two Approaches to Qualitatively Testing a Questionnaire

2.1. The Cognitive Interview

Since its development in the 1980s, the cognitive interview method has become widely used for testing both preexisting (translated) and newly developed questionnaires. CIs engage psychological approaches to assess specific cognitive decision processes in informants, including 1. comprehension, 2. recall, 3. judgment, and 4. answers (Cannell et al. 1977; Jabine et al. 1984; Strack and Martin 1987). CIs evaluate possible response “errors” and problems in a survey and are used to obtain insights into whether a chosen questionnaire is feasible and valid to use for a specific target population (Beatty and Willis 2007; Collins 2015). In this context, the CI is based on the positivist assumption that the individual’s cognitive processes exist within the brain and can be effectively revealed and measured using appropriate tools. If the interviewer is skilled, they can identify flaws in the questionnaire by analysing how informants respond. Ideally, all questions within a questionnaire are tested in a CI. However, this is often not possible due to time limitations and the concentration fatigue of the participants (Scott et al. 2021; Willis 2005). The scope of a CI—including which questions are selected for testing—should be outlined in a CI protocol. This document states the purpose, aim, and research questions of the specific project and guides the interviewer through the interview process, similar to the interview guide developed for SSIs. CIs are often performed in iterative rounds, testing all or selected questions in the questionnaire. The questionnaire may be altered and improved in between the rounds, based on the findings derived from the CI. The iterative rounds may indirectly enable quantitative analyses of the data collected. This is seen, for instance, in a study by Levin et al. (2009), who used CIs to test a questionnaire on dietary intake in a Spanish population. Through content analysis, they counted the number of issues informants had encountered or reported as problematic in the tested questionnaire, such as items that were perceived as ambiguous on a semantic level. They aimed to demonstrate improvements in the questionnaire over iterative rounds by reducing the number of identified issues.

A commonly used interviewer tool during CIs is ‘think aloud’ (Collins 2015), where participants verbalise their thoughts as they answer each question. The positivist assumption behind this method is that, optimally, the verbalisation accurately reflects all the thought processes involved when responding, and by examining this line of thought, one can identify misunderstandings in either comprehension, retrieval, judgment, or answer. It is a relatively easy tool to learn. However, if the topic is sensitive (e.g., personal religious beliefs, political ideology, sexual activity, etc.), the ‘think aloud’ tool might not yield sufficient insights. This is an important concern when studying constructs of spirituality, as ‘think aloud’ might bias the analysis, since the data may not accurately reflect the thoughts of the participants, e.g., they may refuse (either consciously or unconsciously) to verbalise or retrieve incriminating thoughts (Collins 2015).

Table 1. Examples of probes taken from the interview protocols of Stripp et al. in Survey Research Methods (Stripp et al. 2023a).

Think Aloud Example	Probing Examples
Participant reading and thinking aloud in relation to a question concerning belief in a guardian angel. Answer categories: Yes, No, Don’t know		Real time	Retrospective
Participant reading out aloud: “Do you believe in a guardian angel?” Participant thinking aloud: “Well, especially when I was a kid, I liked to think that somebody was watching over me. Now I … ehm… I don’t really know. I am not sure if I really believe in that, but I can still get a feeling sometimes that somebody is looking after me or is with me, you know. So, I guess that means “Yes”.”	Scripted probe	What did the question [specific wording] make you think of?	Do you feel that there are certain spiritual beliefs, which you have had, or have, that were not covered by the questionnaire?
	Improvised probe	That question seemed particularly difficult for you to answer. Could you elaborate on that?	I wonder if there were any words or questions in any of the items that you found it difficult to identify with?

presents an example of data generated from a participant using the think aloud method from Stripp et al. (2023a) in Survey Research Methods.

‘Probing’ is another commonly used interviewer tool, where the interviewer asks scripted—deriving from the interview guide—or improvised open-ended questions to explore individually perceived aspects of specific questions and concepts. Some CI methodologists consider probing to be the ideal tool within a CI, especially in public health areas, where questionnaires might contain sensitive topics (Schuman 1966; Scott et al. 2021). Probing, in general, can be conducted in real time during the questionnaire completion or retrospectively after the informant has completed the questionnaire. Table 1 exemplifies different probes. The example from above used both scripted and improvised probing in real time and retrospectively (Stripp et al. 2023a).

2.2. The Semi-Structured Interview

The SSI as a qualitative research method accommodates a multiplicity of philosophical assumptions deriving from phenomenology, hermeneutics, feminism, grounded theory, and post-positivism. This means that a universal and theory-neutral SSI does not exist. When dealt with in this context, the SSI reflects phenomenological and hermeneutic approaches, aiming at exploring the experiential world from the subjects’ perspectives and describing and interpreting the world as experienced by the subjects. Although the SSI is a research interview that goes beyond daily conversations and is therefore characterised by a power asymmetry between researcher and interview participant, SSI data are generated through what Kvale and Brinkmann call inter views: an interchange of views between the researcher and the research participant (Kvale and Brinkmann 2009). The idea of knowledge as a construction inherent in the conception of inter views contrasts with methodological positivism, with its conception of research as unbiased and knowledge as collected.

An SSI may be used as a standalone method when in-depth knowledge about a phenomenon is needed, or as a basis for instrument development. It involves a specific approach and technique of questioning; it is semi-structured and thus neither unstructured nor structured. Prior to conducting an SSI, an interview guide is developed focusing on certain themes and comprising open-ended interview questions focusing on the knowledge that one seeks to obtain (as formulated in the research questions). However, the researcher retains the freedom to focus on other aspects of the phenomenon should these appear more significant to the participant in the interview situation. This flexibility constitutes the semi-structured aspect of the method. The conception of flexibility contrasts with methodological positivism with its conception of research as following rules and predetermined steps. The interviews are usually transcribed, and the transcripts and audio files constitute the materials for the subsequent analysis of meaning. For more information on SSIs, see also (Rubin and Rubin 2011; Seale et al. 2003).

3. Epistemological Differences and Similarities Between Cognitive and Semi-Structured Interviews

Studies addressing constructs related to spirituality are often caught between a rock and a hard place when it comes to the epistemological approach. A quantitative positivistic epistemology can be applied, making measurement and statistical tools available while at the same time reducing the individual experience of, e.g., communion with the divine to numbers—a practice that obviously does not do justice to phenomenology. If, on the other hand, the researcher focuses solely on the qualitative inquiry into spiritual life, important tools to make general inferences across populations are left behind. CIs are frequently placed under the umbrella of qualitative methods. However, CIs are traditionally underpinned by the positivist conception that language and behaviour reflect the objective reality (of a particular informant) (Willis and Artino 2013). For the method to be effective, the interviewers must minimise the influence of their subjectivity to avoid biasing the responses. Success is connected to establishing accuracy and reliability and is achieved when different individuals, representative of the target population, understand the questions in the same way when going through the survey (Willis and Artino 2013).

Specifically, the tool ‘think aloud’ exemplifies a positivistic conception, according to which the informants’ answers reflect the reality of how informants understand survey items. Following this logic, it is essential to ensure that the informant’s preconception of a survey item is not affected by the researcher’s preconception of a survey item to control and, ideally, avoid bias.

These assumptions stand in contrast to an SSI. Within a qualitative, interpretative paradigm, language is understood to construct—rather than merely reflect—reality. The interviewer’s subjectivity is integral to interrogating dominant patterns of meaning in relation to a phenomenon. Data are thus a product of co-construction between the researcher and the informant and not something that already exist to be collected. Reliability, more than being established through replicability, is established through other means, e.g., transparency in presumptions and methodology (Long and Johnson 2000). Combining CIs with SSIs might, therefore, create epistemological incongruences. Conducting CIs in combination with SSIs (or vice versa) is not without challenges, and the strengths and limitations of this approach need to be considered.

Common epistemological differences between the positivistic and interpretivistic paradigms are presented in

Table 2. Common epistemological differences.

(post) Positivism	Interpretivism/Constructivism
Objectivity—belief in an unbiased truth	Subjectivity—the belief that people construct reality through social and cultural processes
Knowledge reflects a reality which already exists to be collected	Knowledge is a product of co-interpretation and co-construction, e.g., through language
Hypothetical deductive inquiry—belief in the scientific method through which data can be replicated	Inductive/interpretive approaches rely on naturalistic methods, e.g., entering real-world settings to observe, interact, and understand
Reliability is established by repeating identical measurements (replicability)	Knowledge is co-created and never considered replicable. Reliability is established by transparency in approach and subjectivity
‘Detached’, neutral, and unbiased scientist—biases need to be controlled	Involved researcher biases need to be understood and are inseparable from the inquiry and outcomes
The predominant use of quantitative methods in which units are counted, weighed, and analysed with statistical methods	The predominant use of qualitative methods in which observations and interviews are conducted and analysed through hermeneutic analyses
Seeks to explain phenomena	Seeks to understand phenomena

.

Having said this, the CI tradition also includes interpretivistic epistemological assumptions. For example, whereas the tool ‘think aloud’ represents a positivistic assumption—suggesting that the answers provided by the informants reveal how participants understand survey items, without any influence from the researcher—‘probing’ (improvised probing, in particular) represents a more interpretative approach. This method acknowledges that the responses generated during probing are the result of a collaborative process between the researcher and the informant. In this case, knowledge is co-constructed, meaning that both parties contribute to shaping the understanding of the survey items (Willis 2005). Consequently, if considering a continuum between positivist and interpretive paradigms, the ‘think aloud’ tool can be placed towards the end of the positivistic paradigm and the ‘probing’ tool at the opposite end towards the interpretive paradigm (see Figure 1). Similarly, the methodological frameworks of CIs and SSIs may be placed on their respective ends of this continuum—bearing in mind that this is a simplistic conceptualisation of CIs, as this approach also has traditions based on an interpretative paradigm.

Moreover, since CIs are often conducted in iterative rounds, with amendments made to the questionnaire in between rounds, the analytic process within CIs can resemble, to a large degree, the circular hermeneutic analytic process of data generated through SSIs.

It may be possible to argue that if attentive to epistemological differences and similarities, combining CIs with SSIs may in fact bridge the divide between a positivist and an interpretative research paradigm that some researchers, like Sale et al. (2002), have held to be insurmountable. In mixed methods research, such flexibility has often been labelled pragmatic, with reference to Charles Sanders Peirce and William James (Johnson and Onwuegbuzie 2004).

Probing over Think Aloud?

In recent years, researchers have argued that ‘probing’ should be considered the golden standard within CIs due to several limitations within the tool ‘think aloud’ (Scott et al. 2021). Importantly, this is underlined by the fact that establishing content validity is considered more important than reliability (Clifton 2020). This tendency might indicate a shift in philosophical assumptions within the CI method towards a more interpretative paradigm–more closer resembling the COSMIN suggestions. To minimise possible epistemological conflicts between CIs and SSIs, the use of ‘think aloud’ within the CI could be minimised and the researcher could instead increase the use of ‘probing’ when conducting the CI.

Conducting CIs demands that the researcher is familiar with both qualitative and quantitative methodologies. The researcher must be comfortable using scripted and improvised probes during the CIs; hence, the researcher should be familiar with a qualitative methodology, including creating a warm and welcoming atmosphere, building rapport, and making participants feel at ease. Moreover, the researcher should be experienced with quantitative methods, especially survey research methods, when performing the CIs. This provides the researcher with in-depth knowledge of the workings of the questionnaire they are testing and, e.g., which pitfalls they could expect (Scott et al. 2021).

4. Epistemological Approaches in Combination

CIs aim to identify and bridge differences (i.e., cultural, social, and linguistic) between a questionnaire and its target population (Scott et al. 2021). Without CIs, unidentified problems with the questionnaire can affect the data quality and the content validity (Collins 2015). However, CIs only capture informants’ individual mental processing of questionnaire items and might not always solve deeper problems within the questionnaire, i.e., content validity or contextual or design problems (Collins 2015; Scott et al. 2021). In the following, I shall provide an example of qualitative testing using CIs with a few open probes to conclude the interview—which arguably represents an SSI. I discuss some potential strengths and limitations that arise when navigating the epistemological cross-section between CIs and SSIs, which are sometimes pragmatic.

Mainly, I draw on an experience from investigating the existential and spiritual beliefs and needs among adult Danish patients (Stripp et al. 2022b, 2023b). To achieve this, we developed a questionnaire using translated validated instruments measuring spiritual needs and tested it qualitatively among fourteen individuals (Stripp et al. 2023a). We employed both the CI and the SSI methods within the same interview (Stripp et al. 2023a). The CI was conducted first, followed by an SSI involving a few open questions. During the SSI, we asked participants about their individual spiritual beliefs and needs to ensure that the questionnaire effectively captured the spiritual beliefs and needs of the target population. Some main empirical findings from the qualitative testing are reported in

Table 3. Main empirical results from a qualitative testing of a questionnaire on spiritual needs.

Main Findings

The acceptability and comprehensiveness of the questionnaire on existential, spiritual, and religious constructs and needs were assessed as very acceptable in the participants interviewed; CI data were summarised based on transcripts, notes, and observations. The questionnaire was amended based on a deductive theory-driven analysis of these summaries; Think aloud data were compromised by the sensitive nature of the questionnaire topic; thus, probing was increased; Some items/instruments were removed based on participants’ (strong emotional) reactions; The problems identified (i.e., similar to Levin et al. (2009)) dropped over iterative testing rounds from 81 issues in round I to 31 in round IV—interpreted as questionnaire improvement; Multiple participants reported that it had been a very positive, inspiring, and fulfilling experience to be confronted with questions on existential, spiritual, and religious topics that they did not often ponder themselves or discuss with others.

.

In the mentioned qualitative testing with CIs, the questionnaire instruments that were tested, e.g., the Spiritual Needs Questionnaire, primarily tapped into existential, spiritual, and religious constructs (Stripp et al. 2022a). When conducting the SSIs subsequent to the CIs, it became clear from the participants’ accounts that they had lacked a rich conceptualisation of their own existential, spiritual, or religious stance and that the questionnaire had in fact to a large extent broadened their understanding of themselves. As many Danes are not well-versed in the language pertaining to such constructs (Hvidt et al. 2021), this situation in which the CIs had conceptualised and introduced some existential, religious, and spiritual vernacular to the informant was a fruitful point of departure for the SSIs. This resulted in the discovery of an interesting discrepancy that some participants observed about their faith, which was then possible to examine further in the SSIs. As an example, during the CIs some participants had ticked off all the following three items: that they were ‘non-believers’, that they considered themselves ‘Christian’, and that they ‘believed in an afterlife’. The incongruence in the responses of some of the participants regarding belief systems can exemplify what sociologists of religion have described as different secularisation trends of modern people (Davie 1994; Davie et al. 2003). For example, in being selective in what one believes in, not adhering to ‘the whole package’, but choosing beliefs according to one’s own personal needs (e.g., of belonging to a particular ‘Christian’ culture and seeing one’s loved ones again), we see a development from traditional, collective forms of Christianity to individualistic, spiritual beliefs and practices (Hervieu-Léger and Lee 2000). Importantly, it should also be noted that the SSIs may produce valuable information that can be analysed in their own right.

A limitation worth mentioning is that combining CIs with SSIs seemed to induce the risk of exploiting neither the full potential of the CIs nor the SSIs. As such, focusing only on CIs, it would have been possible to examine individual items to a larger extent. Similarly, having made only SSIs would have enabled an even deeper understanding of the phenomenon as experienced by the participant.

It might be possible to consider the effects of positioning a more hermeneutical approach (SSI) at the beginning of an interview session and follow that up with strict questionnaire item testing (CI). Whereas CIs have a rather fixed inquiry form (especially when using scripted probing), SSIs invite the participant through a dialogical form to explore subjective experiences connected to the phenomenon in question, which might facilitate creativity and new thought associations of potential value to the developing stages of questionnaire items. A possible strength of combining SSIs with CIs may thus be that the experiential and contextual aspects of the phenomenon investigated are developed and captured, influencing how meaningful the questions appear to the target population.

As such, such a combined epistemology, although risking reliability, might improve user involvement. A perceived user involvement might be crucial to build rapport and also contribute to tackling response errors and increase content validity (Mes et al. 2019).

A limitation that should be noted is the increased risk of fatigue on part of the participant and researcher. In addition to the challenges that might result from an extended time frame, the shift in the interview approach might appear mentally challenging. Only doing CIs or SSIs might thus appear less demanding to the participant and researcher.

Arguably, it might be possible to use various epistemological approaches when qualitatively testing questionnaire items. The timing of such different approaches might substantially influence the data generated. Some differences are noted in Figure 2.

4.1. Cognitive Interview Before Semi-Structured Interview

In an epistemological understanding, if the CI is positioned before the SSI the informants’ responses to the questionnaire are immediate and spontaneous to a higher degree than if an SSI precedes it, reflecting to a higher extent the ideal circumstances in questionnaire testing by the target population. Starting with a CI, the tools ‘think aloud’ and ‘probing’ could not have been affected by the SSI but might inspire further discussions during a subsequent SSI (Mes et al. 2019). For example an item phrased “Do you consider yourself as: religious, spiritual, religious AND spiritual, or none?” was probed during the CIs. During the SSIs, several of the participants returned to this specific topic, as they wanted to elaborate on their stance on the matter.

Conducting a CI prior to an SSI might, on the other hand, affect the quality of the SSI. It is my experience that informants’ willingness to unfold and elaborate on their experiences in the SSI is reduced if positioned after a CI because of informant fatigue.

4.2. Semi-Structured Interview Before Cognitive Interview

If hypothetically the SSI is positioned before the CI, the SSI could not be affected by the conceptual understandings emerging from completing the questionnaire. On the other hand, it might be a limitation that the CI, conducted after the SSIs, will be influenced by the interaction and discussion that has taken place in the SSI. During the SSIs, a conceptual understanding of the investigated phenomenon and related terms is developed from the interaction between the interviewer and informant (DiCicco-Bloom and Crabtree 2006). This conceptual understanding can be seen as limiting the rationale of the ‘think aloud’ process, in that it impedes an immediate and spontaneous reflective activity regarding the stimuli (the questionnaire items) (Willis 2005). Seen from a positivist viewpoint, this may bias and compromise the reliability and repeatability of the CIs since informants will have participated in an SSI beforehand. It could be argued that if the SSI is conducted first, this will to a larger extent create a more varied predisposition in participants when the CI is commenced. This could perhaps lead to an over- or underestimation of comprehensiveness and comprehensibility.

However, conducting an SSI before a CI creates favourable conditions regarding building rapport, confidentiality, and trust, which might positively affect the quality of the data. For example, having established a good and trusting relationship between the interviewer and informant through an SSI, informants might feel less reluctant to share their thoughts during the ‘think aloud’ activity (Collins 2015; Willis 2005). Furthermore, if the researcher identifies topics that appear too sensitive for the informant during the SSI, the CI protocol can be adjusted in the following to avoid ethical and psychological harm being done to the informant. This is a possible ethical incentive for conducting the SSI before the CI. However, most importantly, the SSI is establishing increasing content validity, as a deeper understanding of the construct and the comprehensiveness of the questionnaire will become apparent.

5. Conclusions

In this article, I have demonstrated and discussed how both positivistic and interpretivistic epistemologies could be used when testing a questionnaire used to assess the spiritual needs of an individual. At this intersection, both strengths and limitations appear, as well as a clarification of methodological differences and similarities between the CI and the SSI. The primary aim was to strengthen the methodological discussion of how questionnaires are developed so that survey instruments, particularly pertaining to difficult-to-measure constructs such as spirituality, become more valid and valuable. For the researcher unacquainted with questionnaire methodology, I refer to more in-depth works on some of the methods described in the manuscript, particularly cognitive interviewing (Collins 2015), semi-structured interviewing (Kvale and Brinkmann 2009), and psychometric testing (de Vet et al. 2011).

Increasing the interpretative approach of CIs might endanger the reliability (and replicability) of the questionnaire testing. However, given the above, it might still be advisable to follow such a development since it is more beneficial for establishing the content validity of the questionnaire. After all, content validity is the most important measurement property of questionnaires (Clifton 2020). While transparency, especially regarding protocol and analysis, is recommended, there is still the issue of how to obtain rich qualitative data while increasing the replicability of that data. However, balancing this dilemma transparently might improve the validity of CIs in health sciences and increase their use.

It may be possible to harness the best from the two worlds of CI and SSI, perhaps sacrificing some reliability but enabling an increased examination of content validity. Hence, the author recommends focusing on an interpretative epistemological approach when qualitatively testing questionnaires touching upon spiritual constructs. Further research is needed to elaborate on how this approach may affect data analyses and the CI process in the development and testing of questionnaire instruments.

Ethical evaluation of the research process is warranted, like in all research. Prolonged interview sessions may strain participants, for example leading to concentration fatigue (Collins 2015). When developing an interview guide for the SSI, it is therefore important to consider the length and scope of the interview and reflect on how conducting an SSI might affect the quality of the CI and vice versa. Practically, it might be advantageous to incorporate a break between the two.

In light of the qualitative testing of questionnaires to secure content validity, the approaches of cognitive and semi-structured interviews highlight essential and interesting inherent philosophical and epistemological conflicts. By simply following a positivistic approach to the cognitive interview, the researcher might be led astray in her effort to address the heart of the matter of the recommended qualitative questionnaire testing: the content validity. Content validity, as we have seen, has a high degree of phenomenological weight, thus requiring skills and sensitivity in this epistemological approach to the generation of knowledge. By being aware and transparent as to the inherent assumptions in the methods applied, researchers working with examining spirituality in various populations using questionnaire instruments will have a better chance of securing the content validity of their questionnaires. This is crucial when addressing challenging concepts such as spirituality.