A Reliability Generalization Meta-Analysis of “What Is Happening in This Class?” (WIHIC) Questionnaire

Abstract

What is Happening In this Class? (WIHIC) is the widely used questionnaire to measure psycho-social aspects of the classroom and explore how these aspects affect student learning and achievement. The purpose of this study is to examine the cumulative estimates of reliability and conduct reliability generalization meta-analysis of Cronbach’s alpha for the WIHIC questionnaire. PRISMA framework is used to identify the papers in three major databases. Assuming a random-effects model, the average internal consistency reliability was 0.85, 95% CI [0.83; 0.87] for total scores and ranged from 0.80 to 0.88 for subscales. There was a substantial heterogeneity among the included articles (I² = 99.04%, Q (23) = 1481.074, p < 0.001). According to mixed model analysis, school context has a significant effect on the total scale and subscales, including teacher support, involvement, investigation, cooperation, and equity. Overall, the reliability generalization analysis of pooling reliability estimates helps in understanding the psychometric properties of the WIHIC inventory in diverse populations.

1. Introduction

In recent years, there have been many attempts at school and curriculum reforms in education systems in every country. Educational planners increasingly recognize that success in educating the young generation depends on how well educators and policymakers understand the symbiotic relationships between student learning and social and emotional factors and learning environments [1]. As most instruction takes place in the classroom, it is crucial to examine the nature of the classroom environment, how it functions, and the dynamics involved in the process. Studies on the effects of the learning environment on student outcomes have grown in the last three decades and have been established as one of the critical aspects of educational research [2]. The researchers investigated the relationships between students’ and teachers’ perceptions of their learning environments, attitudes toward specific subjects, and cognitive outcomes in diverse, multi-cultural settings.

Since the conception of learning environments as a prodigious field of educational research, a considerable numbers of self-report instruments have been developed to measure students’ perceptions of the classroom climate [3]. Earlier attempts to conduct research in the learning environment used Learning Environment Inventory (LEI) [4], Questionnaire on Teacher Interaction (QTI) [5], and Science Laboratory Environment Inventory (SLEI) [6]. The most frequently used learning environment instrument is the What is Happening In this Class? (WIHIC), originally developed by [7]. The WIHIC combined relevant features from a wide range of existing questionnaires with additional scales that accommodate contemporary educational thinking, such as equity and constructivism. The WIHIC has 56 items that are divided equally into the seven scales. The internal consistency reliability values for each subscale are 0.81 for student cohesiveness, 0.88 for teacher support, 0.82 for involvement, 0.83 for investigation, 0.89 for task orientation, 0.67 for cooperation, and 0.81 for equity.

Table 1. Scale Description for each Scale and Example Items in the WIHIC Questionnaire.

Scale	Description	Item
Student Cohesiveness	Extent to which students are friendly and supportive of each other.	I make friendships among students in this class.
Teacher Support	Extent to which the teacher helps, befriends, and is interested in students.	The teacher takes a personal interest in me.
Involvement	Extent to which students have attentive interest, participate in class, and are involved with other students in assessing the viability of new ideas.	I discuss ideas in class.
Investigation	Extent to which there is emphasis on the skills and of inquiry and their use in problem solving and investigation.	I carry out investigations to test my ideas.
Task Orientation	Extent to which it is important to complete planned activities and stay on the subject matter.	Getting a certain amount of work done is important to me.
Cooperation	Extent to which students cooperate with each other during activities.	I cooperate with other students when doing assignment work.
Equity	Extent to which the teacher treats students equally, including distributing praise, question distribution, and opportunities to be included in discussions.	The teacher gives as much attention to my questions as to other students’ questions.

shows the scales in the WIHIC, along with a brief descriptor of each scale and sample items in the questionnaire.

The WIHIC questionnaire has been extensively used in all levels of education in different countries [8]. The studies have shown that the questionnaire has strong reliability and validity across various contexts. In a recent study, [9] examined the preservice teachers’ perceptions of learning environments in a public university in Texas, USA, before and after pandemic-related course disruption. The study used WIHIC to collect quantitative data from 230 teacher education students to explore changes in student perceptions of the learning environment from before to after the switch to remote learning due to the pandemic. The study found a decline in student cohesiveness, teacher support, involvement, task orientation, and equity scales.

In another study, [10] used scales from the WIHIC to measure the perceptions of nursing students on the effect of cooperative learning on academic achievement and the learning environment. The study found a significant difference between the academic achievement of students and their perception of the classroom environment in the experimental and control groups in favor of cooperative learning. [11] developed the New What Is Happening In this Class? (NWIHIC) instrument based on the WIHIC questionnaire by adding two more scales: differentiated instruction and ongoing assessment. The questionnaire was administered to 2556 grade 5 to grade 9 students in China. The study reports the grade and gender differences in the perceptions of their learning environment.

Bizimana et al. [12] investigated whether students taught biology using cooperative mastery learning had different perceptions of learning environment and engagement when compared to those taught using conventional teaching methods. The study involved 298 students, and a modified What Is Happening In this Class? (WIHIC) and Student Engagement Questionnaire (SEQ) were used to collect data. The results indicated a significant difference in the perceptions of the learning environment measured by these two instruments. The cooperative mastery learning group students were perceived as higher on all scales compared to conventional teaching methods group students. In Lithuania, Brandisauskiene et al. [13] highlighted the need for a sustainable school system to ensure the well-being of the young generation. The authors assessed the sustainable school environment variables and students’ emotional and behavioral engagement in learning with the What Is Happening in this Class? (WIHIC) questionnaire, a short form of the Learning Climate Questionnaire (LCQ), and the Student Engagement Scale. Cronbach’s alpha coefficients for each subscale are 0.915 for student cohesiveness, 0.928 for teacher support, 0.906 for cooperation, 0.931 for equity, 0.929 for learning climate, 0.892 for affective engagement, and 0.832 for behavioral engagement. Cronbach’s alpha results suggest that all subscales had an acceptable internal consistency. They found that emotional engagement and behavioral engagement correlated significantly with all WIHIC scales except task orientation. These studies illuminated that the WIHIC questionnaire can be used to examine various dimensions of a learning environment.

Therefore, many studies used the WIHIC questionnaire, reported to be a valid and reliable tool to assess students’ perception of the learning environment. To the best of our knowledge, there have been little to no studies that systematically provide an evaluation of the pool estimates of the internal consistency reliability of WIHIC. Therefore, we used the reliability generalization method, which is the application of meta-analysis in exploring the variability in the scores of reliability estimates of WIHIC. Reliability generalization is one of the methods of combining and analyzing the reliability coefficient alpha value from multiple empirical studies [14]. We also investigated the various characteristics of the studies that might affect the reliability estimates.

2. Materials and Methods

To answer the research questions, we conducted a literature review search and analysis according to the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) guidelines [15]. PRISMA is a widely used framework for reporting and synthesizing literature review following four steps: (1) identifying research literature from database searches, (2) screening articles using inclusion and exclusion criteria, (3) assessing full-text articles for eligibility, and (4) analyzing and reporting the final articles including in the review.

2.1. Data Sources and Literature Search

An initial search of the literature was conducted through three databases: ProQuest, Scopus, and Web of Science. The time frame for this review was from 1996 to May 2022, limiting to the period when Fraser et al. published the WIHIC questionnaire in 1996. The combination of keywords WIHIC and “learning environment” was used to identify the papers in every database. The detailed search strategy syntax used for each database can be seen in

Table 2. Search Strategy Syntax.

Database	Syntax	Number of Articles
ProQuest	WIHIC AND “learning environment”	133
Scopus	TITLE-ABS-KEY (WIHIC) AND TITLE-ABS-KEY (“learning environment”)	68
Web of Science	WIHIC (all fields) AND “Learning environment” (all fields)	34

. Articles were included in this review if they were published in peer-reviewed journals in English. There were no restrictions regarding the design of studies: quantitative, qualitative, or mixed method.

2.2. Study Selection

The literature search identified a total of 240 articles (see Figure 1). As the purpose of the review is to verify the reliability of the WIHIC questionnaire and examine the characteristics of the studies that might affect the reliability estimates, we determined the inclusion and exclusion criteria based on PICOS (Population, Intervention, Comparison, Outcomes, and Study design) format (see

Table 3. Inclusion and Exclusion Criteria based on PICOS.

	Inclusion	Exclusion
Population	School level	Not university or college level
Intervention	NA	NA
Comparison	NA	NA
Outcomes	Reported Cronbach’s alpha level: whether it is the overall alpha or alpha of the subscales for the WIHIC questionnaire	No Cronbach’s alpha value
Study Design	All empirical papers	Not review, discussion, or theoretical papers Non-English papers Not peer-review papers

). After removing the duplicates, the titles and abstracts of the articles were reviewed by the first and second authors if they met the inclusion/exclusion criteria. The last author skimmed and scanned the full papers and assessed their relevance based on the above criteria. Then, the first and second authors reviewed the outputs and finally decided the relevant articles. After applying the PICOS criteria, 84 articles remained for full-text review.

2.3. Approach to Analysis and Synthesis

The 84 full-text articles were reviewed if there was clear information about the use of WIHIC questionnaire and presented the reliability value. Papers that did not present the Cronbach’s alpha value were excluded for final analysis and synthesis. In addition, the articles were excluded if the full text of the articles were presented in another language with their abstracts in English. Following the same shortlisting and consensus-building process above, 24 articles remained for quantitative evidence synthesis.

The random-effects model was performed using a restricted maximum-likelihood method to calculate the cumulative estimates of Cronbach’s alpha and confidence intervals. The common measures for heterogeneity, including I² and Q statistics, were reported. The publication bias was assessed by fail-safe N analysis using the Rosenthal approach. A mixed-effect model was conducted to examine the role of moderator on the overall estimate of Cronbach’s alpha. The analysis was conducted using Jamovi software (Version 2.3.13).

3. Results

We provided a summary of the 24 articles prior to presenting the answers to our research questions. This summary provides an overview of the characteristics of the reported research.

3.1. Characteristics of the Included Articles

This section presents the characteristics of the included articles, including study location; school level, research design, and the number of WIHIC sub-constructs in their studies (see Table 4).

Study location. The twenty-four studies come from thirteen different countries, with eight studies (33%) conducted in the USA, eight studies in Asia, three studies in Australia, and three in Africa. One study was reported for countries such as Turkey, Taiwan, and Australia.

School level (context). Most studies were conducted in secondary schools (83%). Two studies were conducted in primary schools (8%). Two studies were from mixed contexts (primary and secondary schools).

Research design. Eighteen studies (75%) employed a quantitative design. All quantitative studies employed a survey approach. Six studies (25%) employed mixed methods, and these employed more than the survey method, including interviews and observations.

WIHIC questionnaire. In the included studies, fifteen studies used the full version or the shortened form of the original scale. Meanwhile, nine studies used the translated version. In the translated version, the questionnaire was translated into Chinese [16,17,18], Spanish [19,20], Arabic [21], Turkish [22], Korean [23], and Indonesian [24]. In terms of the subscales, the included studies used from five to seven subscales in their studies, as the original WIHIC questionnaire included seven subscales, and the shortened version included five subscales.

Table 4. Characteristics of the included articles.

Authors	Country	Research Design	Methods	School Context	Translated/Original	Subscales
Adamski et al. [20]	USA	Quantitative	Survey	Mixed	Translated (Spanish)	6
Aldridge et al. [16]	Taiwan and Australia	Mixed-method	Survey, interviews, observations	Secondary	Translated (Chinese)	7
Allen and Fraser [25]	USA	Mixed-method	Survey, interviews, observations	Mixed	Original	6
den Brok et al. [26]	USA	Quantitative	Survey	Secondary	Original	7
den Brok et al. [22]	Turkey	Quantitative	Survey	Secondary	Translated (Turkish)	7
Charalampous and Kokkinos [27]	Africa	Mixed-method	Survey, interview	Primary	Original	7
Chionh and Fraser [28]	Asia	Quantitative	Survey	Secondary	Original	7
Dorman [29]	Australia	Quantitative	Survey	Secondary	Original	7
Dorman [30]	Australia	Quantitative	Survey	Secondary	Original	7
Helding and Fraser [31]	USA	Quantitative	Survey	Secondary	Original	7
Khalil and Aldridge [21]	Asia	Quantitative	Survey	Secondary	Translated (Arabic)	5
Kim et al. [23]	Asia	Quantitative	Survey	Secondary	Translated (Korean)	7
Koul and Fisher [32]	Asia	Quantitative	Survey	Secondary	Original	7
Lim and Fraser [33]	Asia	Quantitative	Survey	Secondary	Original	6
Liu et al. [18]	Asia	Mixed-method	Survey, interviews, observations	Secondary	Translated (Chinese)	5
Opolot-Okurut [34]	Africa	Quantitative	Survey	Secondary	Original	5
Rita and Mar-tin-Dunlop [35]	USA	Mixed-method	Survey, interview	Secondary	Original	7
Robinson and Fraser [19]	USA	Quantitative	Survey	Primary	Translated (Spanish)	5
Shadreck [36]	Africa	Quantitative	Survey	Secondary	Original	7
Stein and Klosterman [37]	USA	Quantitative	Survey	Secondary	Original	6
Taylor and Fraser [38]	USA	Quantitative	Survey	Secondary	Original	7
Wahyudi and Treagust [24]	Asia	Quantitative	Survey	Secondary	Translated (Indonesian)	7
Waldrip et al. [39]	Australia	Mixed-method	Survey, interview	Secondary	Original	5
Yang [17]	Asia	Quantitative	Survey	Secondary	Translated (Mandarin)	7

Table 4. Characteristics of the included articles.

Authors	Country	Research Design	Methods	School Context	Translated/Original	Subscales
Adamski et al. [20]	USA	Quantitative	Survey	Mixed	Translated (Spanish)	6
Aldridge et al. [16]	Taiwan and Australia	Mixed-method	Survey, interviews, observations	Secondary	Translated (Chinese)	7
Allen and Fraser [25]	USA	Mixed-method	Survey, interviews, observations	Mixed	Original	6
den Brok et al. [26]	USA	Quantitative	Survey	Secondary	Original	7
den Brok et al. [22]	Turkey	Quantitative	Survey	Secondary	Translated (Turkish)	7
Charalampous and Kokkinos [27]	Africa	Mixed-method	Survey, interview	Primary	Original	7
Chionh and Fraser [28]	Asia	Quantitative	Survey	Secondary	Original	7
Dorman [29]	Australia	Quantitative	Survey	Secondary	Original	7
Dorman [30]	Australia	Quantitative	Survey	Secondary	Original	7
Helding and Fraser [31]	USA	Quantitative	Survey	Secondary	Original	7
Khalil and Aldridge [21]	Asia	Quantitative	Survey	Secondary	Translated (Arabic)	5
Kim et al. [23]	Asia	Quantitative	Survey	Secondary	Translated (Korean)	7
Koul and Fisher [32]	Asia	Quantitative	Survey	Secondary	Original	7
Lim and Fraser [33]	Asia	Quantitative	Survey	Secondary	Original	6
Liu et al. [18]	Asia	Mixed-method	Survey, interviews, observations	Secondary	Translated (Chinese)	5
Opolot-Okurut [34]	Africa	Quantitative	Survey	Secondary	Original	5
Rita and Mar-tin-Dunlop [35]	USA	Mixed-method	Survey, interview	Secondary	Original	7
Robinson and Fraser [19]	USA	Quantitative	Survey	Primary	Translated (Spanish)	5
Shadreck [36]	Africa	Quantitative	Survey	Secondary	Original	7
Stein and Klosterman [37]	USA	Quantitative	Survey	Secondary	Original	6
Taylor and Fraser [38]	USA	Quantitative	Survey	Secondary	Original	7
Wahyudi and Treagust [24]	Asia	Quantitative	Survey	Secondary	Translated (Indonesian)	7
Waldrip et al. [39]	Australia	Mixed-method	Survey, interview	Secondary	Original	5
Yang [17]	Asia	Quantitative	Survey	Secondary	Translated (Mandarin)	7

3.2. Reliability and Heterogeneity

Of the 24 eligible studies, no study reported a Cronbach’s alpha value for the overall scale. Therefore, an average alpha value was calculated by transforming each subscale alpha value into z values using Fisher’s z [40]. Then, the mean of those z values was calculated and back-transformed into Cronbach’s alpha value. The total sample size was 28,696 participants (range n = 81 to n = 3248), with M = 1196 and SD = 924. The main summary statistics for the alpha coefficients of WIHIC total scale and seven subscales can be seen in

Table 5. Mean alpha coefficients, 95% confidence intervals, and heterogeneity statistics for WIHIC total scale and the seven subscales.

Total Scale/Subscale	n	α	LL	UL	Q	I2
Total Scale	24	0.85	0.83	0.87	1481.074 *	99.04%
Student Cohesiveness	22	0.80	0.77	0.84	4705.848 *	99.58%
Teacher Support	23	0.87	0.85	0.90	1920.505 *	99.32%
Involvement	24	0.85	0.82	0.87	1280.051 *	99.1%
Investigation	16	0.86	0.83	0.89	1914.472 *	99.4%
Task Orientation	22	0.82	0.79	0.86	4439.097 *	99.55%
Cooperation	22	0.86	0.84	0.89	5675.091 *	99.51%
Equity	22	0.88	0.86	0.91	2314.499 *	99.47%

Notes. n, number of studies; LL, lower limit; UL, upper limit; Q, Cochran’s heterogeneity Q statistic; I², heterogeneity index. * p < 0.001.

.

From the estimate of the random effects meta-analysis, the mean of total scale alpha coefficients is 0.85, 95% CI [0.83; 0.87]. Regarding student cohesiveness subscale, 22 studies totaling 28,365 participants were meta-analyzed and yielded an estimate of α = 0.80, 95% CI [0.77; 0.84]. Regarding the teacher support subscale, 23 studies containing a total of 25,827 participants were meta-analyzed and yielded an estimate of α = 0.87, 95% CI [0.85; 0.90]. Regarding the involvement subscale, 24 studies comprising 28,696 participants were meta-analyzed and yielded an estimate of α = 0.85, 95% CI [0.82; 0.87]. Regarding the investigation subscale, 16 studies comprising 19,738 participants were meta-analyzed and yielded an estimate of α = 0.86, 95% CI [0.83; 0.89]. Regarding the task orientation subscale, 22 studies comprising 27,662 participants were meta-analyzed and yielded an estimate of α = 0.82, 95% CI [0.79; 0.86]. Regarding the cooperation subscale, 22 studies containing a total of 24,928 participants were meta-analyzed and yielded an estimate of α = 0.86, 95% CI [0.84; 0.89]. Regarding the equity subscale, 22 studies totaling 23,372 participants were meta-analyzed and yielded an estimate of α = 0.88, 95% CI [0.86; 0.91]. Comparing the mean reliability coefficients of these subscales, the teacher support subscale yielded largest estimates (M = 0.87), while student cohesiveness estimated the poorest average reliability (M = 0.80).

Heterogeneity among reliability coefficients was examined by calculating the Cochran’s heterogeneity (Q) test and the heterogeneity index (I²) and constructing a forest plot. Table 5 presents high heterogeneity among the included articles (p < 0.001) and large I² indices (>99%) for the total score and the subscales. Figure 2 presents a forest plot of alpha coefficients for WIHIC total score in each study. Therefore, it can be seen that there was a substantial heterogeneity, Q (23) = 1481.074, p < 0.001, I² = 99.04%, τ² = 0.0027. To address potential publication bias, a funnel plot was produced with a follow-up of the Egger’s test. Funnel plot asymmetry can be suggested by Egger’s test producing a p-value less than 0.001 (see Figure 3).

3.3. Analysis of Moderator Variables

Mixed-model analysis was conducted using year, country, research design, school context, original/translated version of the questionnaire, and number of subscales as the moderators. It would be of benefit to examine the potential impact of these demographic factors was used. Among these factors, this study found that school context has significant effect on the overall estimate (z = 2.72, p < 0.01, 95% CI [0.028; 0.173], R² = 18.74%). No significant effect was seen with year (z = 0.026, p > 0.01, 95% CI [−0.164; 0.169], R² = 0%), country (z = 0.596, p > 0.01, 95% CI [−0.055; 0.103], R² = 0%), research design (z = −0.877, p > 0.01, 95% CI [−0.071; 0.027], R² = 0%), original/translated version (z = 0.434, p > 0.01, 95% CI [−0.035; 0.054], R² = 0%), and number of subscales (z = 0.419, p > 0.01, 95% CI [−0.058; 0.089], R² = 0%).

Separate analyses were conducted for each of the subscales. As presented above, school context has significant effect on the total scale (p < 0.01), and significant effect was also found for the subscales including teacher support (p = 0.020), involvement (p < 0.001), cooperation (p = 0.036), equity (p < 0.001), and investigation (p = 0.006). For the research design as the moderator, the total score of the reliability estimate showed a non-significant p-value (p = 0.38), but there was statistically significant value for the equity subscale (p = 0.005). Similarly, in terms of original/translated version of the questionnaire as the moderator, although there was no significant value for the total scale (p = 0.665), a significant effect was found for the task orientation subscale (p = 0.030). It can be concluded that the reliability of the total scale and subscales excluding student cohesiveness and task orientation were influenced by school context. Furthermore, the equity subscale was influenced by research design of the included studies, and the task orientation subscale was influenced by original/translated version of the questionnaire.

4. Discussion and Conclusions

We focused on the alpha coefficient, which is a commonly reported method in indicating internal consistency of an instrument and usually varies in administration of the instrument. Reliability generalization is used in our study in order to calculate the cumulative reliability estimates of the WIHIC questionnaire and identify study characteristics associated to the variability among the reliability coefficients.

Through the analysis of 24 included studies, the RG estimate of Cronbach’s alpha for the WIHIC total score showed a high reliability value (0.85) even after accounting for potential publication bias. According to [41], the Cronbach’s alpha value is acceptable when it is >0.70. Each subscale of WIHIC also showed high reliability value, and the minimum reliability value is also greater than 0.70 [19,32]. This suggests that the questionnaire can be used with confidence in whole or in part. However, high heterogeneity was seen in our RG analysis of the alpha coefficient, so the results must be interpreted with caution.

Among the demographic factors of the studies, school context (primary, secondary, and mixed) has significant effect on the total scale and subscales including teacher support, involvement, investigation, cooperation, and equity. In our study, we limited the school context into school level, as most WIHIC studies were conducted at university level, as can be seen in the literature [8,9,10]. Further research should be undertaken to include the university level in investigating the RG analysis of WIHIC inventory. In addition, among the subscales, research design (qualitative, quantitative, and mixed method) has significant effect on the equity subscale, and original/translated version of the questionnaire has significant effect on the task orientation.

This paper has its limitations, like most research. Unpublished literature and articles published in non-English were excluded. Further studies should extend the data sources to verify the claims that have been presented. In addition, most papers did not report the Cronbach’s alpha value, and some provided only limited information. The reliability estimates of WIHIC questionnaire should be provided whenever the questionnaire is used. In general, the RG analysis of pooling reliability estimates helps in understanding the psychometric properties of the WIHIC inventory in diverse populations. The analysis also helps in understanding the source of variation across studies.