Towards a Comprehensive School Effectiveness Model of Citizenship Education: An Empirical Analysis of Secondary Schools in The Netherlands

Abstract

We still have only a limited understanding of the effectiveness of schools in promoting citizenship, the factors explaining this effectiveness and the way in which these aspects interact. Using elaborate cross-sectional data from students, teachers, team leaders and school leaders at 78 Dutch secondary schools, this study empirically examines a school effectiveness model of citizenship education in order to achieve a more comprehensive explanation of citizenship competence acquisition. Using multilevel structural equation models, we analyze direct and indirect school-level predictors of student knowledge, attitudes and self-evaluated skills regarding citizenship. Four aspects of citizenship education are examined: the school’s policies regarding citizenship education, its teaching practices, and its professional and pedagogical learning environment (i.e., teaching community and classroom climate). With respect to school policies, positive effects are found for the attention paid to citizenship education in staff meetings. The professional learning environment is related to students’ citizenship competences mainly indirectly, via the average classroom climate. Effects of teaching practices vary: more emphasis on monitoring is more frequently found at schools with lower average levels of citizenship competences, whereas schools that let students choose their own topics in class have on average higher levels of citizenship competences.

1. Introduction

In many Western countries, the last two decades have witnessed an upsurge in the debate about the social outcomes of education. Social outcomes include social returns, social cohesion and social capital, and social and societal competences (Dijkstra et al. 2014a). The latter, which are often referred to as citizenship competences, comprise a range of attitudes, skills and knowledge related to democratic conduct, socially responsible behavior and the ability to handle differences and conflicts (Ten Dam et al. 2011; Ten Dam and Volman 2007; Westheimer and Kahne 2004). In many countries, politicians and society at large are increasingly paying attention to citizenship and how schools can contribute to it (European Commission/EACEA/Eurydice 2017). This development was inspired by the increasing diversification of society, declining social cohesion and the need to strengthen the foundations of the democratic society (Foa and Mounk 2016; Fukuyama 2014; Mounk 2017).

It is unclear how schools contribute to the promotion of citizenship competences and what produces effective citizenship competences. Several recent large-scale studies (e.g., CELS, ICCS and COOL12-18) suggest, however, that what schools do does matter. Various smaller datasets have also contributed to our slowly growing understanding of the relationship between education and the acquisition of citizenship competences by students (e.g., Amnå 2012; Dijkstra et al. 2015; Geboers et al. 2013; Isac et al. 2014; Keating and Janmaat 2016), an important observation being the substantial differences in citizenship outcomes between students from different social backgrounds and between academic and vocational tracks (Munniksma et al. 2017; Schulz et al. 2018).

While these studies have provided valuable insights into predictors of young people’s citizenship competences, these mainly concern classroom characteristics (such as an open classroom climate) and characteristics of the school’s context (such as the socioeconomic and ethnic diversity of its student population). Less is known about the potential contribution of more general effectiveness-enhancing factors that are known from the extensive tradition of school effectiveness research in the social domain (e.g., Hattie 2009; Reynolds et al. 2014), such as a school’s educational policies, the organization of education, the professional learning environment and the interactions between these and other previously examined aspects of citizenship education.

Several scholars have argued for a comprehensive school effectiveness model of citizenship education that combines all potential aspects of citizenship education (cf. Dijkstra et al. 2014a; Maslowski et al. 2009; Reichert and Print 2018; Sampermans et al. 2018) and includes relevant general factors from the school effectiveness research into cognitive (and to some extent, non-cognitive) outcomes. Despite earlier attempts, however, to formulate such a model, so far little effort has gone into the empirical testing of a school effectiveness model of citizenship education. Those studies that did succeed in analyzing conceptual frameworks of citizenship education effectiveness have been either exploratory (Scheerens 2011) or cover potential relevant school factors, such as the school policy level and professional learning environment, to some extent only (Isac et al. 2014).

In a critical analysis of current approaches to modelling educational effectiveness, Creemers and Kyriakides (2006) tend to agree on the need for more encompassing models, arguing that a dynamic model of educational effectiveness should at least be specific about the dimensions on which the measurement of school effectiveness is based, and should define the relations among these dimensions. Recent studies support this argument. Whereas direct effects of factors at the school level are often small, as is consistently shown for a wide range of school outcomes, including citizenship outcomes (Dijkstra et al. 2015; Isac et al. 2014), indirect effects seem to be more substantial, for instance through their influence on classroom-level factors such as the teaching practice (cf. Creemers and Kyriakides 2010; Kyriakides et al. 2010).

To sum up, although we are slowly obtaining a better understanding of factors that contribute to the acquisition of citizenship competences, the range of variables that have been investigated is still modest and comprehensive model estimations are largely absent. Moreover, few datasets with information on citizenship education contain broad and sufficiently elaborated sets of variables to empirically examine this type of model, including multiple levels, dimensions and both direct and indirect effects. Consequently, we only have a limited understanding of the effectiveness of schools in promoting citizenship competences. Building upon previously constructed theoretical models of school characteristics underlying social outcomes and citizenship competences in particular (Dijkstra et al. 2014a; Isac et al. 2014; Maslowski et al. 2009), the current study, therefore, focuses on estimating a comprehensive school effectiveness model of citizenship education. To this end, we used large-scale omnibus-like data on a broad range of aspects of citizenship education collected in 2016 at Dutch secondary schools.

In doing so, we are able to test a framework combining previously tested aspects of citizenship education—more specifically, a school’s teaching practices and average classroom climate—with aspects that have less often been empirically examined with regard to citizenship education, i.e., a school’s educational policies and its professional learning environment, while controlling for relevant student and school characteristics. Using multilevel structural equation models, we analyzed direct and indirect predictors of students’ citizenship knowledge, attitudes and self-evaluated skills. Following these lines, we strive to provide a better understanding of school characteristics that contribute to citizenship outcomes.

As in many other countries, Dutch schools have an obligation to improve “active citizenship and social integration” as is stipulated in legislation since 2005 (Dijkstra et al. 2014b). Schools are free, however, to organize citizenship education according to their own ideas, as long as they respect the basic values of democracy. What content they teach, how much attention they pay to promoting citizenship, and how they meet their citizenship goals (e.g., as part of other subjects, through projects, teaching it as a separate subject, etc.) is up to each school individually. Schools are also free to choose if and how to assess whether students have met their citizenship goals. As a result, schools differ greatly with respect to content, organization, and the quality of citizenship education, and most schools do not measure the effects of their teaching (Inspectorate of Education 2016). Promotion of citizenship competences might be found in curriculum elements, the school’s climate and/or aspects of its pedagogical approach, but it is often unclear whether and how various activities are related. Schools also differ in the outcomes of their citizenship education. Compared to other countries, outcome differences between schools are relatively large, including differences between schools offering vocational and academic tracks (Munniksma et al. 2017; Schulz et al. 2018). As a result of the high level of school autonomy and extensive differences between schools, the Dutch case is well suited to answer the research question of the current study.

2. Theoretical Framework

The school effectiveness model of school quality and social outcomes constructed by Dijkstra and colleagues (Dijkstra et al. 2014a) builds on assumptions taken from general effective school models (e.g., Creemers and Kyriakides 2007; Reynolds et al. 2014; Scheerens 2016). For the purpose of this study, we specifically focused on the acquisition of civic or citizenship competences as social outcome, which, as mentioned earlier, refer to a range of attitudes, self-evaluated skills (also referred to as self-efficacy) and knowledge related to democratic conduct, socially responsible behavior and the ability to handle differences and conflicts (Ten Dam et al. 2011; Westheimer and Kahne 2004). According to Dijkstra and colleagues (Dijkstra et al. 2014a), the social quality of a school concerns all aspects of quality contributing to the acquisition of social competences by students, including a focused approach (e.g., clear goals and coordination), school ethos (e.g., the alignment of shared values, teacher behavior and expectations), classroom climate, and content (both the formal curriculum and opportunities to practice). At the school level, a democratic learning environment (e.g., teachers’ participation and values in favor of learning) and democratic classroom climate (as visible in teacher–student and student–student interaction) are seen as substantial indicators of the quality of instruction. Additionally, the opportunity to learn about and practice democracy at school is considered an important element of effective citizenship education (cf. Dijkstra et al. 2014a; Isac et al. 2014; Maslowski et al. 2009; Scheerens 2011).

In the present study, we distinguish between four main types of school aspects related to citizenship education: (1) the school’s citizenship education approach and policies as manifested in the citizenship vision and the organization of citizenship education; (2) the quality of the (professional) learning environment, focusing on teacher behaviors, expectations and school leadership; (3) citizenship-related teaching practices and opportunities to practice; and (4) the pedagogical learning environment, including students’ perception of the classroom climate.

The school’s citizenship education policies. Educational effectiveness researchers consider school policies as one of the main indicators of the extent to which a school pays attention to a specific topic and hence the level of educational effectiveness (for an overview, see Creemers and Kyriakides 2010; Kyriakides et al. 2010). Examples of school policies are the school’s educational vision, the formulated guidelines, and the resources spent on the organization. School policies are believed to impact student outcomes both directly and indirectly, most importantly by providing guidelines and offering support to teachers and other stakeholders for the implementation of the policies in teaching practices and the learning environment. In a meta-analysis of studies on the dynamic model of educational effectiveness, Kyriakides et al. (2010) indeed found that effective schools were able to develop policies and take concrete action in order to improve their teaching practice and learning environment. In addition, educational effectiveness research emphasize the importance of school policies that support the improvement of these factors, such as resources for the professionalization of subject teachers (Hopkins and Reynolds 2001).

The above can be expected to apply to citizenship education too. Although research on this particular aspect of school is limited, a previous study found that students at schools that had formulated clear visions on citizenship education were more positive about their citizenship skills and also reflected on citizenship themes more often. Reflection on citizenship themes was also more frequent among students from schools that emphasized the learning of social skills. No effects were, however, visible for citizenship knowledge or citizenship attitudes (Dijkstra et al. 2015). In view of relatively modest school effects on citizenship competences as shown by earlier studies (Isac et al. 2014; Schulz et al. 2018), we expect that the effects of school policies on citizenship education will be mainly indirect, by way of their impact on other processes such as the teaching practice and classroom climate. To examine this, we include in our model both school policy-related aspects (the importance attached to citizenship themes) and organizational aspects (the attention paid to citizenship education in staff meetings and the resources available for the organization of citizenship education).

Professional learning environment. The school environment intermediates between the inputs and outputs of a school by functioning as a social system (Hofman et al. 1999). Another important aspect of the school context promoting educational effectiveness is, therefore, the extent to which a positive learning environment has been created at the school (Creemers and Kyriakides 2010; Kyriakides et al. 2010). A distinction is often made between the professional and pedagogical learning environment. The former focuses on the ‘professional’ community of the school, such as (the relationships between) its teachers and the school’s educational or administrative leadership. Various authors on citizenship education have emphasized the importance of a cohesive teacher community, with a strong sense of belonging, a shared vision, common values and practices, and committed to reconciliate potential conflicts (Dijkstra et al. 2014a; Isac et al. 2014; Maslowski et al. 2009; Scheerens 2011). Dijkstra and colleagues (Dijkstra et al. 2014a) use the term school ethos for this, referring to teacher behaviors and expectations towards each other, other staff members, as well as towards their students.

A meta-analysis of the impact of school factors on student achievement found that school leadership did not influence student outcomes directly, or only in a minor way. The authors suggest to focus instead on the impact of the ‘end result’ of school leadership, such as the development of teaching policy (Kyriakides et al. 2010). Important in this respect is paying attention to, for example, the conditional aspects of effective teaching through the formulation of specific citizenship learning goals or providing sufficient time for citizenship teaching. The same can be expected for the expectations of teachers towards each other, and towards their students. Not only do teachers serve as important role models towards their students—especially when it comes to the development of citizenship skill—their expectations and interactions also contribute to a positive classroom climate and the social safety that is required to establish a culture fostering professionalization and growth. Willemse and colleagues (Willemse et al. 2015), for example, found that collaborating and exchanging ideas among colleagues strengthened the relevance attached to citizenship education and its (implicit) presence in teaching practices. In addition to school policies on citizenship education as described above, the model, therefore, includes three elements of the professional learning environment: teachers’ experienced support from the school leadership, their interactions with each other, and their expectations of their students.

Citizenship teaching. A third aspect argued to be crucial for educational effectiveness is the quality of the educational content or teaching and learning practices (Creemers and Kyriakides 2010; Hattie 2009; Kyriakides et al. 2010). Teaching and learning practices include, for example, the pedagogical behavior and teaching methods of teachers, their opinions about education, curriculum content, the opportunities offered for participation in extracurricular activities, and the assessment or monitoring of that what is taught. These practices can also be expected to be important in the domain of citizenship. Opinions differ, however, on the relative impact of citizenship teaching practices. Whereas some authors claim that formal citizenship education through classroom instruction—mainly aimed at stimulating citizenship knowledge—plays an important role, others argue that this type of citizenship education is not enough to promote active, democratic citizenship (for an overview, see Maurissen 2018).

Many studies on citizenship education stress the importance of the type of teaching or learning strategy, highlighting the role of experience-based, active learning strategies (Schuitema et al. 2007; Veugelers 2009). Examples are student participation in school policies, simulations of democratic processes or roleplaying, service-learning and extracurricular activities. Results involving the effects of service-learning and extracurricular activities are, however, mixed (Geboers et al. 2013; Hoskins et al. 2012; Keating and Janmaat 2016). According to recent studies, these mixed results can be explained by the extent of attention paid to in-depth reflection during these activities, and when discussing citizenship-related themes in general. (Knowles et al. 2018; Reichert and Print 2018; Schuitema et al. 2017; Van Goethem et al. 2014). Other classroom practices seen as affecting citizenship competences are those that focus on students sense of ownership and decision-making powers, for example by creating opportunities for them to give their opinion on curriculum content (Bron 2018; Torney-Purta et al. 2008). Finally, the assessment of student outcomes with regard to citizenship education has been argued to matter (Keating et al. 2010).

To be able to provide more insight on the effectiveness of teaching and learning practices and their relation to other aspects of citizenship education, as well as to different citizenship outcomes, we look at a wide variety of practices, ranging from the citizenship themes addressed in class, opportunities for students to choose their own preferred themes in class and role playing in class, to the monitoring of students’ citizenship competences, extracurricular activities and outside school projects.

Classroom climate. The characteristics of the pedagogical learning environment—or more specifically, the classroom climate—is one of the most frequently studied aspects of citizenship education. Research on citizenship education has shown that an open climate in the classroom is one of the most consistent predictors of students’ citizenship competences (Geboers et al. 2013; Maurissen 2018). An open classroom climate is one where students experience their classrooms as safe places to investigate social and political issues and to explore and (respectfully) discuss their opinions and those of their peers (cf. Torney-Purta et al. 2001). Such a climate was found to be a necessary requirement for citizenship education to be effective (Knowles et al. 2018; Maurissen 2018). In addition to an open discussion climate, other aspects that have been argued to be positively related to citizenship outcomes are supportive interpersonal relationships, both among students and between students and teachers (Sampermans et al. 2018; Wanders et al. 2019), and strong feelings of school belongingness (Isac et al. 2014; Maslowski et al. 2009; Scheerens 2011). The relevance of a positive classroom climate is thus related to both a safe atmosphere as a condition for learning (cf. Hattie 2009) and a setting in which people are encouraged to form opinions (cf. Geboers et al. 2013; Isac et al. 2014). We examine this by including students’ experiences of the room for discussion, their view on the support from teachers, and their feelings of belongingness at school.

Research Question and Hypotheses

The goal of the current study is to give a general impression of the factors contributing to the promotion of citizenship competences by schools. To this end, we will provide a rigorous empirical test of a comprehensive school effectiveness model of citizenship education based on a broad estimation of potentially relevant school effects of factors that seem specifically relevant to the acquisition of citizenship competences, more general quality aspects that may be expected to have an indirect influence and the interplay between the pertinent variables. The research question is as follows: What school characteristics contribute either directly or indirectly to an explanation of differences in students’ citizenship competences?

Based on the above overview of the available knowledge, we formulated the following hypotheses.

Hypothesis 1 (H1).

Students’ citizenship competences are positively related to how much attention is paid to citizenship education in the school’s policies, both (H1a) directly and indirectly via the school’s teaching practices (H1b) and via the classroom climate as experienced by the students (H1c).

Hypothesis 2 (H2).

Students’ citizenship competences are positively related to the professional learning environment of the school, both directly (H2a) and indirectly via the teaching practices (H2b) and the classroom climate as experienced by the students (H2c).

Hypothesis 3 (H3).

Students’ citizenship competences are positively related to the teaching practices of the school, both directly (H3a) and indirectly via the classroom climate as experienced by the students (H3b).

Hypothesis 4 (H4).

Students’ citizenship competences are positively related to the classroom climate as experienced by the students.

3. Methodology

Data. The model was tested using data from 78 Dutch secondary schools, 54 of which were part of a sample that was randomly drawn from a list of the full population of secondary schools that have third-grade classes. A stratified random sample of 100 schools was drawn, with a distinction being made between three school tracks: vocational, general, and mixed. For each school, two replacement schools were selected in the event that a school from the first or second sample did not want to participate. A total of 54 schools from this sample participated. In addition, 24 schools were approached via existing contacts to ensure a large enough sample size and sufficient power for the model estimations. The resulting sample proved to be representative of the Dutch secondary school population with respect to the distribution of school track, geographical location, sector (public, private-religious and private non-religious), level of urbanization of the school’s location and school size.

At each school, digital questionnaires were completed by a school leader, a team leader, 15 third-grade teachers (including the mentors of the participating students) and all students in 3 third grades. During the survey, trained test leaders were present to guide the process and answer questions. Students filled out two questionnaires. The first contained questions on their background, societal trust, classroom climate, and citizenship activities at school. In the second questionnaire, students’ citizenship competences were tested. Within each classroom, 14 different versions were distributed. The analyses in this paper are based on the questionnaires of 5172 students, 643 teachers, 62 team leaders and 49 school leaders in 78 schools. An overview of the main characteristics of the four respondent groups can be found in Appendix A.

Dependent Variables. Citizenship competences of students were measured using the Citizenship Competences Questionnaire (CCQ; for an extensive description, including information on its construct validity, see Ten Dam et al. 2011; for an analysis of social desirability bias, see Ten Dam et al. 2013). The CCQ distinguishes between four social tasks that are considered to be representative of citizenship practices among young people aged between 11 and 16 years: acting in a democratic manner, acting in a socially responsible manner, dealing with conflicts and dealing with differences (see Appendix B for the conceptual framework and a description of the content of the scales). The CCQ provides information on the knowledge, attitudes, and self-evaluated skills relating to these four social tasks.

As part of the present study, a new comprehensive test was developed to measure students’ citizenship knowledge (for a description, see (Ten Dam et al. forthcoming). This test was also based on the four social tasks. The knowledge test was comprised of multiple-choice questions with three response options. Students were asked, for example, when a country could be called undemocratic. The answer categories were (a) if political parties criticize each other, (b) if people have to pay high taxes, and (c) if people are not allowed to criticize the government. The students had to choose what they considered the best option. Correct answers were coded as 1. After item analysis, a reliable IRT scale (thetas) was constructed based on 163 items (accuracy of measurement Macc well over 0.90)1.

Citizenship attitudes and (self-evaluated) citizenship skills were measured with 4-point Likert scales. To measure attitudes, pupils were asked to what extent various statements applied to them (e.g., ‘People should listen to each other, even if they have different opinions’). The answer categories ranged from (a) not applicable at all to (d) very applicable. To measure skills, pupils were asked how well they could do certain things such as defending their opinions in a discussion. The answer categories ranged from not good at all (a) to very good (d). The reliability of both scales was high, with Cronbach’s alphas of 0.90 for attitudes (24 items) and 0.86 for skills (15 items).

Independent Variables. If multiple items were available to create citizenship education constructs, exploratory and confirmatory multilevel factor analyses were performed to examine patterns among the pertinent items and construct factor scores. The model fit indices of the constructed scales can be found in

Table 1. Model fit estimates of citizenship education factor scales.

Factor	Chi2	df	p	CFI	RMSEA	SRMR	Valid N
School level (N = 78) School policies on citizenship
Vision CE themes	7.557	5	0.182	0.961	0.102	0.057	49
CE organization	2.116	4	0.714	1.000	0.000	0.016	48
CE meetings	10.948	14	0.690	1.000	0.000	0.038	48
Professional environment
Support school leadership	8.446	5	0.133	0.985	0.100	0.026	69
Agreement teachers	7.534	5	0.184	0.987	0.086	0.026	69
Teacher trust in students	5.939	5	0.312	0.992	0.052	0.036	69
Teaching practices
CE themes in lessons	27.459	5	0.000	0.810	0.255	0.060	69
Monitoring citizenship	2.891	2	0.236	0.995	0.080	0.020	69
Student level (N = 5172) Experienced classroom climate1
Open discussion climate	108.417	10	0.000	0.982	0.044	0.024 | 0.033	5170
Teacher support	27.296	4	0.000	0.994	0.982	0.011 | 0.038	5171
School belonging	41.039	18	0.002	0.995	0.016	0.015 | 0.033	5164

Note: ¹ based on multilevel factor analyses, SRMR values are depicted for within | between.

below. A more extensive description of the items, response categories and factor loadings can be found in Appendix C. The various constructs are briefly described below.

School policies on citizenship education were measured by using information from principals. The school’s citizenship vision was operationalized as the importance attached to various citizenship education themes (e.g., learning about other cultures and learning about democracy), ranging from very unimportant (1) to very important (5). The organization around citizenship education comprised items querying, for example, whether a school regularly addressed citizenship education in staff meetings, whether a continuous learning line existed and whether arrangements were made for the organization of citizenship education. The answers could range from not applicable at all (1) to very applicable (5). The attention paid to citizenship in meetings was measured for seven types of meetings, with answer categories ranging from (almost) never (1) to (almost) always (5).

The professional learning environment was operationalized as the extent to which teachers: (i) felt supported by school leadership (e.g., by taking their opinions seriously); (ii) agreed on how to treat each other, their students and their work; and (iii) trusted (the competence of) their students. The answer range was totally disagree (1) to totally agree (5).

Teaching practices related to citizenship teaching were measured using information from team leaders and teachers. In the case of the teachers, the school-level means were calculated2. Teacher-based factor scales comprised (i) the amount of attention paid to citizenship themes in class, such as learning about other cultures and about democracy, with answer options ranging from no attention (1) to daily attention (5); and (ii) the extent to which teachers actively monitored their students’ citizenship development, with answers ranging from not applicable at all (1) to very applicable (5). Furthermore, three one-item constructs were used, measuring the extent to which teachers let students: (iii) choose the topics discussed in class; (iv) take part in roleplaying; and (v) work on projects for which they have to collect information outside the school (e.g., neighborhood interviews and small-scale research). Answer categories for these items ranged from almost never (1) to almost always (5). In addition, one one-item construct was submitted to team leaders to measure whether extracurricular citizenship activities were organized, with possible answers being no (0) and yes (1).

The perceived classroom climate was operationalized as individual-level and school-level means3 of student experiences with respect to (i) an open classroom climate when discussing societal or political topics, with answers ranging from almost never (1) to almost always (5); (ii) teacher support (e.g., ‘My teachers take the time to talk about what is important for me’), with answers from totally disagree (1) to totally agree (5); (iii) a sense of school belonging (e.g., ‘I feel part of this school’), with answer options totally disagree (1) to totally agree (5).

Control Variables. Previous studies have shown that students of various socioeconomic and ethnic backgrounds vary in their citizenship competences, have unequal access to various aspects of citizenship education and that the effects of citizenship education also vary in these groups (Geboers et al. 2014; Hoskins et al. 2017; Isac et al. 2014; Janmaat et al. 2014; Knowles et al. 2018; Neundorf et al. 2016; Reichert and Print 2018).

Furthermore, classroom composition in terms of the students’ social background also impacts citizenship outcomes (e.g., Deimel et al. 2019; Isac et al. 2011, 2014). We, therefore, controlled for the students’ socioeconomic status (SES) and ethnic background, both at the individual level (i.e., variation between students) and at the school level (i.e., variation between schools in terms of student composition). Student SES was indicated by the average number of books at home. Further, the students’ level of education, distinguishing between senior general and pre-university education (HAVO/VWO) and pre-vocational education (VMBO) was used as a proxy. Regarding ethnic background, a dummy variable was included for students with a non-Western migration background4.

In addition, we controlled for school size and the level of urbanization of the municipality which the school was part of. Descriptive statistics of dependent, independent and control variables can be found in

Table 2. Descriptive statistics (N = 78 schools; N = 5148 students).

Variable	Mean/%	s.e.	Min.	Max.	N
Student level (N = 5148)
Citizenship knowledge	0.998	1.155	−2.413	4.426	5062
Citizenship attitudes	2.852	0.425	1	4	5148
Citizenship skills	2.970	0.375	1	4	5070
Number of books at home	2.331	1.259	1	5	5148
Type of education
- Pre-vocational	0.436	-	0	1	5148
- Senior general/pre-university	0.564	-	0	1	5148
Non-Western background	0.180	-	0	1	5148
School level (N = 78)1
Vision CE themes	4.298	0.493	3.2	5	49
CE organization	3.096	0.990	1	5	48
CE meetings	2.503	0.650	1	4	48
Support by school leadership	3.498	0.354	2.48	4.24	69
Agreement teachers	3.371	0.301	2.64	4	69
Teacher trust in students	3.324	0.276	2.633	3.74	69
Citizenship themes in lesson	3.273	0.327	2.44	4.44	69
Monitoring citizenship	2.587	0.358	1.563	3.341	69
Student choose topics	2.670	0.341	1.556	3.090	69
Role playing	2.865	0.439	1.400	3.167	69
Projects outside school	2.620	0.361	1.750	3.500	69
Extracurricular activities	0.741	-	0	1	58
Open discussion climate	3.303	0.159	2.924	3.629	75
Teacher support	3.367	0.180	3.207	4.022	75
School belonging	3.585	0.151	3.167	3.896	75
School size	855.359	528.229	149	2511	78
Level of urbanization	3.487	1.336	1	5	78

Note: ¹ raw mean scores are depicted for the factor scales.

.

Method. First, a measurement model of citizenship education was constructed at the school level. Based on exploratory and confirmatory factor analyses in Mplus7 (Muthén and Muthén 1998–2017), we selected several constructs comprised of factor scales complemented by various one-items constructs, which together measured a broad range of citizenship education elements. In addition, multilevel factor analyses were performed to examine the various constructs measuring students’ citizenship experiences at the individual and school levels. In total, 11 factor scores were retained for further analyses (see Table 1 and Appendix C).

To keep the number of parameters below the number of clusters (N = 78), we had to build our models stepwise. Factor scores were saved and (combined with the one-item constructs and control variables) used for correlational analyses and multilevel structural equation modelling5. Variables measuring school policies, learning environment and teaching practices were then included in the model one by one in order to analyze both direct and indirect relationships via teaching practices and experienced classroom climate. Our outcome variables were analyzed in three separate structural equation models. To include missing values on our exogenous school-level variables, these variables were explicitly included in the model and given a distributional assumption.

4. Results

4.1. Variation in Citizenship Competences across Schools

An examination of the intraclass correlations (ICC) of our three dependent variables at the school level indicated an ICC of 0.304 for citizenship knowledge, 0.052 for attitudes and 0.038 for skills. This shows that schools differ most in the average level of their students’ citizenship knowledge: 30 per cent of the variation in citizenship knowledge is explained at the school level. The between-schools variation with regards to the students’ mean citizenship attitudes and skills is significantly lower (5 and 4 per cent, respectively). Variance coefficients and intraclass correlations for models including control variables and independent variables can be found in Appendix D.

4.2. Relationships between Citizenship Education and Citizenship Competences

The pairwise correlations between the school-level constructs can be found in Appendix E. In addition to the expected positive correlations between the various variables corresponding to the same theoretical construct, we found that the school’s citizenship education policies, and the organization of citizenship education in particular, correlated positively with various citizenship teaching practices. A professional learning environment, on the other hand, correlated positively with the experienced classroom climate. Interestingly, it also correlated negatively with the attention paid to citizenship themes in class and the monitoring of student citizenship development. The latter two concepts also correlated negatively with the experienced classroom climate.

The results of the multilevel structural equation models for citizenship knowledge, attitudes and skills can be found in

Table 3. Total, direct and indirect effects on citizenship knowledge (standardized betas).

	Total		Total Direct		Total Indirect
	β	p	β	p	β	p
Student-level variables
Classroom climate
School belonging	0.007	0.607
Teacher support	0.002	0.890
Open discussion climate	0.014	0.303
Control variables
Number of books at home	0.105	0.000	0.106	0.000	0.001	0.788
Senior general/pre-university (ref. pre-vocational)	0.214	0.000	0.214	0.000	0.000	0.807
Non-Western background	−0.048	0.002	−0.048	0.002	0.000	0.959
School-level variables
Citizenship policies
CE themes school leader	−0.153	0.220	−0.123	0.437	−0.030	0.739
CE organization	0.026	0.801	0.107	0.439	−0.081	0.410
CE in meetings	0.030	0.800	0.110	0.399	−0.080	0.383
Professional learning environment
School leadership	0.008	0.936	0.048	0.670	−0.040	0.561
Teacher agreement	−0.051	0.556	−0.010	0.902	−0.041	0.478
Teacher trust	0.109	0.340	0.135	0.200	−0.026	0.737
Teaching practices
CE themes in class	0.014	0.845	0.010	0.894	0.004	0.829
Monitoring citizenship	−0.234	0.038	−0.248	0.023	0.014	0.774
CE projects	−0.165	0.071	−0.168	0.085	0.003	0.918
Role playing in class	0.022	0.801	0.022	0.820	0.001	0.984
Students choosing topics	0.195	0.017	0.199	0.029	−0.004	0.905
Extracurricular CE activities	0.186	0.069	0.183	0.101	0.003	0.876
Classroom climate
School belonging	0.248	0.079
Teacher support	−0.199	0.109
Open discussion climate	0.024	0.884
Control variables
Average books at home	0.346	0.000	0.186	0.108	0.161	0.149
Average senior general/pre-university	0.465	0.000	0.368	0.001	0.079	0.514
Average non-Western background	−0.261	0.008	−0.146	0.211	−0.115	0.247
School size	0.187	0.030	0.135	0.120	0.052	0.483
Level of urbanization	0.022	0.778	0.061	0.516	−0.039	0.616

Note: p (2-sided) < 0.05 (bold); p (2-sided) < 0.10 (bold italics); model fit: χ² (39) = 108.060, p < 0.001, RMSEA = 0.019, CFI = 0.941.

,

Table 4. Total, direct and indirect effects on citizenship attitudes (standardized betas).

	Total		Total Direct		Total Indirect
	β	P	β	p	β	p
Student-level variables
Classroom climate
School belonging	−0.007	0.627
Teacher support	0.001	0.953
Open discussion climate	−0.010	0.516
Control variables
Number of books at home	0.124	0.000	0.123	0.000	0.000	0.768
Senior general/pre-university (ref. pre-vocational)	0.065	0.000	0.064	0.000	0.000	0.591
Non-Western background	0.079	0.000	0.079	0.000	0.000	0.736
School-level variables
Citizenship policies
CE themes school leader	−0.174	0.098	−0.190	0.099	0.016	0.824
CE organization	0.028	0.797	0.030	0.833	−0.003	0.980
CE in meetings	0.141	0.234	0.270	0.022	−0.129	0.131
Learning environment
School leadership	0.119	0.217	0.018	0.876	0.101	0.274
Teacher agreement	0.142	0.217	0.185	0.056	−0.043	0.522
Teacher trust	0.173	0.110	−0.112	0.276	0.286	0.003
Teaching practices
CE themes in class	0.108	0.317	0.048	0.599	0.060	0.310
Monitoring citizenship	−0.216	0.083	−0.066	0.523	−0.151	0.110
CE projects	−0.034	0.765	0.086	0.397	−0.121	0.142
Role playing in class	0.198	0.079	0.142	0.185	0.056	0.506
Students choosing topics	0.162	0.149	−0.019	0.872	0.180	0.026
Extracurricular CE activities	0.091	0.457	−0.021	0.843	0.111	0.084
Classroom climate
School belonging	0.304	0.083
Teacher support	0.081	0.630
Open discussion climate	0.320	0.082
Control variables
Average books at home	0.449	0.001	0.044	0.690	0.404	0.001
Average senior general/pre-university	0.317	0.034	0.396	0.004	−0.079	0.674
Average non-Western background	0.417	0.000	0.653	0.000	−0.235	0.074
School size	−0.076	0.403	0.113	0.170	−0.190	0.046
Level of urbanization	0.054	0.636	0.040	0.718	0.014	0.907

Note: p (2-sided) < 0.05 (bold); p (2-sided) < 0.10 (bold italics); model fit: χ² (39) = 116.007, p < 0.001, RMSEA = 0.020, CFI = 0.920.

and

Table 5. Total, direct and indirect effects on citizenship skills (standardized betas).

	Total		Total Direct		Total Indirect
	β	P	β	p	β	p
Student-level variables
Classroom climate
School belonging	0.019	0.170
Teacher support	−0.005	0.753
Open discussion climate	−0.004	0.765
Control variables
Number of books at home	0.063	0.000	0.063	0.000	0.000	0.443
Senior general/pre-university (ref. pre-vocational)	0.021	0.220	0.021	0.224	0.000	0.704
Non-Western background	0.030	0.076	0.030	0.071	0.000	0.446
School-level variables
Citizenship policies
CE themes school leader	−0.453	0.001	−0.290	0.047	−0.163	0.088
CE organization	−0.085	0.559	−0.202	0.303	0.117	0.385
CE in meetings	0.352	0.002	0.597	0.000	−0.245	0.016
Learning environment
School leadership	0.185	0.098	−0.024	0.841	0.209	0.055
Teacher agreement	0.073	0.544	0.080	0.535	−0.007	0.928
Teacher trust	0.298	0.018	0.014	0.906	0.284	0.004
Teaching practices
CE themes in class	0.077	0.453	0.012	0.901	0.065	0.332
Monitoring citizenship	−0.149	0.359	0.023	0.875	−0.172	0.087
CE projects	−0.255	0.026	−0.107	0.321	−0.149	0.081
Role playing in class	0.016	0.895	−0.032	0.787	0.048	0.483
Students choosing topics	0.083	0.505	−0.118	0.357	0.201	0.013
Extracurricular CE activities	0.020	0.885	−0.101	0.407	0.121	0.082
Classroom climate
School belonging	0.180	0.447
Teacher support	0.284	0.273
Open discussion climate	0.308	0.145
Control variables
Average books at home	0.428	0.004	−0.005	0.974	0.434	0.006
Average senior general/pre-university	0.236	0.105	0.102	0.529	0.134	0.529
Average non-Western background	−0.180	0.370	0.175	0.354	−0.355	0.020
School size	−0.245	0.016	0.249	0.045	−0.383	0.004
Level of urbanization	0.226	0.120	0.273	0.050	−0.047	0.729

Note: p (2-sided) < 0.05 (bold); p (2-sided) < 0.10 (bold italics); model fit: χ² (39) = 109.294, p < 0.001, RMSEA = 0.019, CFI = 0.916.

.

Table 6. Conclusions with regard to the formulated hypotheses.

	Knowledge	Attitudes	Skills
Citizenship education policies
H1a		+	+
H1b
H1c
Professional learning environment
H2a		+
H2b			+ 1
H2c	+	+	+ 1
Teaching practices
H3a	+	+
H3b		+	+
Classroom climate
H4	+	+

Note: ¹ since only total indirect effects were found to be significant, no conclusions can be made regarding specific indirect pathways via teaching practices or classroom climate.

offers an overview of the conclusions, in which a plus sign indicates findings supporting the formulated hypotheses. Below, we will shortly describe the results for each aspect of citizenship education, as well as our control variables.

4.3. Citizenship Education Policies

Examination of Table 3 shows no total (direct or indirect) effects of the school’s citizenship policies on student citizenship knowledge. Further examination of the specific indirect effects did, however, reveal two indirect (borderline significant) negative effects of the amount of attention paid to citizenship themes via the monitoring of student citizenship development (β = −0.100, p = 0.094) and via citizenship projects outside the school (β = −0.071, p = 0.089). Both monitoring and outside school projects were found to be negatively associated with levels of citizenship knowledge (see paragraph 4.5 below). The amount of attention school leaders paid to citizenship themes was also negatively related to student citizenship skills (Table 5), both directly and indirectly (borderline significance).

The attention paid to citizenship education in staff meetings was positively related to both citizenship attitudes (Table 4) and skills (Table 5). Interestingly, in addition to this direct positive relationship between citizenship in staff meetings and citizenship skills, a negative indirect relationship was also found. Examination of the specific indirect effects did not reveal any individual significant indirect effect.

Based on these findings, Hypothesis 1a on the positive relationship between attention for citizenship education in school policies and students’ citizenship outcomes can only partially be confirmed: more attention paid to citizenship education in staff meetings is associated with more positive citizenship attitudes and more citizenship skills, yet not with more citizenship knowledge. Hypotheses 1b and 1c on positive indirect relationships between school policies on citizenship education and students’ citizenship outcomes—either via teaching practices (H1a) or via the classroom climate (H1b)—are not supported.

4.4. Professional Learning Environment

An examination of Table 3 revealed no significant total (direct and indirect) effects of a school’s professional learning environment on student citizenship knowledge. Examination of the individual indirect effects did reveal two (borderline significant) indirect effects. A negative indirect effect was found of teacher assessment of school leadership via the frequency of letting students choose their own topics, which was in turn positively associated with citizenship knowledge (β = −0.066, p = 0.086)—and a positive indirect effect of teacher trust in students via students’ feeling of school involvement (β = 0.098, p = 0.080).

A positive indirect effect of the teachers’ average trust in students was also found for citizenship attitudes (Table 4)—partially explained by a (borderline significant) positive indirect effect via the students’ average feeling of school involvement (β = 0.122, p = 0.080)—and for citizenship skills (Table 5). For the latter, none of the individual indirect pathways proved significant.

In addition, a (borderline significant) positive direct relationship was found between the extent of agreement on education between the teachers and student citizenship attitudes, and a (borderline significant) positive indirect relationship between the teachers’ opinions of school leadership and student citizenship skills. Again, none of the individual indirect pathways was significant.

These findings partially support Hypothesis 2a, on the direct positive relationship between the professional learning environment and student citizenship outcomes: more agreement between teachers on the provided education is associated with more positive citizenship attitudes of students. Hypothesis 2c on indirect positive relationships between the professional learning environment at school and citizenship outcomes via the classroom climate, is supported for all three types of citizenship competences. The findings provide no support for Hypothesis 2b, on an indirect relationship via teaching practices.

4.5. Teaching Practices

When we examined the total direct and indirect effects of teaching practices, several significant relationships came to the fore. First, letting students choose their own topics had a direct positive relationship with their citizenship knowledge (Table 3) and an indirect positive effect on both citizenship attitudes (Table 4) and citizenship skills (Table 5). For both attitudes and skills, our data did not show which aspects of the perceived classroom climate explained these relationships, since none of the indirect pathways were significant by itself.

For extracurricular citizenship activities, too, positive (borderline significant) effects were found on knowledge, attitudes and skills. Again, indirect effects were visible for both attitudes and skills (however, none of the individual pathways was significant), while for knowledge only the total effect proved significant.

Roleplaying in the class had a (borderline significant) positive relationship with student citizenship attitudes but not with citizenship knowledge or skills. Interestingly, monitoring of student citizenship development was negatively related to student citizenship knowledge and (with borderline significance) to student attitudes, and—indirectly—skills. Organizing citizenship projects for which students had to collect information outside the school (e.g., through neighborhood interviews or small-scale research) was also found to have a negative (borderline significant) relationship with both citizenship knowledge and skills. Although the latter effect was mainly indirect, none of the individual indirect pathways proved significant.

These findings provide mixed evidence for Hypothesis 3 on the positive relationship between teaching practices and citizenship outcomes. In line with our expectations, letting students choose their own topics and organizing extracurricular citizenship activities is positively related to their citizenship competences: a direct relationship is visible for citizenship knowledge (Hypothesis 3a), an indirect relationship for citizenship skills (Hypothesis 3b), and both direct and indirect relationships for citizenship attitudes. The opposite is true for monitoring student citizenship development and organizing outside school citizenship projects: both practices are found to be negatively related to citizenship outcomes, either directly or indirectly.

4.6. Perceived Classroom Climate

Of our three indicators of the perceived classroom climate, the students’ average feeling of school belonging was (borderline significant) positively related to their citizenship knowledge (Table 3) and attitudes (Table 4). An open discussion climate was also positively related students’ citizenship attitudes. No significant effects of classroom climate were found on student citizenship skills (Table 5).

These findings support Hypothesis 4 on the positive relationship between the perceived classroom climate and student citizenship outcomes, yet only for citizenship knowledge and attitudes.

4.7. Student Characteristics and School Context

In our final, comprehensive models of citizenship education, student SES, as indicated by the number of books at home showed positive effects: students from more favorable backgrounds had more citizenship knowledge (Table 3), more positive attitudes (Table 4) and more skills (Table 5). Levels of citizenship knowledge and attitudes were also higher for students from senior general and pre-university education than for students from vocational education. Moreover, students with a non-Western background had less citizenship knowledge than students with a Western background, but more positive attitudes and more skills.

Most of these effects were also present at the school level, with a few exceptions. At schools with more students with a non-Western background, citizenship skills were found to be lower. Furthermore, in addition to a positive direct effect of the number of students with a non-Western background on citizenship attitudes, a (borderline significant) negative indirect effect was present.

School size also mattered: levels of citizenship knowledge and skills were higher at larger schools, while scores on citizenship attitudes were lower. An indirect negative association between school size and citizenship skills was, however, also present. Finally, schools located in more urbanized areas reported, on average, higher scores on citizenship skills.

5. Conclusions and Discussion

In this paper, we empirically tested a comprehensive school effectiveness model of citizenship education, including a broad range of school factors that could be expected to explain the differences in the school’s citizenship outcomes either as general characteristics of effective schools or characteristics that specifically contribute to citizenship. In doing so, we followed up on calls for a more complete picture of the role of schools in the acquisition of citizenship competences, including school leaders’ vision and goals, teachers’ actual classroom practices, as well as the practices as perceived by students (Dijkstra et al. 2014a; Kerr et al. 2009). The citizenship outcomes examined concern both the cognitive dimension of citizenship (i.e., knowledge) and the active and affective dimensions (i.e., self-evaluated skills and attitudes). Applying multilevel structural equation modelling, we examined direct and indirect effects on student citizenship outcomes of citizenship school policies, the professional learning environment, citizenship teaching practices and the perceived classroom climate.

After controlling for student and school context characteristics, almost half of the investigated hypotheses on positive relationships between aspects of citizenship education and student citizenship knowledge, skills and particularly attitudes appeared to be supported—albeit effects were small. An examination of the magnitude of the effects furthermore indicated stronger relationships for students’ citizenship knowledge than for students’ citizenship attitudes and skills. The average level of citizenship knowledge also varied more between schools than it did for attitudes or skills. Even though school effects were small, various authors have emphasized that these effects are to be considered important, bearing in mind that a school is able to reach a large number of students, and that the underlying factors can be steered through the school’s policies (Dijkstra et al. 2014a; Isac et al. 2014; Sampermans et al. 2018; Reichert and Print 2018). At the same time, since the examined school factors seem to explain only a modest part of the variation found in students’ citizenship outcomes, we should be careful when drawing conclusions. With these reservation in mind, the following conclusions can be drawn from the results presented here.

Although the number of significant relationships is modest, we found evidence that aspects of school policies are important for students’ citizenship development. First of all, the importance attached to citizenship themes in school policies correlated with several teaching practices which in turn proved relevant to citizenship outcomes. In our explanatory analyses it was also shown that the attention paid to citizenship education in staff meetings related positively to student citizenship outcomes. At the same time, several negative relationships were found, in particular with regard to citizenship skills. Even though a more explicit emphasis on citizenship education in school policy might thus be part of the solution to effective citizenship education, it is not necessarily related to all aspects of effective citizenship education.

Our findings furthermore highlight that the importance of school climate and the characteristics of teacher–student and student–student interaction for the development of citizenship competences is not restricted to the experiences of students (often referred to as the pedagogical learning environment), but extends to the professional learning environment as well (cf. Willemse et al. 2015). Positive relationships were for instance visible for the teachers’ amount of trust in their students and students’ citizenship skills and attitudes. Relationships between student citizenship attitudes and the extent to which teachers agreed on the educational vision of the school (e.g., how to assess student results and interact with students) were also found, as well as between student citizenship skills and the extent to which teachers perceived support from their school leadership. Feelings of community created by and between staff members, of which the above elements can be considered examples, thus seem to be an important aspect of (effective) citizenship education. Looking at teaching practices, effects varied depending on the type of practice. The positive relationships found for letting students choose their own topics for upcoming lessons (on both knowledge, attitudes and skills) were the most robust. The positive results found for extracurricular citizenship activities also underline the potential influence of the school. Furthermore, citizenship attitudes were on average more positive at schools where roleplaying in class was more often encouraged.

Interestingly, at schools where teachers paid more attention to monitoring of the development of their students’ citizenship competences, students were found to have on average less citizenship competences than at schools where teachers had less insight into their students’ citizenship development. A good interpretation of these results, which seem to deviate from earlier findings (cf. Keating et al. 2010), requires more specific analyses of interpersonal interactions within the school with possible explanatory factors, such as the use of monitoring as a response to lagging results, disruptive behavior or the (more authoritarian) approach of teachers. The findings could also reflect the school’s efforts to more closely tutor struggling students, rather than a negative effect of monitoring per se. This is equally true for external projects (e.g., neighborhood interviews or small-scale research), which also appeared to be negatively related to students’ citizenship knowledge and skills. Both findings illustrate that further research is needed to investigate these and other aspects of citizenship education and their link to various citizenship outcomes. It would for instance be interesting to further examine whether the found relationships are typical for a certain school culture, context or school type.

Finally, in our analyses, the effects of classroom climate seem less pronounced than reported elsewhere (cf. Geboers et al. 2013). Although students’ average feelings of school belongingness were positively related to their citizenship knowledge and attitudes, an open discussion climate was only positively related to students’ citizenship attitudes, and no relationships were present for the experienced teacher support. As expected, supplementary analyses show that all three aspects of the perceived classroom climate relate positively to student citizenship outcomes when examined in separate univariate models (Appendix F). A notable finding (based on relevant correlations) is that classroom climate also seems to interact with the professional learning environment and with the in-class teaching. Therefore, the way in which these aspects interact is a relevant subject for further research aimed at supporting educational practice and policy.

In view of the cross-sectional nature of our data and the complexity of our model, it is too early to draw causal or final conclusions about the effectiveness of the elements of citizenship education that we examined. Although the models fit the data well and the variance explained is as could be expected, the stability of the models requires replication studies involving comparable data. It is important to consider that the relationships found, both the negative and the positive ones, are descriptions of correlations between the type of citizenship education adopted by the school (i.e., policies, learning environment, teaching practices and classroom climate) and the citizenship knowledge, attitudes and skills of their students. For causal interpretations, longitudinal research is necessary.

It should also be taken into account that we have focused on general effects, leaving group-specific effects aside. The results did, however, indicate a consistent positive relationship between socioeconomic status (measured by the number of books at home) and citizenship competences, not only at an individual level, but also at the school level. This points to a confirmation of findings from earlier studies that point out both unequal access to as well as varying effects of citizenship education with regard to students’ socioeconomic background (Hoskins et al. 2017; Janmaat et al. 2014; Neundorf et al. 2016). Moreover, it emphasizes the importance of paying particular attention to social inequality and disadvantaged students in relation to citizenship education. Future research focused on mechanisms explaining the differences found between students from various socioeconomic backgrounds and between schools that vary in their socioeconomic composition can provide support in this respect.

The model presented here provides a first step to further improve the effectiveness of citizenship education. In particular for youth that already start with a disadvantage, or students or schools that are not doing too well regarding citizenship competences, the current study provides guidelines on which citizenship education elements are associated with positive student outcomes, and which elements add to this. The model estimations illustrate the importance of a comprehensive explanation of differences in school effectiveness as expressed in citizenship outcomes, with not only direct effects being investigated but also the interplay between various factors such as teacher activities, the influence of the wider school setting in which teachers make their choices and how students perceive these. The relevance of citizenship for students and society and—as our study shows—what schools can contribute through a well-considered arrangement of teaching practice, contextual factors and school policies underlines the importance of the further development of a comprehensive school effectiveness model of citizenship education.