Fostering Pre-Service Physics Teachers’ Pedagogical Content Knowledge Regarding Digital Media

Abstract

Since the use of digital media opens new possibilities in physics education, pre-service teachers should develop pedagogical content knowledge (PCK) regarding digital media during teacher education. In the joint project DiKoLeP (German for: digital competencies of pre-service teachers in physics) of RWTH Aachen University, the University of Graz and the University of Tübingen, we therefore developed and implemented a university teaching concept with certain core elements to foster this digital-media PCK of pre-service physics teachers. The teaching concept was implemented as a university seminar at the three universities, considering the common core elements as well as individual curricular requirements. We evaluated the teaching concept in a pre-post-design with a recently developed knowledge test to measure digital-media PCK. Our findings indicated that the developed knowledge test is suitable to validly measure the digital-media PCK of pre-service physics teachers. Furthermore, first results of the empirical evaluation at the three participating universities showed a significant increase in students’ digital-media PCK with a small-to-medium effect for students who attended the theoretical as well as the practical sessions of the seminar.

1. Introduction

The use of digital media has opened new possibilities in education that can be leveraged in different ways. In physics teaching, for example, the use of digital media offers a new quality of visualisation to enhance students’ conceptual understanding [1,2]. To benefit from these possibilities in classrooms, pre-service physics teachers should develop ‘subject-specific digital competencies’ such as pedagogical content knowledge (PCK) regarding digital media or technology [3]. To foster this digital media-related PCK, effective learning opportunities need to be offered in physics teacher education.

In the following sections, the theoretical background of this article is outlined. First, the general framework of pedagogical content knowledge is described. Second, the digital media aspect of PCK is focused on. Third, possible ways of how this digital-media related PCK can be fostered in science teacher education are shown. By the end of this chapter, the main aims and research questions for this article are presented.

1.1. Modelling and Measuring Pedagogical Content Knowledge of Pre-Service Science Teachers

Shulman introduced pedagogical content knowledge as a part of teachers’ professional knowledge including “the ways of formulating the subject that make it comprehensible to others” and “an understanding of what makes the learning of specific topics easy or difficult” [4] (p. 9). Accordingly, PCK represents the knowledge that is required to teach subject-specific content understandably and effectively. This knowledge distinguishes a teacher from a non-teaching subject-matter expert as well as from a non-subject teacher [5]. Ever since Shulman’s introduction of PCK, this knowledge—its modelling, measuring, and development—has become an important topic in teacher education research.

Recently, a group of international researchers from the science education field developed the Refined Consensus Model of PCK in science education to overcome difficulties in comparing international research on PCK [6]. The model distinguishes between three interdependent realms of PCK: (1) collective PCK, (2) personal PCK, and (3) enacted PCK [6]. Collective PCK is shared knowledge about science education, e.g., by a group of educators or researchers. It is often based on literature and research findings but comprises even more than that as it also includes more local knowledge. Personal PCK refers to the state of knowledge of an individual teacher and reflects, for example, his or her own experiences in teaching. Enacted PCK is utilised by an individual teacher when acting during the practice of teaching science, i.e., in planning lessons, actual teaching, or reflecting on lessons and student outcomes [6]. As enacted PCK focuses on teachers’ actions, it cannot be measured adequately with (paper–pencil) knowledge tests. However, knowledge tests are suitable to measure personal PCK, as has already been done in several studies, e.g., refs. [7,8].

Models of teachers’ PCK often divide PCK into different facets. We understand facets to be the constituent components or subsets of knowledge that form teachers’ PCK. Matching Shulman’s definition, most of these models focus on two facets: instructional strategies and students’ understanding [9,10,11]. In physics, some models additionally contain the facet experiments either independently, e.g., ref. [12], or included in the facet instructional strategies, e.g., ref. [9]. In the German project ProfiLe-P (German for: professional knowledge of physics student teachers), the researchers developed a comprehensive model of PCK [13,14] by synthesising different existing models and frameworks for teachers’ PCK, e.g., refs. [10,15,16,17]. The comprehensive model considers eight facets of PCK in physics, one of them called ‘digital media’. However, a reduced model was used for the test development in ProfiLe-P, which did not include the digital-media facet [12]. Motivated by the growing impact of digital media in physics education, this paper focused on the digital-media facet of pre-service physics teachers’ PCK (digital-media PCK).

1.2. Pedagogical Content Knowledge Regarding Digital Media

Digital-media PCK includes knowledge about possibilities and limitations of content-specific use of digital media in teaching [13] and is therefore similar to the central technological pedagogical content knowledge (TPACK) of the TPACK framework [3]. The knowledge category TPACK represents the necessary knowledge for high-quality teaching with technology, including, e.g., the understanding of how using different technologies can help the learner to understand the subject [3]. The TPACK framework, however, is general and does not focus on a certain subject. Meaningful integration of digital media into teaching however needs to consider subject-specific requirements such as the use of digital data acquisition in physics [18]. Hence, subject-specific test instruments for TPACK or digital-media PCK need to consider subject-specific requirements. This lack of subject specificity is also seen in normative conceptualisations for digital competencies, e.g., the DigCompEdu framework [19]. Following that desideratum, a German working group of researchers (Working Group Digital Core Competencies) developed a framework for digital competencies specific to natural sciences called Digital Competencies for Teaching in Science Education (DiKoLAN) [18]. The DiKoLAN framework specifies digital competencies in seven competency areas. Four competency areas (Documentation, Presentation, Communication/Collaboration, and Information Search and Evaluation) are considered to be more general and three are more subject-specific for natural sciences (Data Acquisition, Data Processing, and Simulation and Modelling). The competencies for each area are described in tabular overviews based on the four technology-related knowledge dimensions of the TPACK framework: Special Tools (technological knowledge), Content-specific Context (technological content knowledge), Methods and Digitality (technological pedagogical knowledge), and Teaching (TPACK). When it comes to describing digital-media PCK in physics, the corresponding subject-specific competencies that focus on Teaching are particularly important.

Thus, an important part of digital-media PCK is understanding how to use digital media that are specific to a subject, in our case physics. Examples of physics-specific media and their usage are (i) mobile devices or systems for digital data acquisition in physical experiments [2,20], (ii) computer simulations as a subject-specific working method to gain knowledge in physics [18,21,22], or (iii) explanation videos for explaining complex physical contents [23]. Following the SAMR model [24], it is important to not simply substitute conventional tools with digital media but to modify and redefine learning tasks for students by realising the possibilities of digital media. Subject-specific knowledge about the possibilities and limitations of such media usages is required to use those media adequately in typical physics teaching situations, as knowledge can be seen as a disposition for action [25]. According to the Refined Consensus Model of PCK (cf. Section 1.1), actions regarding the use of digital media in real lessons demand enacted PCK regarding digital media, while knowing the possibilities and limitations of subject-specific digital media in typical subject-specific teaching situations represents parts of a teacher’s personal PCK. The latter, personal PCK regarding the use of digital media in physics lessons, is the knowledge we wanted to foster and investigate in this study.

1.3. Learning Opportunities to Foster Pre-Service Teachers’ Digital-Media PCK

To foster pre-service teachers’ digital-media PCK, it is important to prepare them for the use of subject-specific digital media in teaching. This can be achieved by following the synthesis of qualitative evidence (SQD) model [26]. The SQD model outlines seven key themes regarding the preparation of pre-service teachers to integrate digital media into their future teaching on the level of teaching courses (pp. 138–140):

• Aligning theory and practice
• Using teacher educators as role models
• Reflecting on attitudes about the role of technology in education
• Learning technology by design
• Collaborating with peers
• Scaffolding authentic technology experiences
• Moving from traditional assessment to continuous feedback.

Following this approach, university seminars aiming to foster digital-media PCK should thus combine theoretical parts with practical parts and provide pre-service teachers with the opportunity to plan and implement the use of digital media in authentic teaching situations. The pre-service teachers should collaborate, receive feedback, and reflect on their own experiences as well as on the role of digital media in education.

The SQD model outlines the structural and methodological design of seminars to effectively foster digital-media PCK in teacher education, but it does not specify the content (or media) to be addressed. Requirements regarding the content of such seminars could consider subject specificity by focusing in particular on the use of digital media that are specific or typical in physics teaching, as described in the previous section. The SQD model can also be implemented in a subject-specific course or seminar to create an effective intervention regarding the acquisition of subject-specific TPACK, e.g., ref. [27].

In the context of teacher education, many studies evaluating such learning opportunities regarding digital media use self-reports as their measuring method to assess pre-service teachers’ digital-media related knowledge or TPACK [28]. However, these instruments have been criticised in terms of validity [29,30] because they measure self-efficacy rather than actual professional knowledge [31].

1.4. Aims and Research Questions

In the joint project DiKoLeP (German for: digital competencies of pre-service teachers in physics) of RWTH Aachen University, the University of Graz and the University of Tübingen, we wanted to develop and evaluate a university teaching concept to foster pre-service physics teachers’ digital-media PCK. This immediately led to our first aim, which is presented in this paper: (1) Development of a teaching concept regarding the meaningful use of digital media in physics education that can be implemented in various universities. We implemented the teaching concept at each participating university as a seminar consisting of common core elements in rather theoretical sessions and university-specific practical sessions.

We evaluated the teaching concept, among other things, by investigating the development of the participating pre-service physics teachers’ digital-media PCK. To overcome shortcomings in validity of self-report-based instruments which are commonly used in this field, we chose a proximal measurement in our evaluation. This led to our second aim: (2) Development and validation of a knowledge test to measure pre-service physics teachers’ digital-media PCK. Following an argument-based approach to validation, we investigated different aspects of construct validity [32]. Finally, our third aim focuses on the evaluation itself: (3) Empirical evaluation of the teaching concept regarding pre-service physics teachers’ development of digital-media PCK. The following research questions arose from these last two aims:

• To what extent can the developed test instrument be used to validly measure digital-media PCK?
• Does the participants’ measured digital-media PCK increase across the seminars at the three participating universities?

As the teaching concept was designed to foster digital-media PCK, the corresponding hypothesis to the second research question is that the digital-media PCK increases when the pre-service physics teachers attend the seminars following the developed teaching concept.

2. Materials and Methods

2.1. Development of the Teaching Concept

We developed the teaching concept in our project (aim 1) based on the SQD model (cf. Section 1.3) by applying the key themes of the SQD-model to the whole concept as well as to individual seminar sessions. For example, according to the SQD model, the teaching concept should align theory and practice (key theme 1). We took this into account by including more theoretical as well as more practical sessions in the teaching concept. Moreover, theory and practice are aligned within a session as some smaller practical tasks were integrated in the rather theoretical sessions of the concept. To provide authentic technology experiences (key theme 6) and collaboration with peers (key theme 5), the pre-service teachers can practice using digital media in teaching situations and work together in pairs or groups. These practical experiences as well as the role of digital media in teaching in general are reflected in discussions and by feedback (key themes 3 and 7).

We chose the seminar’s content and digital media by conducting a needs analysis among pre-service physics teachers and reviewing related literature, e.g., refs. [2,18]. The needs analysis showed which media pre-service physics teachers already know of and use which digital media they are interested in and which digital media they still need to learn about in the seminar. Related literature suggested which digital media are important for physics teaching because of the subject specificity. Examples are simulations, digital data acquisition, and video analysis [18]. Moreover, the literature review helped to identify essential content about these digital media, such as important empirical findings on their use in physics teaching [2].

This developed teaching concept provides the basis for designing the seminars to be implemented at the participating universities. Considering the different curricular conditions, the seminar concepts share common core elements which are defined in the overarching teaching concept, especially for the theoretical sessions, but differ in the practical sessions. Starting from this initial concept, we carried out several steps of evaluation and re-design (roughly following one of the basic ideas of a design-based research approach [33]) to develop the teaching concept iteratively by drawing on the results of our empirical evaluation (cf. Section 3.3). The initial teaching concept and its implementation at the different locations is described in Section 3.1.

2.2. Development and Validation of the Knowledge Test Measuring Digital-Media PCK

2.2.1. Test Development

When developing the test instrument for measuring digital-media PCK (aim 2), the first step was to create a model of digital-media PCK in physics (cf. Section 1.2). We developed the model by reviewing subject-specific literature, e.g., refs. [2,18] and adapting general literature on the use of digital media in teaching, e.g., refs. [34,35]. In doing so, we primarily focused on aspects and contents that are specific to physics (cf. Section 1.2) and that add a content-related functional value of digital media compared to conventional tools (e.g., according to the levels of Modification and Redefinition in the SAMR model [24]). The final model comprises four different categories: (i) subject-related fundamentals, (ii) digital data acquisition, (iii) simulations, and (iv) explanation videos (Figure 1).

The category subject-related fundamentals represents knowledge of fundamental concepts regarding the use of digital media in physics teaching situations as well as overarching knowledge about the use of digital media for a specific purpose in physics teaching. Although this category does not represent physics-specific media, it was considered in the model because fundamental concepts on the use of digital media are also relevant for subject-specific teaching situations, e.g., considering design principles of multimedia applications to help students to develop conceptual understanding of a physics topic. The category digital data acquisition comprises knowledge about digital data acquisition (DDA) systems and their use for data acquisition, processing, and analysis in physics teaching. The category simulations includes knowledge about simulations and their use for gaining knowledge and modelling in physics teaching. These categories both represent a rather subject-specific competency area of the DiKoLAN framework (cf. Section 1.2) and were therefore considered in the model. Furthermore, the category explanation videos contains knowledge about those criteria that characterise effective explanation videos and their use in physics teaching. This category is included in the model because explanation videos are especially useful to explain complex content which is very common in physics teaching. For this reason, there is subject-specific literature on the use of explanation videos in physics teaching, e.g., ref. [23], which can be consulted for item development.

In a second step, the above-presented model was used as a starting point to develop test items in a targeted manner. Thus, at a structural level, our model for item development distinguishes between the four categories of the facet digital media outlined above on the one hand (cf. Figure 2). On the other hand, inspired by the model for item development that was used in the project ProfiLe-P [12,14], a distinction was made between three cognitive activities to create items with different requirements: (i) reproduce, (ii) apply, and (iii) analyse. Whenever a test item needed to be embedded in physical content, mechanics was the focus [36]. To ensure an objective and efficient evaluation of the test, all the items were designed as multiple-choice items (multiple select).

Based on the model in Figure 2, we created a knowledge test for digital-media PCK in physics that comprised 17 items. The test instrument was implemented as an online test and piloted during the summer semester of 2021 with 116 pre-service teachers from six universities in Germany and Austria (male: 58, female: 52, missing data from six pre-service teachers). The participants were on average 24 years old (M = 23.7; SD = 3.1, ranging from 19 to 35) and in their seventh semester (M = 6.9; SD = 3.1, ranging from 1 to 15). The sample consisted of 92 (79%) pre-service physics teachers and 24 (21%) pre-service teachers of other subjects (chemistry and technology). Based on the results, three items were deleted, and one item was newly developed to optimise the instrument. The resulting 15 multiple-choice items each consist of four to seven options that must be answered with yes or no. Figure 3 shows a sample item that is part of the category digital data acquisition and the cognitive activity reproduce.

To condense the content of the multiple-choice items and to reduce the influence of scoring by guessing, a threshold coding rule for scoring based on Kprime [37] was implemented. This coding rule allows achieving 0, 1, or 2 points for each item depending on how many options were answered correctly in that item. For example, an item including six options such as the sample item in Figure 3 would be scored as follows: 0 points if three or fewer options are answered correctly, 1 point if four options are answered correctly, and 2 points if five or all six options are answered correctly.

The pilot study sample described above, or part of it, was used in some of our validation studies. Participants’ test scores were calculated using 13 items, i.e., without the three excluded items, the newly developed item (for which data were not yet available), and another item that had to be excluded for data analysis due to an error in the online test.

2.2.2. Test Validation

To gain evidence for our first research question regarding the validity of the developed knowledge test (cf. Section 1.4), several qualitative and quantitative studies were conducted to investigate different aspects of construct validity [32].

In particular, to delimit the focused digital-media PCK from other knowledge aspects such as general pedagogical knowledge (cf. Section 1.1), two other test instruments were used within our pilot study: an adapted short scale of an established pedagogical knowledge test [38] and an online-based PCK test in physics from the project Profile-P-Transfer [39] consisting of multiple-select items. The latter instrument comprises four subscales of PCK: (i) instructional strategies, (ii) students’ misconceptions, (iii) experiments and teaching of an adequate understanding of science, and (iv) PCK-related theoretical concepts. Thus, this PCK test does not focus on digital media. Because our developed digital-media PCK test was designed to measure a specific facet of the more general construct PCK, we expect the correlation with a PCK test to be higher than with a pedagogical knowledge test.

To furthermore examine discriminant evidence concerning the external aspect of construct validity [32], we also had pre-service teachers of other subjects (chemistry and technology) answer the developed test instrument in the pilot study and compared their test scores with those of pre-service physics teachers. We expected the pre-service physics teachers to outperform those of other subjects since the instrument was intended to measure subject-specific digital-media PCK.

Regarding the content aspect of construct validity [32], the curricular fit of the test items (curricular validity) was examined. We therefore carried out an expert survey with ten physics teacher educators in Germany and Austria by letting them assess to what extent the knowledge required to answer the items correctly can be learned in the respective university physics teacher education programme in Germany and Austria. For each item, we had them answer the question “Can the necessary knowledge be acquired in your university’s physics teacher training programme to answer this item correctly?” on a 6-point rating scale from “Yes, absolutely” to “No, definitely not”. The mean of these ten answers yielded the curricular fit for each item with possible values from 1.0 (very good fit) to 6.0 (no fit at all). In addition, the experts could voluntarily provide an explanation or comment on their evaluation of the individual items.

Finally, the substantive aspect of construct validity [32] was investigated through a think-aloud study with four pre-service physics teachers to identify problems in understanding the test and tasks, as well as the underlying areas of knowledge used to answer the test. The participating pre-service teachers were therefore asked to answer the test while expressing their thoughts aloud. Since the analysis did not lead to major changes in the instrument or to unexpected findings that are necessarily important for this article, the results of the think-aloud study are not presented and discussed in detail. We also investigated the structural aspect of construct validity of our test [32] by comparing a four-dimensional Rasch model based on the four categories of our digital-media PCK modelling (cf. Figure 1) with a one-dimensional model. These results are also not presented in detail, as the multidimensional analysis based on these individual categories is not relevant for this article.

The results of the first three validation studies mentioned above are described in more detail in Section 3.2.

2.2.3. Further Test Quality Criteria

After the pilot study and the validation studies, the optimised test instrument was used on other occasions with pre-service physics teachers in addition to the evaluation study. These previous data were analysed using a one-dimensional Rasch model to investigate further test quality criteria of the optimised test instrument. The corresponding data sample so far comprises N = 183 cases from 136 pre-service teachers (47 additional cases because of repeated measures). The participants were on average 24 years old (M = 24.1; SD = 4.4, ranging from 19 to 50) and in their seventh semester (M = 6.7; SD = 2.6, ranging from 1 to 15). The results of the Rasch analysis are presented in Section 3.3.

2.3. Study Design of the Empirical Evaluation

To evaluate the developed teaching concept (aim 3) and to answer our second research question, we investigated the development of our pre-service physics teachers’ digital-media PCK in the three participating university seminars in a pre-post format using the newly developed knowledge test.

Thus, the study took place in the context of German and Austrian university teacher education programmes. The corresponding seminars are attended by rather advanced students. Not all seminar participants took part in the empirical evaluation, as participation in the study was not compulsory. The sample for which data are reported below consisted of N = 23 pre-service physics teachers from the three universities (RWTH Aachen University, University of Graz, and University of Tübingen) who fully participated in the seminars as well as the study survey. Eleven students were female and twelve students were male. On average, the students were in their eighth semester (M = 7.9; SD = 2.8, ranging from 5 to 15) and 26 years old (M = 26.0; SD = 5.2, ranging from 21 to 42). The seminar in Tübingen took place in the summer semester of 2021 (7 students) and the seminars in Aachen and Graz in the winter semester of 2021/22 (5 + 11 students). Due to the COVID-19 pandemic, some of the seminar sessions had to be held online. The number of online sessions varied depending on the location and thus added unaccountable variance to the study setup.

The students answered the developed knowledge test for digital-media PCK at the beginning of the seminar (pre-test) and the end of the seminar (post-test). For the data analysis, the participants’ test scores for the pre-test and the post-test were calculated and a dependent samples t-test was conducted for comparison. The participants’ test scores were calculated from 12 items, as one of the 13 items used for analysis in the validation studies was subsequently excluded due to its curricular fit (cf. Section 3.2.3). We used classical test scores instead of person parameters for this analysis as the sample size was too small to analyse the data adequately with a Rasch model.

We will continue the evaluation study in the following semesters and therefore increase the sample size to answer our second research question. In this paper, preliminary results from our surveys so far, i.e., the sample described in this section, are presented.

3. Results

Before we present the results regarding our two research questions in this chapter, the developed teaching concept (aim 1) is described, as it presents the context of our evaluation study.

3.1. Teaching Concept

The developed teaching concept forms the basis for all three participating university seminars of our project DiKoLeP. To implement the teaching concept as a specific seminar at each participating university, it is adapted and implemented taking the local requirements (e.g., location-specific curricula) into account. However, the concept consists of common core elements to guarantee comparability between the three seminars resp. universities. After briefly describing the overarching teaching concept and its structure, we characterise these common core elements of the teaching concept, i.e., the theoretical part of the concept. Then the rather individual practical parts of the seminars are outlined. Figure 4 provides an overview of the structure of the seminars by showing their similarities and differences for the participating universities.

The teaching concept follows a theory–practice structure starting with a theoretical part to introduce theoretical concepts regarding the use of digital media in general as well as in physics teaching. The following practical part of the concept allows the pre-service teachers to practice the use of selected digital media in authentic teaching situations (cf. Figure 4).

The theoretical part of the teaching concept contains specific core elements that are taught in each of the participating university seminars. These core elements were pre-structured based on a literature review (cf. Section 2.1) and finally chosen based on a pre-service physics teachers’ needs analysis (for additional information see [40]). The theoretical part covers many different digital media and aims to provide a broad overview of how they can be used effectively in the physics classroom. Nevertheless, the core elements also involve fundamental and general concepts regarding the use of digital media in teaching as an introduction to the seminar. These include the Cognitive Theory of Multimedia Learning [34] and the Cognitive Load Theory [35], empirical findings regarding the use of digital media in science teaching, e.g., ref. [1] and the SAMR model to reflect the integration of technologies in teaching [24]. These basic concepts are required for reflecting and planning the use of digital media in specific physics teaching situations. We furthermore identified the following subject-specific or typical media as further core elements: digital data acquisition, e.g., ref. [2], mobile devices as experimental kits [20], (tablet-PC supported) video analysis, e.g., ref. [41], explanation videos [23], interactive screen experiments [42], computer simulations [21,22], and augmented reality [43].

In this theoretical part, the teaching concept is to discuss possible uses and design principles of these media as well as empirical findings on their use in physics teaching. To align theoretical contents with practice within the sessions, smaller tasks and activities were included that enable initial experiences with each medium. Additionally, discussions were integrated into the theoretical sessions to allow the pre-service physics teachers to reflect on the role of digital media in teaching and its possibilities and limitations to encourage a reflective approach (cf. Section 2.1).

The practical part of the teaching concept aims to allow the pre-service physics teachers to explore a particular medium intensively by planning and performing an authentic teaching situation with it. They work in groups or pairs and get feedback for their developed learning material and their teaching performance from their seminar instructor as well as from their peers. This practical experience differs between the seminar in Aachen and the seminars in BBB Graz and Tübingen. The pre-service physics teachers in Aachen develop an experimental learning circle in groups that consists of several smaller experiments and includes the use of digital media. The students conduct their learning circle with a real school class at the end of the seminar. They receive feedback from different perspectives, both on the learning materials they have developed (during the seminar from their fellow students and the seminar instructor) and on their teaching performance at school (from the pupils and their physics teacher). The pre-service physics teachers in Graz and Tübingen undergo two cycles of practical experience. For each, they plan a lesson for a specific teaching situation (physical content, grade level, etc.) using a digital medium they have studied in depth. They perform a teaching unit of this lesson with the seminar group and reflect and discuss the performance afterwards with their fellow students and the seminar instructor (cf. Figure 4).

3.2. Test Validation

To answer our first research question regarding the validity of the developed knowledge test, we carried out several smaller studies to gain evidence for construct validity. As mentioned in Section 2.2.2, the results of three of these validation studies are presented in the following subsections.

3.2.1. Delimitation from General Pedagogical Knowledge by Correlation Analysis

To ensure that our developed test instrument measures a specific aspect of PCK—i.e., subject-specific knowledge—and not general pedagogical knowledge regarding the use of digital media, we compared pre-service physics teachers’ test scores on the digital-media PCK test with their scores on the test of pedagogical knowledge (PK) and the more general PCK test (cf. Section 2.2.2).

Table 1. Manifest correlations of the knowledge tests in the validation study (Pearson’s r).

	Digital-Media PCK	General PCK (Excl. Digital Media)	PK
Digital-media PCK (α = 0.69)	−
General PCK (excl. digital media) (α = 0.67)	0.46 ** (N = 43)	−
PK (α = 0.74)	0.32 ** (N = 105)	0.34 * (N = 41)	−

This study was carried out with part of the pilot study sample. Cronbach’s Alpha of each test is given for the respective sample surveyed with it. The sample sizes vary because the PK and PCK tests were only administered on part of the sample due to the longer test time. * p < 0.05, ** p < 0.01.

shows the correlations of the pre-service teachers’ test scores in the three respective tests. All correlations are significant. However, as expected (cf. Section 2.2.2), the correlation of the digital-media PCK score with the general PCK score is higher than with the PK score.

3.2.2. Discriminative Validity Aspect by Surveying Pre-Service Teachers of Other Subjects

During the pilot study, we let pre-service teachers of other subjects answer the test instrument to see how they would perform compared to the pre-service physics teachers (cf. Section 2.2.2). This study was carried out with all 116 participants of the pilot study (cf. Section 2.2.1) consisting of 92 pre-service physics teachers) and 24 pre-service teachers of other subjects (chemistry and technology).

Table 2. Descriptive statistics of the surveyed student groups regarding their digital-media PCK.

Pre-Service Teachers	M	SD	Minimum	Maximum
Physics (N = 92)	14.02	4.31	4	22
Other subjects (N = 24)	10.75	3.26	6	17

The participants could achieve a maximum of 26 points (13 items, cf. Section 2.2.1).

shows the descriptive statistics of the participants’ test scores.

Consistent with our expectation that pre-service physics teachers will outperform those of other subjects (cf. Section 2.2.2), an independent samples t-test showed that the mean difference of the groups regarding their physics-specific digital-media PCK was significant (t(114) = 3.467, p = 0.001) with a medium-to-large effect size (d = 0.80).

3.2.3. Curricular Validity Aspect by an Expert Survey

The expert survey to investigate the items’ curricular fit was conducted with ten physics teacher educators in Germany and Austria. Figure 5 shows the frequency distribution of the curricular fit of the items. Eleven out of fifteen items had good or very good fit values (≤2.0). The fit values of the remaining four items were still acceptable (≤2.7).

As all items showed at least an acceptable curricular fit, we did not see a reason to exclude an item from the test at first. However, the results of the optional comments showed that some experts did criticise one item concerning its connection to the use of digital media in physics teaching. Since the item also had a comparably bad curricular fit value (2.44), we finally removed it from the test instrument. The experts’ comments were also the reason for minor changes in three other items (e.g., change of wording or removing individual options).

3.3. Further Test Quality Criteria

After revising the test instrument based on our results from the pilot study and the validation studies, we analysed the optimised test instrument regarding further test quality criteria using a one-dimensional Rasch model (cf. Section 2.2.3). In our previous data sample (cf. Section 2.2.3), a variance of 0.64 and an EAP-reliability of 0.74 were observed. The item outfit (0.86 < MNSQ < 1.21) and the model fit (SRMR = 0.08) showed acceptable values. The Wright map indicates that the difficulty of the test instrument was appropriate for the surveyed sample as the item parameters (item difficulties) were all within the range of the person parameters (person abilities) except for one (Figure 6).

3.4. Empirical Evaluation of the Teaching Concept

Regarding the evaluation of the developed teaching concept (aim 3), our second research question is to investigate the development of participating pre-service physics teachers’ digital-media PCK.

Table 3. Descriptive statistics of participants’ digital-media PCK in the pre-test and post-test.

Digital-Media PCK	M	SD	Minimum	Maximum
Pre-test	14.00	3.92	3	20
Post-test	15.57	4.46	6	22

The participants could achieve a maximum of 24 points (12 items, cf. Section 2.3).

presents the descriptive statistics of our participants’ digital-media PCK in the pre-test and post-test.

To answer the second research question, a dependent samples t-test was conducted to compare the pre-service physics teachers’ test scores in the pre-test and post-test. The analysis showed a significant improvement with a small-to-medium effect size (t(22) = −2.160, p = 0.042, d = 0.45).

4. Discussion

The first aim of the current work was to develop a university teaching concept to foster pre-service physics teachers’ digital-media PCK which can be implemented in physics teacher training at various universities. The development of the teaching concept and its implementation for the three participating universities was successfully realised. The teaching concept offers an overarching structure and certain core elements to align the implemented seminars for our study. However, the seminars have some individual parts to consider the curricular requirements of the respective university. In that way, the developed teaching concept is not limited to being implemented only in these three universities but allows for broader use. Up to now, the respective seminar was conducted once at each participating university. This first run-through generally worked well, as indicated, for example, by the feedback from the participating pre-service physics teachers. Nevertheless, the teaching concept is currently re-designed based on our initial experiences as well as results from our empirical evaluation.

The second aim was to develop a suitable knowledge test to proximately measure digital-media PCK rather than based on self-reports. The instrument was specially designed to be objective and efficient in both the testing and analysis phases (cf. Section 2.2.1). Therefore, the final knowledge test was administered as an online-based instrument that consists of 14 multiple-choice items with four to seven options per item (multiple select). To avoid scoring by guessing, a specific coding rule was applied. Since some items are quite long or complex (i.e., if they are embedded in a physics teaching situation), the test time is about 25 min.

Regarding the first research question, our validation studies indicated that the developed knowledge test is suitable to validly measure digital-media PCK of pre-service physics teachers. As expected, our results showed that the test instrument correlates more strongly with a subject-specific PCK test than with a general pedagogical knowledge test (cf. Section 3.2.1). Furthermore, our findings suggest that our test instrument measures a facet of PCK and thus subject-specific knowledge as pre-service physics teachers performed better in our test than pre-service chemistry or technology teachers (cf. Section 3.2.2). Moreover, we found that the kind of knowledge our test measures seems to be achievable in German-speaking university physics teacher education programmes, as most items have a good curricular fit (cf. Section 3.2.3). Finally, our findings showed that the developed test instrument showed acceptable fit values and reliability when analysed using a one-dimensional Rasch model. As most studies do not assess digital-media PCK or TPACK with a proximal measurement [28], the presented results of the developed test’s validity and quality criteria are difficult to compare to other studies. In a study analysing the effectiveness of subject-specific TPACK-modules with another proximal measurement, Lachner et al. assessed pre-service teachers’ subject-specific TPACK with knowledge tests for five different subjects other than physics [27]. Each of these tests consists of eight open-ended items that address a subject-specific teaching problem through short text-based vignettes. While the reliability of one test was rather low (Cronbach’s alpha of α = 0.54), the other four tests showed a good reliability (Cronbach’s alpha between α = 0.72 and α = 0.77), comparably to the presented reliability in this work. Other studies investigating PCK in physics with knowledge tests also report reliabilities in similar ranges, e.g., [7,8].

The third aim was to investigate the development of our pre-service physics teachers’ digital-media PCK across the teaching concept. The evaluation so far showed a significant increase in digital-media PCK for students that participated in the seminars following the developed teaching concept (cf. Section 3.4). This result is coherent with those of other studies that investigate the effectiveness of university courses or interventions with the aim to foster pre-service teachers’ digital competencies or TPACK, e.g., refs. [27,44]. For example, an evaluation of a pre-service science teacher training course designed to foster the digital competencies following the DiKoLAN framework (cf. Section 1.2) showed significant increases in participants’ self-efficacy expectations regarding digital competencies covered in the course. For those competencies that are relevant for the current work—competencies of the subject-specific competency areas Data Acquisition, Data Processing and Simulation and Modelling and the focus Teaching (i.e., TPACK)—self reported increases were found with mostly medium effect sizes [44]. In the study described above, the effectiveness of subject-specific TPACK-modules implemented in subject-pedagogy courses across five subjects was compared to regular subject-pedagogy courses without specific TPACK-modules [27]. The design of these TPACK-modules was also based on the SQD model (cf. Section 1.3). In addition to the investigation of technology-related self-efficacy, subject-specific TPACK was assessed with open-ended knowledge tests. Results of this quasi-experimental study showed a medium effect of the TPACK-modules [27]. Thus, the result of the current work is similar to other studies evaluating university learning opportunities that aim to support the acquisition of pre-service teachers’ digital competencies or TPACK and that are designed based on similar principles such as the DiKoLAN framework or the SQD model. However, comparability is limited because either the methods of measurement or analysis are different, or the evaluation does not refer to physics teacher education. Additionally, the presented results should be viewed with caution because of the small sample size. We will increase our sample size by continuing the evaluation of the teaching concept in the following semesters.

4.1. Limitations

The teaching concept was to be conducted as a face-to-face university seminar. Due to the pandemic, parts of the seminars had to be held online, for which they were not originally designed. This leads to two limitations regarding the implementation of the developed teaching concept. First, the number of online lessons varied in all three participating seminars, which limits comparability. Secondly, the teaching concept was not evaluated as it was originally designed. For example, the intended combination of theory and practice and authentic experiences with digital media in the seminar sessions (cf. Section 2.1) could not be implemented so well in an online format.

As mentioned above, the small study size is another limitation of our evaluation study. It is quite common to have comparatively few pre-service physics teachers in German or Austrian teacher education programmes. Our sample size was further decreased by many dropouts. These might be caused by low test motivation in the post-test due to the long test time as we used further instruments in our evaluation that are not relevant to this work and therefore not presented in this paper. The post-test was conducted at a time in the semester when pre-service teachers may have been busy with exams, which could also be the reason for dropouts. Varying motivation of the participants when completing the test may furthermore lead to distortions in the results. The small sample size further limits our analysis, as we cannot evaluate the data for the individual seminars so far. In addition, we cannot rule out the possibility that the increased knowledge we observed does not come from other learning opportunities the pre-service physics teachers had during this time.

Our digital-media PCK test itself has a comparatively long test time and high complexity, which could also negatively affect the participants’ test motivation. Moreover, the individual options of a multiple-choice item can be quite complex to make a yes or no decision. On the other hand, the format to answer with yes or no in each option might lead the participants to guess. We addressed this issue with our specific coding rule (cf. Section 2.2.1). However, this might still distort the results and the validity. Furthermore, the instrument’s test time is higher than for any self-report survey in this field. However, we wanted to focus on the advantages in terms of validity that a proximal knowledge test offers and therefore came to terms with the rather long testing time that this approach would entail.

4.2. Further Research

Continuing our project in 2022 and 2023, we will increase the study sample size for the empirical evaluation of our teaching concept. In doing so, we aim to evaluate the data for the seminars individually for each university. We will also investigate students’ knowledge development regarding the different categories of digital-media PCK as we have indications from our structural validation study (cf. Section 2.2.2) that these categories can be seen as different dimensions.

Additionally, the summative evaluation will be complemented with further qualitative research to obtain a more detailed picture of the pre-service physics teachers’ PCK development during the seminars. Through retrospective interviews after the seminars, we will investigate to what extent the changes in the participants’ test performance are related to the elements of our teaching concept or possible other learning opportunities (cf. Section 4.1). Furthermore, we aim to identify which aspects of the common core elements or location-specific parts of the teaching concept are particularly conducive to learning. This way, we intend to develop hypotheses for effective learning opportunities regarding the use of digital media in physics teaching that can then be tested with a larger sample than just our universities.

5. Conclusions

This article presented the ongoing work of the joint project DiKoLeP which aims to develop and evaluate a university teaching concept to foster pre-service physics teachers’ digital media PCK. The evaluation was conducted in a pre-post-design using a newly developed test instrument to measure this specific kind of knowledge.

The teaching concept described in this article is currently implemented at three different universities, considering the respective curricular requirements. The concept is therefore suitable for adaptation and implementation at other universities and physics teacher training programmes. Since the empirical evaluation shows an increase in digital-media PCK among pre-service physics teachers that participate in the university seminars following this teaching concept, it can be regarded as a best-practice example for university seminars to support the acquisition of digital-media PCK in physics teacher education. However, the teaching concept is not yet finalised and will be further improved in the future based on additional research.

By using a knowledge test to proximally measure pre-service physics teachers’ digital media PCK, we attempt to overcome the problems of self-report based measures in this field. As the presented validation studies indicate, the developed test instrument is suitable to measure pre-service physics teachers’ digital-media PCK. In particular, the knowledge test can be used to evaluate university courses in physics teacher education that focus on the use of digital media that are specific or typical to the subject physics. Further validation can provide insights whether the chosen categories of the knowledge test can be seen as empirically verified subscales. In this way, individual categories of the test instrument could be used for empirical studies with an interest in researching specific digital media aspects when only limited test time is available.