Designing Performance-Based Professional Development: Stakeholder Views on Essential Competencies and Approaches

Abstract

Research has consistently shown that the instructional skills, or competencies, that teachers bring to the table influence the opportunities students have to learn. Teachers build those competencies over time, often through professional development (PD); however, they often cite PD as insufficient, in part because it can focus on the wrong competencies. In a series of focus groups, we systematically examined which teaching competencies teachers, teacher leaders, school administrators, and professional learning leaders identified as most critical for future PD efforts supporting K-8 mathematics and science teachers. In addition, we explored their stated rationales and their vision for how digital performance-based PD might support development of those competencies. Participants identified four competencies as most critical: (1) collaborative problem-based learning; (2) differentiation; (3) promoting engagement; and (4) eliciting student ideas. Rationales included ways in which they are essential for students, difficult to implement, and have insufficient support at present. Participants preferred the idea of digital and interactive role-playing PD over more traditional approaches, also citing the need for some agency in controlling the PD parameters and for the availability of rich feedback. Implications include the need for future research to incorporate stakeholder voice to close the gap between what is needed and what is provided and for follow-up studies to target a larger and more representative sample.

1. Introduction

1.1. Overview

Research has consistently shown that the instructional skills that teachers are able to engage in and use in their classrooms matter in the kinds of opportunities students have and the kinds of learning that take place (Fauth et al., 2019; Rivkin et al., 2005; Rockoff, 2004). It is also widely accepted that teachers acquire instructional skills, also called teaching competencies, not just in advance of entering the classroom but over time, as their expertise deepens through on-the-job experience (Darling-Hammond et al., 2017). It makes sense, then, that ongoing professional development (PD) opportunities are necessary. These opportunities are nearly always provided through schools or districts during designated times (T. Grossman & Hirsch, 2009). Unfortunately, they are often critiqued as not addressing teachers’ most pressing needs or not easily transferring to classroom implementation (Darling-Hammond et al., 2017). Moreover, the direct and subsequent effects of the COVID-19 pandemic have changed the challenges encountered by teachers, shifting their PD needs to include more support for technology use, ways to support student well-being, and professional resilience (Bozkurt et al., 2022; Hargreaves, 2021).

It is thus critical that designers of teacher PD understand which teaching competencies are most critical for effective teaching and why, as well as how these learning opportunities should ideally be designed. Otherwise, we risk not meeting the needs of teachers or, worse, directing their limited time to focusing on competencies that are not the ones they most need help with. To that end, our research team conducted a series of focus groups to examine which teaching competencies teachers, teacher leaders, school administrators, and professional learning leaders identify as the most critical ones for the focus of future PD efforts. We also aimed to learn why they view these teaching competencies as PD priorities and to understand what PD features they view as most important and useful for PD activities to address their priority teaching competencies. Specifically, we explored how priority competencies could be developed via digital, performance-based tools, such as teaching simulations, which ask the teacher to perform some aspect of the work of teaching. We refer to such approaches as performance-based approaches because they demand that the participating teacher do something, performing some piece of the work of teaching outside of the classroom. Recent technology advancements, particularly the use of digital learning platforms, simulations, and AI in educational contexts, coupled with the empirically demonstrated potential of practice-based teacher education, make the features of performance-based approaches particularly critical to understand. Our research focus was on PD for K-8 mathematics and science teaching, guided by two research questions:

• RQ1: What are the key teaching competencies that various stakeholders identify as high priority for K-8 math and science teachers to practice in PD, and why are these competencies seen as the highest priority?
• RQ2: What features/components of performance-based PD were identified by various stakeholders as important design characteristics, and why?

1.2. Background and Literature Review on Teaching Competencies

The term teaching competencies is defined by Sykes and Wilson (2015) as “a core set of practices in which teachers engage” (p. 7) for effective teaching. Others (e.g., Fauth et al., 2019; Jentsch & König, 2022; Reuter et al., 2022) frame teaching competencies as the set of knowledge, skills, beliefs, dispositions, and personal characteristics that relate to effective teaching and support professional responsibilities. These include instructional practices that teachers use during direct interaction with students in classrooms, such as eliciting and probing student thinking, facilitating discussions, or explaining content. They also include instructional practices that teachers engage in outside of class time, such as adapting curriculum materials and lessons to address their students’ needs or reflecting on students’ completed assignments, as well as non-instructional practices such as collaboration with colleagues and communication with families. Scholars have cataloged the most important teaching competencies, referring to them as core teaching practices (Charalambous & Delaney, 2019; Kloser, 2014) or high-leverage teaching practices (Calabrese Barton et al., 2020; Capobianco et al., 2018; TeachingWorks, 2024; Wilburne et al., 2018). Most of the prevalent frameworks cover general teaching competencies which could be deemed critical in all grades and content areas, although there are some content-specific catalogs, such as the framework for ambitious science teaching practices (Luehmann et al., 2023; Windschitl et al., 2020) or the set of high-leverage teaching practices in special education (Klimaitis & Mullen, 2021). The prevalence of such catalogs suggest that scholars are prioritizing some competencies over others.

Two rationales are most often cited for the prioritization of certain teaching competencies. The first is that these are the ones that teachers tend to use most often and, as such, are most likely to be impactful in either a positive or negative direction (P. Grossman et al., 2009). Another argument is that certain competencies are harder for teachers to learn to do well and/or are less likely to be picked up on-the-job, making them more critical to address in training (Ball & Forzani, 2011).

Prior to beginning the study, we conducted a review of the literature on critical teaching competencies, delimited to K-8 science and mathematics, to the past ten years, and to the study of in-service teachers. We selected 31 papers for review based upon their relevance to our research questions. These articles were analyzed to identify and define the competencies that the authors identified as their primary focus or described as integral to STEM learning. We found that these articles discussed competencies at varying levels of grain size, and that there was sometimes overlap where one specific competency was seen as part of enacting another broader competency. In total, we identified nine competencies that were identified in the selected literature as critical for supporting K-8 science and mathematics learning (

Table 1. Competencies identified in the literature.

Name of the Competency	Key Aspects of Engaging in This Competency	Supportive Literature/Citation
Supporting student-generated evidence-based explanations	Develop students’ ability to generate explanations for processes, events, or phenomena that integrates disciplinary reasoning and supporting evidence.	(Bae et al., 2022; Capobianco et al., 2018; Osborne et al., 2019; Lotter et al., 2018; Luehmann et al., 2023; Radloff & Capobianco, 2020)
Engaging students in collaborative problem-based learning	Facilitate student-driven collaborative learning where students assume responsibility and work together to solve authentic, complex problems.	(Capobianco et al., 2018; Luehmann et al., 2023; Radloff & Capobianco, 2021; Suh & Seshaiyer, 2019; Waddell, 2014)
Noticing student thinking	Identify attributes of student thinking about mathematical or scientific concepts, social-emotional signals, or aspects of students’ personal or cultural lived experiences and use them to inform instructional decisions.	(Calabrese Barton et al., 2020; Chan et al., 2021; Kang, 2022; Rosebery et al., 2016; van Es et al., 2017)
Facilitating student discussions	Engage students in discussions about content in which students share ideas, appreciate perspectives, listen actively, provide one another feedback, and co-construct knowledge.	(Braseth, 2022; Kloser, 2014; Kohen & Borko, 2022; Trevisan et al., 2020; Waddell, 2014)
Eliciting student ideas	Use questioning and prompts to encourage students to share and discuss their content-specific ideas so that the teacher can build on them.	(Braseth, 2022; Capobianco et al., 2018; Dieker et al., 2019; Kloser, 2014; Luehmann et al., 2023; Radloff & Capobianco, 2020; Roth et al., 2019; Walsh et al., 2023)
Connecting math and science to students’ cultures and communities	Draw connections between science and mathematics content and students’ cultural identities, lived experiences, and local communities so that students understand how science and/or mathematics can help them to improve and care for their communities.	(Bae et al., 2022; Calabrese Barton et al., 2020; Chiu et al., 2021; Luehmann et al., 2023; Sheth, 2019; Suh & Seshaiyer, 2019; Waddell, 2014)
Supporting students’ mathematics and science literacies	Help students use age-appropriate math and science communication forms such as discipline-specific vocabulary, and understand how mathematicians and scientists communicate about ideas.	(Fauskanger & Bjuland, 2019; Fazio & Gallagher, 2019; Kang, 2022; Thomas & Drew, 2022)
Using teacher questioning	Formulate different types of questions to elicit students’ interpretation, analysis, critique, evaluation, and explanations.	(Dieker et al., 2019; Lotter et al., 2018; Roth et al., 2019; Thomas & Drew, 2022; Waddell, 2014; Wilburne et al., 2018)
Engaging students in disciplinary argumentation	Engage students in argumentation with one another supporting ideas with evidence or justification and considering and critiquing the ideas of others.	(Lotter et al., 2018; Osborne et al., 2019; Walsh et al., 2023)

).

While other competencies were mentioned in the literature, we chose to focus on these nine because each appeared in four or more articles; the articles were specific to improving K-8 science and mathematics learning, and each had a cohesive definition we were able to articulate using the literature. Two other competencies, learning goals and scaffolding instruction, were omitted as they were described by the literature in general terms that were not discipline-specific. A third competency, linking and connecting scientific and mathematical ideas, was omitted because it only appeared in three articles and a clear and cohesive definition for this competency was not prevalent across those three articles. In this study, we aimed to explore the connection between the final nine critical competencies and their use in digital environments, especially in the context of teacher PD.

1.3. Opportunities for Teachers to Develop Teaching Competency

The development of teacher competencies is conceived as a sustained, long-term process (Feiman-Nemser, 2001; Sancar et al., 2021). Teacher preparation programs often provide a foundational understanding of content knowledge and pedagogy, although the length and depth of that preparation varies across programs and graduates may feel unprepared to address the challenges and complexities of the first years of teaching. Opportunities for in-service teacher learning can be provided by PD programs, mentoring, and induction initiatives. When well-designed, they can effectively help teachers, especially new teachers, grow their critical teaching competencies (Lee & Santagata, 2020).

Traditional definitions of PD (e.g., Desimone, 2009) highlight processes and activities aimed to increase teaching learning and professionalism that can serve to change instruction to benefit student learning. Other definitions specify PD as part of a lifelong learning process in a professional environment (Sancar et al., 2021). We use the definition of Fraser et al. (2007) that highlights PD formal learning activities initiated by agents different from the teacher. PD can vary in time and depth, ranging from short-term courses to comprehensive, long-term programs grounded in classroom activity (Jentsch & König, 2022). Research has identified key features of effective PD, including having a specific content focus, incorporating problem solving and collaboration, the use of face-to-face training at the beginning of the program, follow-up support or feedback, and, notably, including opportunities to enact the learned practices in the classroom (Darling-Hammond et al., 2017; Popova et al., 2022). While PD can provide valuable knowledge and competency development, its effectiveness can be limited by factors such as relevance to classroom practice, instructional quality, and opportunities for immediate application. For example, Guskey (2000) emphasized the importance of aligning PD with teachers’ specific needs and providing opportunities for reflection and practice, highlighting the importance of relevance. Research on PD (e.g., Darling-Hammond et al., 2017; Popova et al., 2022) also notes that there is often a discrepancy between known features of effective PD and PD as enacted. For instance, PD may focus on knowledge acquisition and credit completion rather than development of teaching competencies and their classroom enactment. These are not simply research observations; teachers often note the limitations of PD programs, such as the discrepancy between program emphases and personal learning needs and the gaps between learning in a PD setting (workshops or meetings) and the opportunities to enact learning in classrooms and schools (Harwin & Baker, 2024) As a result, educational researchers, district and PD leaders, and policy makers have advocated for more PD that helps teachers practice actual teaching competencies by developing not just their knowledge but also their ability to apply that knowledge to teaching (Davis & Haverly, 2022; Forzani, 2014; Goodson et al., 2019; P. Grossman, 2021; Zuo et al., 2023).

Another form of support for in-service teacher learning, often provided separately from traditional PD structures, is observation and feedback. Feedback might be provided after a formal evaluation, be formative in nature, or be part of structured or informal observations between peers. Timely and actionable feedback can offer valuable insights into teachers’ areas for growth and development. This type of feedback, which describes performance explicitly and illustrates ways to improve towards quality instruction, is highly valued by teachers (Keiler, 2018; McLeod et al., 2019). Nevertheless, Finster and Milanowski (2018) highlight that teachers’ perceptions of the feedback are linked to perceptions of change in their own teaching practices. A well-designed PD program focusing on improvement of instructional practice, therefore, should incorporate feedback on the teaching performance, as well as teacher reflections and analysis of practice.

Research has focused more on describing teaching competencies to support instruction in the digital era and in technology-enhanced learning environments (e.g., Castañeda et al., 2022; Skantz-Åberg et al., 2022) but less on what teachers need to participate in digital and performance-based PD. Some studies (e.g., Xerri & Campbell, 2016), however, mention the importance of teachers’ identification of their professional learning needs in digital environments. Skantz-Åberg and colleagues (2022) note that teacher PD for digital competencies should shift from an individual focus to more sustained institutional approaches, which can be accomplished via performance-based and digital PD. In the next section, we will describe how digital approaches can enhance teacher learning in performance-based PD.

1.4. Performance-Based Approaches to Teacher PD

One strategy to address the gap between knowledge-focused PD and classroom practice is the use of practice-based approaches that engage teachers in acting out the work of teaching. This sometimes takes the form of digital approaches; simulation tools, such as virtual classrooms and scenarios, can offer a safe and controlled environment for teachers to experiment with different teaching strategies (Mikeska et al., 2023; Mikeska et al., 2022; Mikeska & Lottero-Perdue, 2022; Mikeska et al., 2021; Phelps et al., 2023). We refer to such approaches as performance-based approaches because they demand that the participating teacher do something, performing some piece of the work of teaching outside of the classroom. The underlying logic of these approaches is that they provide opportunities to practice teaching practices or skills outside of the classroom, providing a safe space for experimentation and mistake-making without consequence to students. Because the tasks can be focused narrowly on particular teaching skills and customized to context (Fairman et al., 2022), they also provide a mechanism for collecting feedback, whether from other teachers, instructional coaches, or teacher educators (Howell & Mikeska, 2021).

One additional affordance of digital approaches is convenience. Currently, most PD for in-service teachers tends to occur in-person as part of district or local, informal programs, but this makes it difficult for schools and districts to meet the growing and varied learning needs of their teachers (Powell & Bodur, 2019). Innovations like digital teaching simulations allow teachers to practice an aspect of teaching with student avatars who respond in real time or via short, scenario-based performance tasks where teachers can try out a specific practice in response to a written classroom scenario via an online platform (Kaufman & Ireland, 2016; Mikeska et al., 2021; Phelps et al., 2023). Because these occur in digital formats, they are customizable (not every teacher receives the same PD) and flexible (teachers can sign on at a convenient time) (Fairman et al., 2022). Empirical evidence suggests the potential of these digitally based simulations for pre-service teacher learning (Ade-Ojo et al., 2022; Bondurant & Amidon, 2021; Dieker et al., 2019; Mikeska et al., 2022), as well as for in-service teacher learning (Bondie et al., 2021; Dieker et al., 2019; Mikeska et al., 2023; Mikeska & Lottero-Perdue, 2022). To date, however, their use with pre-service teachers (PSTs) has been more widespread; for example, as of this writing, one of the most widely used platform providers, Mursion, estimates that 95% of their education clients are institutions of higher education, with only 5% being PD providers or school districts (C. Guimaraes, personal communication, 26 August 2024). While there are other platforms, and some institutions of higher education may also serve in-service teachers (ISTs), this strongly suggests these approaches are more commonly used with PSTs than ISTs.

We reviewed the literature from 2013 to 2023 focused on digital simulation platforms for use by ISTs, leading us to identify Mursion, TeachLive, SimSchool, and Teacher Moments as the most commonly referenced digital teaching simulation platforms. Of these, Mursion was mentioned most often in published studies of digital simulation approximations of practice (Aguilar & Kang, 2023; Dove et al., 2023; Mikeska & Lottero-Perdue, 2022; Mikeska et al., 2023; Wernick et al., 2021). Mursion simulations as described in these publications involved interactions with between one and five sim-students played by a live actor, made to look, sound, and act like students using AI and computer-aided supports. In some cases, a “sim-mentor” introduced the simulated lesson and reappeared and led the participants through a reflection after the lesson. Simulations ranged from 8 to 30 min. Research using the related TeachLive virtual environment describes it as giving users “a sense of immersion with both physical and social presence wherein the teachers interact with student-avatars in real time, holding authentic discussions on varied content areas” (p. 64) followed by feedback via after-action reviews (Dieker et al., 2019). In contrast, the SimSchool platform offers teachers simulated interactions with a single student asking for help. It provides information on the student’s prior academic performance and personality, along with a menu of pre-defined instructional strategies and verbal responses—some considered correct, and others considered incorrect. The platform also displays the simulation objectives and indicates progress made toward them (Dove et al., 2023). Unlike Mursion, TeachLive, and SimSchool, Teacher Moments is designed more for practice-based short drills that could be completed quickly and offer practice on discrete, specific teaching skills (Reich, 2022). Another teaching simulation platform, the Foundational Assessment of Competencies for Teaching (FACT), also offers opportunities for teachers to quickly practice a discrete skill through interactive performance tasks (Phelps et al., 2023). The performance tasks developed for FACT have the potential to be used with in-service, as well as pre-service, elementary school teachers of mathematics. The FACT tasks include a virtual whiteboard for use during audio recording of teachers teaching the lesson, along with details about the practice-teaching situation and the specific skills a user is to demonstrate when teaching the lesson (Phelps et al., 2023). Across these digital simulation approaches, there were several sources of variation, including what was given to the participant in advance, how they were expected to respond to the prompt (e.g., selecting options, recording audio or video or using a whiteboard, typing a response, or having a conversation), the prompt itself (e.g., samples of student work, audio or video prompts, seeing students talk through their work in the moment), the pacing and amount of time allotted for both preparation and performance, and the surrounding supports such as whether and how feedback was provided (Howell et al., 2019).

Our review also revealed evidence of the effective use of digital simulations for teacher practice of four of the nine competencies identified by our review of prior research: facilitating student discussions (Mursion and TeachLive), using teacher questioning (Mursion), eliciting student ideas (Mursion and SimSchool), and engaging students in disciplinary argumentation (Mursion) (Aguilar & Kang, 2023; Dieker et al., 2019; Dove et al., 2023; Mikeska & Lottero-Perdue, 2022; Mikeska et al., 2023). This suggests that the field lacks research evidence to suggest whether such approaches are appropriate and effective for the other five key competencies. Digital simulations have also been used to provide ISTs with opportunities to practice behavior management (e.g., Yang et al., 2021). The FACT tasks enable practicing of teaching tasks including “questioning to elicit student understanding” and “representing concepts using a mathematical model” (Phelps et al., 2023). Digital simulations have also been used to provide ISTs with opportunities to practice behavior management (e.g., Yang et al., 2021). A study focused on insights from teacher preparation experts (Donehower Paul et al., 2020) concludes that simulations should be especially useful for practice related to behavior management as well as for the use of open or closed questioning techniques, creating an environment of respect and rapport, providing positive feedback, using assessments in instruction, and helping students elaborate on new information. However, Lindberg and Jönsson (2023) pointed out potential drawbacks of using avatars and simulations in teacher education, such as technical difficulties and constraints that limit avatars’ capacity to move and interact in ways similar to a real classroom (for example, in hands-on activities). If these technical limitations are not accounted for, they may be interpreted as pedagogical shortcomings of the teachers.

2. Materials and Methods

2.1. Participant Sample

Participants in this study were individuals located in the United States who actively held one or more of the following roles at the time of the study: in-service elementary and/or middle school science and mathematics teachers; instructional coaches who support elementary and/or middle school mathematics and science teachers; school district leaders and school administrators that serve K-8 students and work directly with elementary and/or middle school mathematics and science teachers; teacher educators who support in-service science and/or mathematics teachers; and PD providers who support in-service elementary and/or middle school science or mathematics teachers. Participants were recruited through organizational mailing lists and professional contacts of the research project team. The team also recruited individuals who previously participated in their research studies and indicated interest in additional research opportunities.

There were 59 participants across eight focus groups, with participants grouped according to their role as shown in

Table 2. Focus group participants by group.

Group ID	Group Description	n
A1	Elementary school (K-5) science and math teachers	8
A2	Elementary school (K-5) science and math teachers	9
B1	District leaders and school administrators	8
B2	District leaders and school administrators	5
C1	Teacher educators, instructional coaches, and PD providers	9
C2	Teacher educators, instructional coaches, and PD providers	6
D1	Middle school (6–8) science and math teachers	7
D2	Middle school (6–8) science and math teachers	7

. Participants were selected for diversity according to the following characteristics, prioritized in the order listed: (1) math and/or science content area experience; (2) race and/or ethnicity; (3) gender; (4) prior experience with practice-based PD; (5) school type (i.e., public, private, or charter school); (6) school location (e.g., urban, rural, suburban; geographic location); and (7) years of experience teaching or supporting teachers in K-8 science and/or mathematics education. Participant demographics are included in Appendix A.

2.2. Focus Group Protocol

We opted to conduct focus groups for this study because of our focus on emergent perspectives. Although focus groups do not necessarily collect representative data or allow for comparisons between individuals, their advantage is in the opportunity to capture participants’ responses to the insights of others, which can elicit richer thinking than other approaches, such as individual interviews. The focus groups were scheduled as eight two-hour sessions on Microsoft Teams, each co-led by two members of the research team. We conducted two sessions for each educator role (Table 2) to limit each to no more than ten participants and ensure all participants’ perspectives could be heard. In addition, holding two sessions per group was necessary to accommodate the inclusion of participants in different time zones. A focus group protocol was developed through an iterative process in which the team worked to draft questions, provide feedback, and revise. One notable change made after the first focus group session (A1) was to add a question to the first part of the protocol asking participants for their own definition of the competency under discussion even in cases where we had provided a definition (i.e., when it was one of our nine key competencies identified in the literature). Although the focus was not on capturing individual perspectives as much as eliciting rich discussion, we took steps to support a balanced representation of perspectives. The facilitators paused the discussions as appropriate to solicit input from participants who had not yet spoken and encouraged participants whose experiences and insights might differ from those already shared to voice their thoughts.

Part 1 of the final protocol focused on identifying competencies participants felt were important to focus on in PD for K-8 mathematics and science teachers and on identifying their rationales for prioritizing those competencies. Participants were first asked to brainstorm competencies using the mentimeter digital tool to collect responses (see Appendix B for images). Mentimeter collects data in the moment and displays real-time aggregated responses on screen; using it enabled us to collect input and then immediately ask participants to respond to it. After brainstorming, participants were then asked to consolidate around three competencies. We then shared the nine competencies we had identified based on the literature review described in Section 1.2 and described to participants what each of the competencies means. We then presented participants a rank ordering task in which they were asked to rank a combined list of twelve competencies: the three group-brainstormed and the nine pre-identified competencies. The resulting highest-ranked three competencies were then used as the competencies of focus for the remainder of the first part, in which follow up questions focused on defining each of the three competencies and providing rationales for their importance.

Part 2 of the protocol focused on designing practice-based tools to support PD. Participants were shown three videos of existing practice-based tools and were presented with a list of features to attend to as sources of variation among the tools, defined in

Table 3. Features of digital platforms presented/coded.

Features	Definitions
Time	Includes pacing, overall time allowed, number of sessions, and whether one can pause, rewind, or do-over.
Stimulus	This refers to what the participant is presented with. This includes any preparatory materials and examples, samples of student work, audio or video prompts, or anything else provided in advance of responding.
Preparation	Describes what is expected or allowed in terms of preparation before the participant is asked to respond.
Response Type	This refers to the type of response that is expected, which might include selecting options, recording audio or video, drawing on a whiteboard, typing a response, or having a conversation.
Feedback	This refers to when feedback is received, from whom, and the nature of the feedback. This might include self-reflection, written or spoken feedback, and might be more or less formal. It might also include indirect feedback such as providing a rubric in advance which communicates expectations.
Content/Grade Level a	This refers to the elements of PD that center on and correspond to the subject matter taught in classrooms and the specific grade levels being addressed.

^a Content/Grade level was not initially presented to the participants and was added during analysis as a frequently mentioned feature.

. They were then asked to create concept sketches, in which they collectively brainstormed how they might combine these features or others to create an idealized PD experience using a practice-based digital tool, focusing on a particular competency. For this section, they were asked to focus on concept sketches for the first two of the competencies identified initially and for a third competency, which the research team pre-selected for the purpose of having complete data across all focus groups on one competency—Connecting math and science to students’ cultures and communities. In two cases (focus groups B1 and D1), only two competencies were discussed during this section of the focus group due to time constraints.

We note that we encountered an issue with the mentimeter software during data collection that caused some discrepancy. Specifically, the displayed rankings on which the focus group leaders based their decisions did not always update in real time, causing some cases in which incorrect decisions were made about which competencies to move to discussion. As we discuss in the Section 3, the effects on part 1 were minimal, and there were no effects on part 2.

2.3. Analysis

2.3.1. RQ1: Identification of Priority Competencies and Rationales

Data from the ranked ordering was extracted from mentimeter and organized into tables. Our first analytic step was to consolidate competencies. Because each focus group session began with an open brainstorm of competencies, some represented the same idea but were described in slightly different words than appeared in other descriptions or in our literature-informed list. In such cases, two team members reviewed and decided on a consensus basis where the competencies were close enough to label as the same competency. For example, for analytic purposes, “Differentiation—meeting the needs of different populations” was combined with “differentiation of science and math learning”, “meeting the needs of specific populations (differentiation, universal design, English language learners)” and other listings that foregrounded differentiation. This consolidation allowed us to produce a more meaningful ranking list and also to group data by competency for analytic purposes. Transcripts of the focus group sessions were downloaded and cleaned in preparation for analysis by correcting minor transcription errors and placing them into Excel so that each utterance could be coded.

To capture the nature and variability of definitions given for the highly ranked competencies, we coded the data from part 1 of the focus group as follows: First, two raters open-coded the data at the utterance level for evidence of definitional attributes and for evidence of rationales. For example, under the competency “differentiation”, for the following utterance, both raters noted the definitional attributes of having students work in groups and via different modalities, and both raters noted as a rationale that different modalities support students differently: “In my classroom in particular, I mean, there’s me, myself and my aid. So, we’ll split the group up. I mean, I have a small group as it is, but we’ll split it up even more, so that it’s smaller, and we’ll work, maybe like 2 to one maybe 3 to one, you know. Just so it’s smaller, and we can really engage them. There’s breaks included with our learning, you know, in my classroom. There’s a lot of breaks in between where that probably isn’t something in. Say, a typical Gen. Ed. Classroom but the work is much different. Some kids learn better on the iPad, and some kids learn better with the worksheet in front of them. It just really depends what their learning style is and how we can engage them differently” (Participant 05, Elementary Teacher Group 2).

For definitional attributes, each rater independently reviewed the definitional attributes under a given competency and generated a provisional list of focus themes, giving category names to the types of evidence they had collected during open coding. For example, continuing with differentiation, one rater identified four focus themes for defining attributes of differentiation: assigning different content or tasks, providing different modalities for learning, giving more or less attention to different individuals or groups depending on their level of need, and supporting student agency or choice. The pair of raters then met to consolidate provisional themes into consensus themes, working to produce a set of non-overlapping descriptors that would succinctly capture the data, including variability across definitions offered by participants. As part of reconciliation, they also noted, for each resulting theme, its presence or absence in each of the focus group sessions from which data were drawn for that competency, allowing us to examine whether there was variation in definitions by stakeholder group. A third rater then reviewed and consolidated themes where possible and constructed a summary analytic memo of the results. A parallel process was conducted to code rationales.

2.3.2. RQ2: Concept Sketches

During the second portion of the focus group, participants were provided several examples of performance-based digital tools and directed to brainstorm what PD they would like to see for K-8 math and science teachers using digital performance-based tools like or unlike the given examples, focused on particular teaching competencies. The participants were shown three examples of digital tools: Mursion, Teacher Moments, and FACT. These three digital simulations in particular were selected to ensure representation of a reasonable variation of features and functions (

Table 4. Examples of performance-based digital tools.

Features	Mursion	Teacher Moments	Foundational Assessment of Competencies for Teachers (FACT)
Stimulus Type	Avatar-based	Scenario-based	Video Prompt (Avatars/Real Students)
Response Type	Verbal, Video	Text-based or Audio Record	Text Based or Audio Record
Feedback Type	Immediate, Self-Reflection, Post-Performance Feedback	Self-Reflection by prompts	Post-Performance Feedback
Time (Learning/Performance)	Real-time	Asynchronous	(Fixed Time to prepare and perform)
Adaptability	Adaptive	Deterministic	Deterministic
Number of Players	Single	Single	Single
Engagement Type	Interactive	Non-interactive	Interactive
Accessibility	Phone, Tablet, Desktop, VR headsets	Phone, Tablet, Desktop	Phone, Tablet, Desktop

) and because videos showing how these simulations work were accessible to the research team. Of note, the participants were reminded frequently that these examples were just examples of possible digital tools and invited to think broadly about genres. They were also told that although FACT was developed to be an assessment, they should only consider it an example of a digital practice-based PD tool and not an assessment.

Coding involved attention to two aspects of the transcript data. First, we coded for particularly critical features of the PD approaches. We coded features within a pre-determined list of features we had named and referred to during the focus groups, with the addition of one feature that was identified early on as prevalent (Table 4). As rater training, prior to paired coding, the full group of raters group-coded a single focus group transcript and met twice to reconcile, making updates to the codebook to ensure collective understanding. After rater training, each transcript was coded by a pair of raters who then met to reconcile discrepancies, coding for mention of the feature and describing the nature of the comment about the feature.

Utterances mentioning features were then gathered into a single document (e.g., one document included all utterances tagged as ‘feedback’), and one rater was assigned to read across the coding, group comments thematically, and summarize comments on that feature into analytic memos, with a second rater assigned to read behind and verify the accuracy of the summarization. Subsequent edits to the analytic memo were made in discussion between the two raters. For the Section 3, we summarized the analytic memos to provide the reader with a clear idea of what each theme represented, along with examples of the utterances used to identify the theme.

3. Results

3.1. RQ1: Priority Competencies

A listing of the competencies, showing their ranking by focus group, is shown in

Table 5. Competencies by number of focus groups in which they ranked among top three.

Competency	Literature/New a	# of FG in Which Competency Ranked in the Top Three	# of FG in Which Competency Was Discussed
Engaging students in collaborative problem-based learning	Both	6 (A1, B1, B2, C1, D1, D2)	8 b
Differentiation c	New	4 (A1, A2, C2, D1)	4
Eliciting student ideas	Literature	4 (A2, B1, B2, D1)	2 d
Promoting student engagement	New	3 (A2, B2, C1)	3
Connecting math and science to students’ cultures and communities	Both	2	2
Supporting student-generated evidence-based explanations	Literature	1	1
Noticing student thinking	Literature	1	1
Using teacher questioning	Literature	1	1
Creating space in classroom interactions for exploration, uncertainty, ambivalence, and different ways to express ideas.	New	1	1
Facilitating student discussions	Literature	0	0
Supporting students’ mathematics and science literacy	Literature	0	0
Engaging students in disciplinary argumentation	Literature	0	0

^a Competencies marked “new” resulted from the brainstorming within the focus groups rather than being generated from the literature review. ^b Competency was discussed in FGs A2 and C2 despite being ranked fourth in each due to a screen display error. ^c Differentiation was also ranked sixth in FG B1 but not discussed because it was not in the top three; it was not available for ranking in other groups because it was not nominated within the group. ^d Competency not discussed in A2 and B1 due to screen display issue.

ordered by the number of focus groups in which the competency was ranked among the top three for the group. Results are shown for all nine of the pre-determined competencies, as well as for participant-generated competencies that were ranked among the top three in at least one focus group. For the first four competencies listed (those that were ranked among the top three for more than 25% of the groups) we provide detail as to which focus groups listed them among the top three. We also note separately the number of focus groups during which the competency was discussed during the group, providing transcript data for coding. In most cases, the competency was discussed during the group if it was ranked among the top three, as this was the expected procedure. As noted in Table 5, a mentimeter screen update delay resulted in some cases in which data were not collected for a competency (eliciting student thinking had data from only two focus groups rather than four) or were collected in cases where they should not have been (engaging students in collaborative problem solving had data from eight focus groups rather than six) but did not change the competencies for which we had data available to analyze. In all cases, we elected to analyze the full set of data available in the transcripts even if a screen display error had caused us to have more or less data than expected.

Table 6. Rationales for importance of top four competencies.

Competency	Rationales for Importance
Collaborative Problem Based Learning	Supports student collaboration and socialization skills, which are even more important post-pandemic Supports student empowerment, motivation, perseverance in problem solving, social emotional wellbeing. Results in positive student learning outcomes Challenging to implement within existing curriculum and school structures
Differentiation	Challenging to implement due to challenges specific to diverse demographic student groups, scarcity of classroom resources, and the need to empower teachers so they could address the specific student needs. Insufficient PD and support for working with ELLs Represents a paradigm shift for some teachers whose training did not typically focus on differentiation
Promoting Student Engagement	Cultivates positive teacher–student relationships Ensures students feel valued within the classroom community Learning cannot happen without engagement Engagement ensures less off-task behavior
Eliciting Student Ideas	Supports adjusting instruction based on students’ current understanding Supports peer to peer learning Developing a safe classroom culture Valuing student voice Challenging to engage and implement

presents the most frequently named rationales for the top four competencies.

The sections that follow describe how participants conceptualized and justified the importance of each of the top four teaching competencies across all eight groups: (1) collaborative problem-based learning; (2) differentiation; (3) promoting engagement; and (4) eliciting student ideas. We reiterate that focus group methodologies do not necessarily quantify consensus; because participants are engaged in a conversation, they respond to the ideas presented by others, which may be novel and unlike ideas placed on the table in other groups. The purpose of these descriptions is to provide the reader with an understanding of how each competency was conceptualized in the cases that it was discussed, and a general sense of the types of statements that support our summarization.

3.1.1. Collaborative Problem-Based Learning

Collaborative problem-based learning was ranked within the top three across six focus groups, and participants described it in largely consistent ways, although there was some variation. Participants described collaborative problem-based learning as the teacher selecting a real-world problem with multiple viable solutions that is engaging and compelling for students, such as by incorporating students’ cultural and lived experiences. Participants emphasized the use of problems that support interdisciplinary learning and diverse skills alongside scaffolding to support student collaboration and discourse, including making contributions by building on one another’s ideas. Participants also described the importance of supporting student agency and autonomy during learning so that students are better prepared for real-world experiences that require collaboration and problem-solving. And, finally, in addition to choosing problems well, there was also a focus on supporting collaboration skills in general. As one participant from focus group D2 (Secondary Teacher Group 2) stated, “The collaborative part of this is also important getting our students to understand how to work in a group. How to ensure that you are contributing to the group and not just throwing your ideas out there, but building on someone else’s extremely important, especially outside of the school setting” (Participant 05, Secondary Teacher Group 2).

The idea of collaboration and socialization skills also came up when participants were prompted to discuss why collaborative problem-based learning was so important, with participants citing the need to support students’ development of these skills as particularly acute in the aftermath of the COVID-19 pandemic, wherein students lacked opportunities to practice and develop those skills. In addition, participants asserted that engaging students in this type of learning supports student empowerment, motivation because, in the words of a participant from Group B2 (District/Administration Group 2), “When you tie learning together and make it purposeful for a broader purpose, then kids get more excited—they’re engaged, they want to do something” (Participant 02, District/Administration Group 2). Another participant in the same focus group built upon that idea by adding that “promoting collaboration, you’re also probably going to help with overall social, emotional well-being and the confidence of their student” (Participant 04, District/Administration 2). Participants in one group also extended this idea to describe the benefits of engaging students in a productive struggle to normalize the experience of struggling to solve a problem, which in turn promotes perseverance.

In considering the need to develop innovative PD opportunities to support teachers’ capacity for facilitating collaborative problem-based learning, participants asserted that creating and structuring collaborative environments and supporting students as they engage with ill-defined problems is quite difficult, particularly for novice teachers who may feel nervous about the content. A participant from Coaches/TE/PD Group 1 asserted that “a lot of teachers, especially at elementary school, don’t want ill-defined problems that may have multiple avenues or multiple answers… I think sometimes that cautiousness around problem-based learning or setting up those kinds of structures sometimes stems from their own nervousness about the content” (Participant 08, Coaches/TE/PD Group 1). In addition, some participants described school and district level demands to prioritize collaborative problem-based learning over other teaching approaches and skills due to the possible benefits for students, further necessitating PD in this area that may be difficult for teachers with varying degrees of experience and content knowledge. Nearly all of the focus groups highlighted that positive student-learning outcomes result from engaging students in collaborative, student-led learning opportunities. However, some participants also described the challenges of engaging in problem-based learning within a curriculum or school that is ill-structured for supporting problem-based learning, such as lacking structures for engaging students in interdisciplinary learning.

3.1.2. Differentiation

Differentiation was discussed across four focus groups (Elementary Teacher Group 1, Elementary Teacher Group 2, Coaches/TE/PD Group 2, and Secondary Teacher Group 1) and was characterized by all groups by the need to adapt instruction to meet the individual needs of different students; however, the ways in which participants characterized those differences varied greatly. In two groups, participants described differentiation in relation to learning styles1. Some participants made a specific reference to learning styles theory, such as describing students as auditory or kinesthetic learners, while one participant in Elementary Teacher Group 2 made a more general assertion about modality: “some kids learn better on the iPad, and some kids learn better with the worksheet in front of them, it just really depends on what their learning style is and how we can engage them differently” (Participant 05, Elementary Teacher Group 2). One participant in Coaches/TE/PD Group 2 discussed the need for providing students with multiple means of demonstrating knowledge and offering student choice of how to show mastery by offering assessments in different modalities. Another participant in Elementary Teacher Group 2 discussed ways in which they adapt teaching to support learning: “Let’s say I take a lesson about animals. I’ll have a PowerPoint presentation on the board for those that need to see. I’ll have, you know, maybe an iPad or Google Lapbook for those that need to touch and worksheets for those that need to read” (Participant 08, Elementary Teacher Group 2).

Another crucial aspect of differentiation was attending to students’ individual needs by providing scaffolding, which was discussed in three groups: Elementary Teacher Group 2, Coaches/TE/PD Group 2, and Secondary Teacher Group 1. For example, one participant in Secondary Teacher Group 1 described a tiered system with three different versions of the same task, and “depending on the struggle that I see the student have, I will pass out an index card which will have a little bit more scaffolding, depending on where I think they’re at” (Participant 02, Secondary Teacher Group 1). Other adaptations to instruction included grouping students homogenously and providing groups with different instruction or levels of scaffolding, which was discussed by two groups (Elementary Teacher Group 2 and Secondary Teacher Group 1). One participant in Elementary Teacher Group 2 described how they utilize homogenous grouping using the breakout rooms feature on Zoom: “I normally have the high group, and like I might check on them one time, and they’re just working on like an assignment. And then, and you know, the other rooms I might visit more frequently, and they might be given different problems based on their need and what they’re understanding” (Participant 06, Elementary Teacher Group 2). Participants described grouping students according to known or assumed individual characteristics, such as learning styles, prior knowledge, gender, learning needs, and test scores or other assessments. For example, a few participants grouped students by gender largely due to perceptions that boys may be more hyper and need different activities than girls.

Participants expressed a variety of rationales for how and why it was important to differentiate instruction, including challenges specific to student demographic groups (expressed by all four constituencies), the inadequacy or scarcity of classroom resources (Elementary Teacher Group 2 only), and meeting the needs of diverse learners by empowering teachers (Coaches/TE/PD Group 2 and Secondary Teacher Group 1). Demographic groups with specific needs identified by participants included student ability levels, literacy skills, and variation in attention span and the ability to sit still. An example offered by a participant in Elementary Teacher Group 2 included “If a kid has ADHD, you might give them a shorter assignment, you might break things up more for them” (Participant 07, Elementary Teacher Group 2). Regarding resource scarcity, another participant in Elementary Teacher Group 2 commented that many of the resources available to educators at their school are more tailored toward general education rather than special education students, with an abundance of resources that work for some but not all students, resulting in the need for teachers to modify and scaffold existing resources for students (Participant 05, Elementary Teacher Group 2). With regard to empowering teachers to meet the needs of diverse learners, a participant from Secondary Teacher Group 1 commented on shifting school demographics because “there’s more students who are moving to the country, who have different backgrounds. And you know, just different cultural changes” (Participant 06, Secondary Teacher Group 1).

As to why differentiation is important to focus on for the development of innovative practice-based PD, there was a general sentiment that differentiation is difficult, and thus, there is a higher need for supporting students. One participant in Elementary Teacher Group 1 described teaching in a school comprised of “about 73% English language learners, and the areas that they seem to be struggling in is what are problems and just the academic language overall, and we don’t get a lot of PD support and how to break down the academic language for them” (Participant 02, Elementary Teacher Group 1). A different participant in Secondary Teacher Group 1 echoed this sentiment of needing more support, stating, “As far as prioritizing, this is an area of opportunity where traditionally maybe in the science classrooms in particular, we haven’t differentiated and so it is kind of a paradigm change. And so I think, putting it up at the top is a way for all of us to kind of recognize. This is an area for us to improve, to meet the needs of our students” (Participant 07, Secondary Teacher Group 1). This demonstrates a particular need to attend to supporting differentiation in different disciplinary contexts.

3.1.3. Promoting Engagement

Promoting engagement was characterized by participants across the four focus groups in which it appeared (Elementary Teacher Group 2, District/Administration Group 2, Coaches/TE/PD Group 1 and 2) in a variety of ways, including comments that described student behavior more clearly than the teaching moves that support it. For example, participants cited students demonstrating on-task behavior, participating in exploration and meaning-making, and being attentive and focused on learning as signs of engagement. Participants asserted that teachers promote engagement when they engage in student-centered instruction and work to elicit student curiosity and interest, as articulated by a participant from Coaches/TE/PD Group 2: “I think of being more student-centered and not always seeking compliance in the classroom, but rewarding curiosity and questioning, wanting to know more about that topic, to really draw [students] in to be interested in the topic, and to even seek out answers or seek out processes that answer those questions” (Participant 03, Coaches/TE/PD Group 1). This contrasts with some of the perspectives from groups Elementary Teacher Group 2 and District/Administration Group 2, which focused on student behavior aligning with the teachers’ expectations. A participant from Elementary Teacher Group 2 asserted that a lesson has “flopped” when “you have students just staring at you, or getting up out of their seats, going and blowing their nose, and you know, sharpen their pencils and what not” (Participant 03, Elementary Teacher Group 2). Other participants in the same group focused more on the teacher’s behavior to promote engagement, suggesting that student engagement happens when “the teacher might be doing less while the students are doing more” (Participant 07, Elementary Teacher Group 2) during student-centered instruction. Participants across three focus groups (Elementary Teacher Group 2, District/Administration Group 2, and Coaches/TE/PD Group 1) also drew connections between differentiation and promoting engagement, seeing them as either wholly intertwined or positioning differentiation as the foundation that supports student engagement.

The cultivation of a positive student–teacher relationship as part of promoting engagement was noted by three focus groups (Elementary Teacher Group 2, District/Administration Group 2, and Coaches/TE/PD Group 1) and was connected to the idea of ensuring that students feel valued within the classroom community. Two groups discussed how promoting engagement, and, in particular, helping students feel valued, is supported by drawing real-world connections (Coaches/TE/PD Group 1) and connecting disciplinary learning to students’ cultures, communities, and lived experiences (District/Administration Group 2 and Coaches/TE/PD Group 1). One participant from District/Administration Group 2 elaborated on the importance of attending to students’ identities and cultures, “because if the students don’t see themselves in what they’re learning, or don’t feel valued as a learner in the classroom, they’re not going to be engaged. So I guess, recognizing… the classroom community and the different cultures that make up that community” (Participant 02, District/Administration Group 2). In Coaches/TE/PD Group 1, one participant noted that “teachers will often complain that students are apathetic, that they’re disinterested… as they get older, it, they become very interested in social aspects… there are things outside of school that are going on and so it’s all about engagement… real world connections, and cultural connections. Because kids need to know, why is this important to me? Why does it matter? It’s not just something that my teachers telling me I need to know” (Participant 04, Coaches/TE/PD Group 1). However, another participant in this same group contested this assertion: “Some of these kids, you know, they’re coming from tough situations. They’re really not gonna care about school. And I think we can get engagement by making things fun, even though it’s not gonna maybe be particularly meaningful or relevant to them. Sometimes our focus is like, let’s make it fun, like, let’s sneak in the vegetable, so to speak so that they’re still learning happening despite the attitudes, instead of directly addressing the attitudes” (Participant 09, Coaches/TE/PD Group 1). These examples demonstrate that, while participants agreed on the importance of promoting engagement and generally agreed about what an engaged class would look like, they held a variety of views on how to best support student engagement, particularly for those students for whom the teacher has attributed their low student engagement to challenges within the students’ lives outside of school.

Other justifications for the importance of promoting engagement included perceiving student engagement as integral for student learning by asserting that learning cannot happen without engagement. In addition, two groups (Elementary Teacher Group 2 and District/Administration Group 2) emphasized promoting engagement because high levels of engagement were associated with less off-task behavior, rather than on the impact of promoting engagement on learning directly. Participants were more likely, for promoting student engagement, to discuss the importance of the competency in terms of its payoff than in terms of its difficulty or the need for more support for teachers.

3.1.4. Eliciting Student Ideas

For eliciting student ideas, this competency was described along four characteristics, all of which appeared in both groups that discussed this competency: (a) eliciting students’ prior knowledge; (b) understanding students’ current thinking to identify gaps in student knowledge and understanding; (c) students participating in expressing and articulating their ideas; and (d) supporting student agency and ownership of learning. Overall, participants’ discussion of how they conceptualized eliciting student ideas was more consistent than the other competencies previously described, although this may simply reflect the lower number of groups in which it was discussed.

One participant in District/Administration Group 1 described eliciting student ideas as “When students are talking, you not only hear what they understand and where they are, you hear what they’re really confused about. Because they may say something that they think is, you know, the right answer. And, and all this, and you’re going okay, and we will start there and unwind that” (Participant 08, District/Administration Group 1). This idea of eliciting student ideas as a springboard for learning was echoed by others in the same group, such as by another participant who described developing next curricular steps based on the student thinking that has been expressed. Participants also described having students sharing their ideas as providing students with a shared experience to return back to which supports the development and internalization of students’ vocabulary for expressing their ideas.

When asked to justify the importance of this competency, responses across both groups varied. In both groups, participants discussed the importance of eliciting student thinking in order to adjust instruction based on what students do and do not understand, especially because having students articulate their ideas to one another as peers supports students in making connections and understanding what they do and do not know. Secondary Teacher Group 1 further discussed the value of peer collaboration as part of eliciting student ideas because “When students hear other students express ways to solve problems, something might resonate with them” (Participant 01, Secondary Teacher Group 1). Participants attributed this benefit to students being able to hear ideas expressed in more familiar terminology from their peers rather than in the terms that may be used by textbooks and teachers. District/Administration Group 1, on the other hand, focused on the importance of developing a classroom culture that feels like a safe space to engage in the risk-taking associated with students expressing their own ideas. Participants in District/Administration Group 1 associated the cultivation of such a space with valuing student voice so that “students know that we’re there because of them, and that they are the center of the whole learning” (Participant 08, District/Administration Group 1). Secondary Teacher Group 1 also highlighted that engaging in this competency is particularly challenging, further emphasizing the importance of PD for this competency. District/Administration Group 1 described the importance of PD opportunities that help to surface “current levels of understanding about what these practices look like in the classroom and provide some examples to really assess their thinking” (Participant 01, District/Administration Group 1).

3.2. RQ2: Critical Features of PD

Table 7. Features discussed in each focus group by competency.

Focus Group	Competency Name	Time	Stimulus	Preparation	Response Type	Feedback	Content and Grade Level
A1	Differentiation	✓	✓	✓	✓	✓	✓
A1	Engaging in collaborative problem-based learning	✓			✓
A1	Connecting math/science to cultures and communities	✓	✓		✓	✓
A2	Differentiation	✓	✓	✓	✓	✓	✓
A2	Promoting engagement		✓	✓	✓	✓	✓
A2	Connecting math/science to cultures and communities	✓	✓		✓
B1	Connecting math/science to cultures and communities	✓	✓	✓	✓	✓
B1	Facilitating student discussion	✓	✓		✓	✓
B2	Promoting engagement	✓	✓	✓	✓	✓	✓
B2	Engaging in collaborative problem-based learning	✓	✓	✓	✓	✓	✓
B2	Connecting math/science to cultures and communities	✓	✓	✓	✓
C1	Noticing student thinking	✓	✓	✓	✓	✓
C1	Promoting engagement	✓	✓	✓	✓	✓	✓
C1	Connecting math/science to cultures and communities		✓	✓	✓	✓	✓
C2	Differentiation	✓	✓	✓	✓	✓	✓
C2	Creating space and classroom interactions for exploration, inquiry, uncertainty, ambivalence, and different ways to express ideas	✓	✓	✓	✓	✓	✓
C2	Connecting math/science to cultures and communities		✓		✓		✓
D1	Engaging in collaborative problem-based learning	✓	✓	✓	✓	✓	✓
D1	Connecting to students’ culture and communities		✓		✓	✓
D2	Connecting math/science to cultures and communities	✓	✓		✓	✓	✓
D2	Using teacher questioning	✓	✓	✓	✓	✓	✓
D2	Engaging in collaborative problem-based learning	✓	✓	✓	✓	✓	✓

Key: A = Elementary school teachers; B = District/School administrators; C = TEs/coaches/PD providers.

shows which features were discussed in each focus group session during the discussion of the competencies listed. For each focus group, the competencies are listed in the order they were discussed. Of note, every feature was discussed at some point in time during every focus group session, with the exception that content and grade level was not discussed in District/Administration Group 1. Response type was the sole feature that was discussed with respect to every competency in every session. We do not report on the proportion of time spent on each feature because the amount of time allocated per competency varied considerably between focus group sessions and was, in many cases, a direct function of the focus group leader’s in-the-moment decisions about when to move on rather than reflecting the feature’s relative importance among participants. As noted previously, the mentimeter delay did not affect which competencies were discussed with respect to critical features of PD, but there were two focus groups for which only two competencies were discussed in total due to time constraints.

In the following section, we discuss the most prominent themes that arose in discussion of the desired features of PD (

Table 8. Most frequently mentioned themes listed by feature.

Themes	Description	Focus Groups in which Themes Appear
Stimulus
Avatars approximate real students	Describes a desire for the representation of students to be as accurate as possible	B2, C2, D1, D2
Teachers’ Agency and Choice to select Simulation features	Suggests participants in the PD should have agency to set some of the parameters of the experience, including choosing background characteristics of student avatars	A1, B1, B2, C1, D1
Modeling Teaching Practice	Suggests a model of the teaching practice be available as part of what the participant is provided	A2, B1, B2, C2, D2
Response type
Practicing with avatars/simulations	Suggests using avatars and/or simulations and practicing in a virtual classroom, sometimes as a preamble to trying specific strategies in the classroom.	A1, A2, B1, B2, C1, C2, D1, D2
Feedback
Feedback from knowledgeable other	Includes references to receiving feedback from a coach, administrator, or other experienced individual, especially one with local knowledge and strong teaching expertise.	A1, A2, B1, B2, C1, C2, D1, D2
Feedback from peer	References to peer feedback often included more detail than simply peer observation.	A1, B1, B2, C1, D1, D2
Feedback from Avatars and real students	Involves feedback received from avatar or real students	A1, A2, C1, D1, D2
Self-Reflection	Refers to comments referencing to reflecting on the feedback	A2, B1, D1
Point at which feedback is given	Comments reference three time points where feedback can be given: along the way, immediately after, and after the fact.	A1, C1, D1
Set goals and Criteria for Performance and Judgment	References to comments about setting expectations, and clear standards of success	A1, A2, C1, D1, D2
Content/Grade Level
Content should be proximal	References to content tailored to what the teachers are teaching.	A2, B2, C2, D2
Adapt to different needs of students	References to content that reflects students that teachers are teaching.	A1, A2, C2, D1
Using virtual tools/simulations	References to the potential affordances of such approaches being digital.	A1, B2, D1, D2
Time
Multiple PD Sessions and Iterative Cycles	References to needing multiple sessions and iteration with feedback/reflection.	A1, A2, B1, C1, D1, D2
Shorter Duration Tasks	References to keeping the tasks reasonably short, allowing for repetitio where appropriate.	A1, B2, D2
Preparation
Time and Opportunity for Preparation and Planning	Reflects the need for time and opportunity to prepare for digital simulations.	A2, B1, B2, C1, C2, D2
Other
Use of Technology in PD (Simulations/AI, etc.)	References to where technology such as AI could automate processes.	B2, C1, D1, D2

). While the focus group methodology does not readily lend itself to quantification as a measure of consensus, we limit our presentation of the results as follows to avoid over-representing themes that were not discussed frequently or across participant group classifications: (1) themes are listed if they emerged in discussion in a minimum of four of the eight sessions (50%) or (2) themes are listed if they emerged in discussion in a minimum of three of the eight sessions (37.5%) if those three sessions represent three different participant groups.

3.2.1. Critical Feature: Stimulus

Three prominent themes emerged from the data coded as discussion of the stimulus (Table 8), which included what participants in the PD would be given to prepare and respond to in the performance-based situation.

The first theme ‘Avatars approximate real students’ appeared in four focus groups —Elementary Teacher Group 2, District/Administration Group 2, Coaches/TE/PD Group 2, and Secondary Teacher Group 2. In general, comments within this theme reflected a Mursion-like environment and a desire for the representation of students to be as accurate as possible, including with respect to diversity: “And I’d like the idea of having different types of students in our classroom, because in a reality we do have different types of students” (Participant 07, Secondary Teacher Group 2).

A clear rationale for the need for authenticity was that avatars need be authentic enough to ensure participant buy-in, and there was some skepticism about whether this is possible. While most groups gravitated toward the Mursion format, some participants in Elementary Teacher Group 2 and District/Administration Group 2 felt that doing the lesson with real students, either within a real classroom or with real students controlling the avatars, would be a better approach: “you use those [real students] as the avatars, and they don’t have to reveal their real identities… It’s gonna give you some more authenticity. If you’re using real students, I mean, it can be like, I. So, I think someone like I say it can be avatar or whatever, and they can be acting as an avatar. But it I just think it has to be some type of real, so real students” (Participant 02, Elementary Teacher Group 2).

The second theme, ‘Teachers’ Agency and Choice to select Simulation features’, appeared in six focus groups: Elementary Teacher Group 1, District/Administration Group 1, District/Administration Group 2, Coaches/TE/PD Group 1, and Secondary Teacher Group 1, representing the idea that participants in the PD should have agency to set some of the parameters of the experience, including choosing background characteristics of student avatars. One participant, for example, suggested a drop-down menu that would allow you to choose cultural characteristics close to your own local areas, stating that this would increase engagement: “Yeah, I think just having that flexibility… even if it’s just a dropdown menu where you kind of pick like the culture, and then the PD will be able to just instantaneously adapt to a lot of those cultural norms……… That would be……. really great cause… you could really connect with the PD a lot more” (Participant 08, Elementary Teacher Group 1). Others mentioned wanting to choose something about the task focus itself. For example, in both Coaches/TE/PD Group 1 and Secondary Teacher Group 1, participants suggested the idea of a task library to choose from.

The third theme ‘Modeling Teaching Practice’ appeared in five focus groups: Elementary Teacher Group 2, District/Administration Group 1, District/Administration Group 2, Coaches/TE/PD Group 2, Secondary Teacher Group 2, and reflects the suggestion that a model of the teaching practice be available as part of what the participant is provided. While most references were to an exemplar of desired practice, one participant suggested that examples of different levels of success would be useful for sustaining engagement: “I was also thinking that it might be kind of good if at the end of whoever participates in the simulation that they could get levels kind of like, could they watch a simulation that somebody else did, that they might have done an excellent job, at an average job at, and that they failed, and maybe in different ways. For a struggling teacher, it’d be nice if they leveled it so they could start with kind of easy stuff, easing them into that, and then the average. And then, like, we want you to be proficient with most of our students these days which are having behavior problems, but kind of maybe work them into it so that they stay engaged. And they don’t give up if they know they’re failing with that simulation.” (Participant 03, District/Administration Group 2).

3.2.2. Critical Feature: Response Type

As noted previously, response type was the only feature that was discussed in every focus group for every competency. Only one theme was prominent with respect to response type: practicing with avatars/simulations (Table 8), and this theme also came up in every focus group, suggesting that using avatars and/or simulations provides an opportunity to engage and interact before entering the classroom, think on your feet and adapt quickly, and modify the lesson as needed. In most cases, the referenced simulation type was Mursion, but there were exceptions. In District/Administration Group 1, for example, participants suggested using Teacher Moments then Mursion in that order to provide a scaffolded learning experience: “I was just gonna say my point was kind of scaffolding. like starting with chunks and letting them do a little bit of practice with some short pieces of teaching, and then moving on to like putting those together with the avatars.” (Participant 08, District/Administration Group 1).

Although interactive avatar simulation was a clear preference, there was not always agreement, even within a single session, about the affordances of simulations and, in particular, about what it means that it is a performance. For example, most participants in Secondary Teacher Group 2 preferred the Mursion interactive format, contrasting it to the types of PD they engage in now: “…the very first one that we saw where it was all interactive. That was much better to PDs I attend now.” (Participant 07, Secondary Teacher Group 2) Another participant in the same session, however, described Mursion as “the worst one” (Participant 05, Secondary Teacher Group 2), because of the time requirement and the possibility that someone will judge the performance. Their description leans toward seeing simulation as evaluative and invasive: “Minutes of required. You’re on video. You’re on audio…. You could end up being graded by how you do on these simulations versus how you actually are in your classroom” (Participant 05, Secondary Teacher Group 2).

3.2.3. Critical Feature: Feedback

The feature feedback included reference to when feedback is received, from whom, and the nature of the feedback, including self-reflection, personal or general feedback, and at varying levels of formality (Table 8). We also included, under the feedback feature, any material given to the participant that would establish a standard of quality against which they might evaluate their own performance. Multiple themes emerged from the discussions of feedback, including types of feedback: from a knowledgeable other, peer feedback, feedback from students/avatars, self-reflection. Other themes reflected aspects of when feedback should be provided and setting goals and criteria.

The first theme, ‘feedback from knowledgeable other,’ was evident in all eight focus groups and included references to receiving feedback from a coach, administrator, or other experienced individual. Many of these comments related to the expertise of the individual, with a strong preference for someone who knows the local environment and who really knows what effective teaching practice is and how to enact it. Another widely shared idea was that of the personal touch: “And so I would, I would say an important component would be for your adult avatar to not be an avatar, but to be an actual person. And so you meet the person ahead of time, you know. Hi! My name is [name]. I’m gonna be your coach today. Here’s what we’re gonna do… then you have a real live person, and it feels it feels like someone cares about you” (Participant 01, District/Administration Group 1). That said, there were some reservations, as noted under response type, that feedback could be used punitively, and in both District/Administration Group 2 and Secondary Teacher Group 2, the need for the feedback to be confidential was surfaced.

The second theme, ‘feedback from peer’, appeared in six focus groups. One participant described a cohort model and how that might be managed within an online system: “I think about when, like my favorite coaching was when I could have like a cohort, and we were observing each other right classroom, and then coming back for the debrief conversation. And then in some cases we would observe the coaching conversation that’s happening” (Participant 07, Coaches/TE/PD Group 1). The affordance of asynchronous observation was also mentioned: “You all should go in and watch another teacher and give feedback, and it’s just not enough time in a day. You don’t have enough subs to be able to do it, but with that first one [Mursion] is very, very possible” (Participant 07, Secondary Teacher Group 2).

The third theme, ‘feedback from Avatars and real students’, appeared in five focus groups: Elementary Teacher Group 1, Elementary Teacher Group 2, Coaches/TE/PD Group 1, and Secondary Teachers Groups 1 and 2. Participants noted that feedback directly from avatars could be more valuable than expert feedback: “I would want feedback from the avatars, like what they learned, what they were still confused by… I would value that more honestly than like an outside expert’s opinion. Because that’s what we’re really going for. Like, do the students understand? And what did they like about the lesson? And what did they not?” (Participant 01, Secondary Teacher Group 1).

The fourth theme ‘Self-Reflection’ appeared in Elementary Teacher Group 2, District/Administration Group 1, and Secondary Teacher Group 1. Most mentions of reflection were about reflection after the fact, with guided questions, but one participant suggested taking advantage of the affordance that simulations need not progress in linear time to allow pausing to reflect at any time: “I also want that. I want it to be an option like any point I can hit self-reflect area. And I could type in like, hmm, I’m noticing I’m struggling with this or after I’ve read my feedback” (Participant 01, Secondary Teacher Group 2).

This last point relates to the fifth theme, ‘point at which feedback is given,’ appearing in Elementary Teacher Group 1, Coaches/TE/PD Group1, and Secondary Teacher Group 1. Comments reference three time points where feedback can be given: along the way, immediately after, and after the fact. Several participants mentioned the value of immediacy in feedback, and one noted that ideally feedback would be immediate but also accompanied by a written record to support reflection, even if that report comes later: “I think that immediate feedback is good, but then also having some type of report that’s generated. If we’re focused on some type of teacher competency, then there needs to be a criteria of whether we’re meeting that competency or whether we’re not and have it be clearly defined. So and then areas of growth that are also pinpointed. So that way, we actually have something to build on” (Participant 03, Elementary Teacher Group 1).

The sixth theme, ‘Set goals and Criteria for Performance and Judgment’, appeared in Elementary Teacher Groups 1 and 2, Coaches/TE/PD Group 1, and Secondary Teacher Groups 1 and 2. One participant noted that setting expectations is critical: “I really think it’s important that that the teachers don’t just go through a lesson and then walk away, that there’s kind of not to say requirements, where they’re forced to do certain things. But there is some kind of goal setting. There is some type of expectation of. You know, you’re gonna try certain techniques. It’s so easy to do PD and just do it once and done and literally be done” (Participant 07, Secondary Teacher Group 1).

3.2.4. Critical Feature: Content/Grade Level

Three prominent themes emerged related to the content or grade level of the PD. The first, ‘content should be proximal’, reflected the need for the content represented in the PD to be closer to what teachers are teaching in the classroom, with the clear rationale being the immediacy of impact on teaching. As one participant put it, “I don’t wanna watch a professional development for someone who teaches eighth grade when I teach first graders, I want, like the topics that I have to teach and the strategies that are gonna be effective with the age group that I have so making it targeted for. I don’t. Even if you could get down to the grade level on the subject, I think that would be most helpful” (Participant 7, Elementary Teacher Group 1).

The second theme, ‘adapt to different needs of students’, appeared in Elementary Teacher Groups 1 and 2, Coaches/TE/PD Group 2, and Secondary Teacher Group 1. This came up most frequently with respect to differentiation: “Well, I would think that it’s gotta be embedded in in some sort of, you know curriculum, so that you know it’s how you actually adapt this specific curriculum or this specific exercise for your different groups of students instead of just generalize like, you know, what?” (Participant 2, Coaches/TE/PD Group 2). However, participants also discussed this adaptation to reflect different students as related to their own learning needs: “So when I am finished, everything has been centered around me and personalized to me again. I mean, we always talk about personalized learning for students. But we haven’t really focused on personalized professional development for teachers. And I think this gives us an opportunity to do that by having that menu where we have these different, drop-down boxes where we could either change certain things or had certain features and then have the PD customized to exactly what we need. And so, if you want that lesson for tomorrow morning, the Pd. Is going to be based on that specific lesson based on the parameters that you have set. So that would be amazing” (Participant 2, Secondary Teacher Group 2).

The third theme, ‘using virtual tools/simulations’, appeared in four focus groups: Elementary Teacher Group 1, District/Administration Group 2, and Secondary Teacher Groups 1 and 2. While overlapping with some prior ideas, this theme represents specifically how virtual tools and approaches could create affordances. For example, in one focus group, the participants discussed generating their own simulations which would align to their needs but which then could be shared elsewhere: “So I think would be really amazing for each Plc team to be able to create some type of virtual interactive model of what their common challenge is. And typically, it’s around. You know, this grade level is working on how to incorporate math and different content areas, or how to differentiate instruction in, you know, reading whatever it happens to be. But if they could create something like this, and then other PLCs [professional learning communities] could utilize that tool. You would have amazing firsthand PD. Without having to sit through weekly Plc meetings to get and understand what their common challenge is. So I think it would just really streamline” (Participant 1, Elementary Teacher Group 1). Another suggestion was a system that could automatically generate rich problems based on prior lessons: “Maybe thinking of like a lesson that you’ve done over the you know, recent few weeks. And that’s something that you input and whatever system works to help build some kind of really solid question that could be a project problem-based learning for your class. And then, after those first few minutes it, it goes into how to actually like, facilitate it” (Participant 4, Secondary Teacher Group 2).

3.2.5. Critical Feature: Time

Two prominent themes emerged with respect to time: the need for multiple, iterative cycles (present in six focus groups), and the need for shorter tasks (present in three focus groups). The first theme included references to trying something a second time after receiving feedback or reflecting and the need for deep, sustained practice over time, and experiences that build on one another. The second reflected the potential affordance of trying out multiple approaches to the same challenge in close proximity to inform reflection about how the scenarios played out differently, as well as the observation that a shorter (20 min) session would likely be as beneficial to learning as a longer one, making it pointless to go longer than needed.

3.2.6. Critical Feature: Preparation

The one prominent theme with respect to preparation reflected the need for time and opportunity to prepare before engaging in something like a digital simulation, and this appeared in six focus groups: Elementary Teacher Group 1, District/Administration Groups 1 and 2, Coaches/TE/PD Groups 1 and 2, and Secondary Teacher Group 2. In part, this reflected a general sense that teachers never have enough time to plan: “And then, if I really wanted, you know, if I’m really going sky, you know, sky high here. provide opportunities for teachers to actually sit down and plan, because that’s the biggest complaint I get from my staff is, you know, we don’t have planning time, even though this year they have 2 preps which I get. It’s still not enough. A nice planning time, and maybe even have a mentor teacher to plan with them, or the teachers who are experts at it, to plan with them. If I’m really reaching and going for the, you know, going for the for the big pie in the sky. That would be my one, my dream PD” (Participant 4, District/Administration Group 1). Other rationales included that the target competencies, such as differentiation, might be difficult and require preparation, or that customizing the lesson as described under other features might take some time.

3.2.7. Other Themes

We captured themes under the code ‘other’, when they did not map onto one of the features listed. One theme, ‘use of technology/automaticity/AI’, emerged across four focus groups: District/Amministration Group 2, Coaches/TE/PD Group 1, and Secondary Teachers Groups 1 and 2. Generative AI was mentioned frequently in terms of the possibility it might generate scenarios on the fly, animate student avatars, or generate feedback on performances. There was skepticism about AI’s ability to do these things well, and an acknowledgement of the potential dangers, but also a fairly widespread level of comfort with the idea that we are probably close to a point where AI can be imagined as successful in such uses, and an appreciation for the convenience it might afford. In a few cases, individuals expressed preference for AI. One participant, for example, noted that they would feel trust in AI provided it demonstrated the right level of expertise: “I’d be okay with it as long as I knew with some, or, you know, chat, a chat, GPT. Or whatever. As long as I felt like the person or whoever had experience, you know, and expertise in the area, I would be okay with it…” (Participant 06, Elementary Teacher Group 2). Another form of automaticity that was mentioned was the possibility of linking the PD system to district databases that would facilitate some of the customizations mentioned under other features (student demographics, curricular approach, etc.).

4. Discussion

4.1. Discussion of Findings

Our results suggest several takeaways about the most critical competencies to focus on in developing performance-based PD for K-8 math and science teachers. We did not see full alignment between the literature-identified competencies we identified in advance and those most often identified as among the top three competencies in the focus groups. In particular, it stands out that differentiation was identified in half of the focus groups as a top three competency despite it not having been identified in the literature on teaching practices specific to mathematics and science instruction (Darling-Hammond et al., 2017; Desimone & Garet, 2015). While speculative, one could imagine that differentiation might be a higher priority post-COVID with the literature lagging behind a bit in emphasizing it, or that differentiation might be treated in the literature as content-agnostic, making it less likely to come up in a search of mathematics and science teaching, but still a challenge for teachers to implement in the context of mathematics and science. Regardless, this finding suggests that the field may need more research on how to differentiate instruction effectively in K-8 mathematics and science classrooms, and in how to support teachers in learning to do so. Also of note is what did not come up—for example, skills such as classroom management were not named as critical competencies, although some participants linked student engagement with preempting management issues, aligning with the findings of Pianta et al. (2012) that fostering engagement is central to maintaining a productive classroom environment. This suggests a reframing of priorities, where engagement subsumes traditional management concerns.

We also note that there was no clear evidence that different stakeholder groups have divergent views on what is most critical. There was variation between the focus groups, both across and within stakeholder categories. We cannot, of course, discount the possibility that there might be variation hidden due to the focus groups methodology, in which a particular group can self-reinforce along a direction introduced by an individual or small subset of the vocal participants (Krueger & Casey, 2015). A follow-up study building on our findings and focusing on individual response could confirm or disprove our tentative finding.

In our questioning, we asked participants for rationales for why the named competencies were those for which it would be most important to develop professional learning opportunities. Some rationales focused on the importance or primacy of the competency itself, referencing payoff to student learning, or stating that something like engagement is essential because learning cannot happen without it. Likewise, some rationales referenced dependencies among competencies, like the need to differentiate to ensure student engagement or the need for promoting student engagement to head off classroom management issues. Other rationales focused less on the importance of the competency writ large and more on the need for support. These rationales included references to the relative difficulty of the competency, both in general, such as the difficulty of choosing good problems for collaboration, and in specific situations, such as differentiating for English Language learners. Finally, some rationales referenced external pressures, such as a school district placing greater emphasis on a particular approach or students being less well-prepared as a result of pandemic skill declines, as reasons greater professional support would be needed. We do not have rationales for why lower ranked competencies were ranked lower but can infer that they were likely ranked lower with respect to the same criteria, which could include being less important for student learning, more easily mastered, competencies for which quality PD is already available, or ones for which performance-based approaches make less sense.

With respect to the critical features of performance-based approaches, we saw no patterns overall or by stakeholder group in the degree to which particular features were mentioned, suggesting that all of the features discussed are important to consider for each competency. This aligns with P. Grossman et al. (2009), who emphasizes the centrality of task design and teacher performance in effective PD. The only clear trend was that preparation and content/grade level were mentioned in only about half of the discussions of connecting math and science to students’ cultures and communities. Response type, as noted previously, was the only feature discussed for every competency in every focus group, which makes sense because a performance-based task is largely characterized by what performance it requires.

Generally speaking, participants were drawn to more interactive PD, with a great deal of discussion of personalization (to district needs, population, teacher expertise level, grades and content). Participants seemed open to the incorporation of AI but conscientious of its potential drawbacks, such as skepticism around the capability of AI to generate realistic student responses (Luckin et al., 2016). They also really valued feedback but offered mixed reactions on who it should come from, with some concerns about privacy and generally a sense that feedback is more meaningful and better received when embedded in a relationship with someone who has deep expertise (Bryk & Schneider, 2002).

We also noted that some competencies elicited fewer clear responses—the clear example of this being connecting math and science to culture and community, which was seen as important but, when discussed in terms of PD, elicited little or no detail and, in some cases, explicit statements that participants did not really know what an interactive digital PD might look because culture and community are intensely local constructs. This finding resonates with Banks et al. (2007), who argue that cultural responsiveness in teaching requires deep contextual understanding that may not easily translate to standardized digital tools. This is valuable information to have; not all competencies will be best supported by digital interactive tools, and those tools should be restricted to competencies for which they make sense, with PD of other kinds reserved for other critical competencies.

It is also worth noting that while we did not ask the more basic question, ‘do you want this PD at all,’ there was a sense across participants and focus groups of tentative enthusiasm for this approach, particularly for the interactive approach. Coupled with this, however, was a sense of hesitancy about whether the vision of a fully customizable, focused, supportive system like those they were imagining was practically achievable and some hints that PD that did not meet these standards would be considered a waste of valuable time. This suggests that maintaining communication with stakeholders throughout the design process is warranted, as input on details is likely critical in determining how useful the designed PD tools will be, and how receptive stakeholders will be to their use.

This enthusiasm for interactive approaches suggests, as noted in the literature, that current PD is not entirely meeting stakeholders needs and that they would like something different. One policy implication is that districts might implement selection criteria for PD providers that emphasize interactive approaches. One focused implication for researchers and designers of such PD is that differentiation is an acute need post-pandemic (and may have been pre-pandemic as well) and more PD is needed to support teachers in this area, although our small sample makes this conclusion somewhat inferential. A more generalizable practice implication may be the value of centering practitioners’ voices and creating agency in systems that deliver PD so that we can be more certain we are responsive to their needs.

4.2. Limitations

Our study has several inherent limitations, both in the methodology employed and in the potential barriers to future implementation of an online PD platform. In terms of methodology, one limitation relates to the tendency of a group to converge around certain ideas that may not represent the individual thinking of each participant and a reliance on participants to generate ideas, which precludes having uniform data on all ideas, since some may not have been raised in other focus groups. Our sample was also limited in size and with respect to geographical representation. Our literature review, as described previously, was also limited in scope, and a broader reach might have captured some of the other competencies identified by stakeholders. We also had missing data in some groups. In Elementary Teacher Group 1, as noted in Section 2, we had not yet added the question asking participants to define competencies in every case; therefore, we do not have participant-generated definitions for the literature-supported competencies. As noted elsewhere, due to screen display issues, we have missing data where competencies were not discussed, although their rank order status suggests they should have been, and while the effects of this were minimal and there is no evidence that the issue was more systemic than the delays we accounted for, this does call into question the reliability of the software. Another limitation was the uneven prior exposure of participants to digital PD, which may have led some of them to base their perceptions of what was possible solely on our short demos despite our caution not to do so. Finally, participants had variable previous experiences with technologically mediated tools to support teacher PD, which could have resulted in potential biases in their feedback about the affordances and drawbacks to using digital, performance-based tasks to support teacher PD and the nature of the performance tasks and feedback that would be useful and possible in such an online PD platform.

Regarding future implementation, the skepticism expressed by participants points to possible barriers to implementing digital, performance-based tools as part of teacher PD across varied educational settings. Cost is always a concern, as developing and deploying such technological tools typically requires funding, which can be limited in school districts, and a number of the features that were suggested as important, such as easy customization, remain technologically challenging under current conditions. A related challenge is ensuring that the tool is sufficiently adaptable to meet the needs of teachers at differing levels of experience, with diverse student populations, and who teach across varied levels and content areas and topics, and in different locations. Finally, ensuring that the digital tools are easy to access and use is critical to ensuring their future and continued use. Future development and research should take these potential barriers into consideration and work to find practical, feasible, and comprehensive solutions, making sure to include practitioners as decision makers throughout the process.

5. Conclusions

Our results suggest that participants from all four stakeholder groups found the idea of performance-based PD to be worthwhile for math and science teachers. Broadly, they emphasized the importance of collaborative problem solving and differentiation as the most critical competencies, followed by eliciting student thinking and promoting engagement. This consensus aligns with prior research advocating for interactive, practice-based PD formats (Mikeska et al., 2023; Phelps et al., 2023).

Notably, differentiation emerged as a critical competency across several focus groups, despite its limited emphasis in the literature on math and science-specific teaching practices. This underscores the potential lag in research capturing the practical challenges teachers face in classrooms today, such as addressing diverse student needs post-COVID. In particular, the emergence of differentiation as a priority suggests one area where PD based on the mathematics and science literature on teacher learning might miss the mark. Conversely, competencies such as classroom management, which are often considered foundational, were not explicitly prioritized. This suggests that practitioners might implicitly link such skills to broader competencies like student engagement, reinforcing the importance of practitioner-driven insights in shaping PD.

These findings also resonate with the literature on high-leverage teaching practices (TeachingWorks, 2024) and effective PD strategies (Popova et al., 2022), which emphasize the need for PD to focus on skills that have a high impact on classroom practices, particularly those challenging to master through traditional training. Moreover, the prioritization of competencies such as eliciting student thinking and promoting student engagement aligns with the frameworks on core teaching practices (Charalambous & Delaney, 2019; Capobianco et al., 2018), underscoring the role of student-centered strategies.

Participants trended toward valuing interactive formats coupled with the opportunities for reflection and actionable feedback, further supporting prior findings on effectiveness of practice-based PD approaches (Keiler, 2018; Dieker et al., 2019; Mikeska et al., 2023). However, the varied responses regarding the applicability of this format across all competencies suggest that a one-size-fits-all approach may not be universally advantageous. This highlights the need for customizable PD offerings that incorporate diverse stakeholder perspectives.

Overall, the fact that the competencies prioritized by participants did not entirely align with the literature-informed lists (Bae et al., 2022; Calabrese Barton et al., 2020) suggests that it is worth consulting with practitioners and attending to a diverse set of stakeholder groups, as we did in these focus groups, because the literature may not give us an adequate view into what is valued by those closest to the classroom. While participants broadly supported the potential of performance-based PD, there was a cautious optimism about its practical implementation, with an emphasis on ensuring it remains customizable, contextually relevant, and supported by ongoing stakeholder engagement to maximize its utility.

Future research might prioritize the involvement of stakeholders as we did and might build on this work by conducting a broader survey of a more nationally representative sample to ensure greater generalizability. Further exploration is also needed to determine how competencies like differentiation can be effectively supported through performance-based PD, while addressing hesitancy around digital tools, such as AI, in generating realistic scenarios and feedback.