Identifying Professional Development in Teaching & Learning Needs in Higher Education: A Measure

Abstract

The evolving landscape of higher education necessitates a re-evaluation of professional development (PD) frameworks to support instructors in enhancing their teaching practices. This paper introduces the Professional Development in Teaching and Learning Recommender (PDTLR), developed to identify instructors’ needs based on salient categories of PD in teaching and learning common across higher education. Utilising Rasch Measurement Theory (RMT) and a sample of 245 university instructors from a state-funded university in Singapore, the PDTLR was found to provide a reliable and valid measure of PD needs in teaching and learning and addresses inadequacies in existing instruments for identifying higher education instructors’ PD needs. The PDTLR represents a significant advancement in identifying PD needs within higher education, offering a valuable resource for institutions aiming to foster continuous improvement and professional growth among their faculty.

1. Introduction

In recent years, the higher education landscape has undergone significant transformations, necessitating a re-evaluation of professional development (PD) frameworks that support instructors in enhancing their teaching practices. As the demands on higher education instructors (i.e., university teachers, sessionals, adjuncts, casuals, lecturers, or more broadly, educators) evolve, so too must the tools used to identify their PD needs and assess outcomes. This paper introduces the Professional Development in Teaching and Learning Recommender (PDTLR), a newly developed measure aimed at addressing critical inadequacies identified in the existing literature regarding the validation and effectiveness of instruments intended to identify higher education instructors’ PD needs in teaching and learning. The PDTLR is designed to provide a reliable and valid measure of PD needs in teaching and learning, utilising Rasch Measurement Theory (RMT) to ensure robust validation processes.

The literature reveals a concerning trend in that multiple existing measures of PD in teaching and learning within higher education lack rigorous validation and often rely on methodologies that present inadequacies. For instance, while some instruments have been developed to assess various competencies, they frequently fall short in terms of psychometric rigour, failing to apply modern test theories such as item response theory or RMT [1]. The PDTLR seeks to fill this void by employing RMT, which allows for a more nuanced understanding of the psychometric properties of the measure, including item endorsability, thereby enhancing the validity of the measurement outcomes [2].

Unlike factor analytic procedures which require item parameters to be sample dependent and response distributions to be normal, RMT offers a systematic approach to evaluating the psychometric properties of assessment tools and provides a framework for understanding how individual items function within the broader context of the measure [3]. This approach not only facilitates the identification of reliable and valid item scores but also ensures that the measure is sensitive to the diverse needs of higher education instructors across various disciplines and contexts [4]. By utilising RMT, the PDTLR aims to establish a robust foundation for assessing PD in teaching and learning needs, ultimately contributing to improved educational practices and outcomes.

In the subsequent sections of this paper, salient categories of PD in teaching and learning that can be reliably identified through the PDTLR are presented. These categories, informed by empirical research and validated through rigorous psychometric analysis, will serve as a framework for instructors seeking to enhance their teaching practices and improve student learning outcomes. The PDTLR not only represents a significant advancement in the identification of PD in teaching and learning needs within higher education but also offers a valuable resource for institutions aiming to foster a culture of continuous improvement and professional growth among their faculty.

2. Literature Review

The need for a validated instrument to recommend PD in teaching and learning for higher education instructors is underscored by the growing recognition of the complexities involved in quality teaching practices. Research has suggested that PD in teaching and learning is crucial for enhancing teaching quality and improving student outcomes [5]. However, many existing PD programmes lack a systematic approach to identifying the specific needs of instructors, which might lead to ineffective training and wasted resources. This literature review synthesises current research on higher education regarding PD in teaching and learning needs, evaluates the psychometric properties of existing questionnaires, and proposes salient categories for PD in teaching and learning.

A significant body of literature highlights the importance of understanding instructors’ perspectives on their PD needs. For instance, ref. [6] suggested that teachers’ perceptions of their PD needs directly influence the effectiveness of PD programmes. This aligns with findings from [7], who proffered the need for targeted and specialised PD initiatives for adult educators. These studies collectively suggest that a more nuanced understanding of instructors’ needs is essential for designing effective PD programmes.

Despite the recognition of the importance of PD, many existing approaches for identifying instructors’ PD in teaching and learning needs on a broad-scale basis, such as the use of questionnaires, present inadequacies. These inadequacies are in part compounded by the absence of psychometric rigour (e.g., Rasch analysis) in their development. Ref. [8] highlighted the importance of understanding teachers’ perceptions of their PD needs, yet multiple studies rely on self-reported data without rigorous validation processes. Similarly, in conducting a cross-cultural study on home learning environments utilising modern test theory (Rasch analysis) to assess differential item functioning, ref. [9] found that many existing tools do not incorporate similar comprehensive validation processes. Ref. [10] further illustrated this point by validating an ICT assessment tool through the Rasch model, demonstrating the importance of robust methodologies in ensuring the reliability and validity of educational assessments.

This inadequacy is critical, as the effectiveness of PD is contingent upon accurately identifying instructors’ needs and, hence, measuring the impact of interventions; any reliance on inadequately validated or unvalidated instruments can lead to biased results and hinder the development of effective PD strategies. This suggests that there is a need for a robustly validated instrument that can accurately identify PD in teaching and learning needs of higher education instructors.

2.1. Categories of PD in Teaching and Learning

The literature review undertaken as part of this study identified salient categories of PD that are critical for enhancing teaching and learning in higher education. These categories (i.e., assessment, pedagogies/instructional approaches, delivery modes and communication, and digital learning) present commonalities across various published measures related to identifying PD in teaching and learning, as suggested by [5], reflect the multifaceted nature of teaching and learning in higher education, and underscore the importance of a comprehensive approach to PD.

2.2. Assessment

Assessment is a salient category for PD in teaching and learning, as effective assessment practices are crucial for enhancing student learning outcomes. Research indicates that instructors require PD in formative and summative assessment strategies to better evaluate student performance and provide meaningful feedback [11]. This need is echoed by studies that emphasise the importance of aligning assessment practices with learning objectives to foster student engagement and achievement [12].

2.3. Instructional Approaches

Instructional approaches is another salient category within PD in teaching and learning, as the adoption of innovative instructional approaches is essential for promoting active learning and student engagement. Studies have shown that instructors benefit from PD that focuses on pedagogical strategies, such as collaborative learning and inquiry-based teaching [13].

2.4. Delivery Modes and Communication

Both traditional classroom teaching and the shift towards blended and online learning environments necessitate that instructors develop competencies in various delivery modes and communication strategies. For example, instructors need to be equipped with skills to establish psychological safety in the classroom as it is directly correlated with student engagement and motivation; research has suggested that when students perceive their classroom as a safe space, they are more likely to participate actively in discussions and collaborative activities, thereby enhancing their learning experience [14].

2.5. Digital Learning

The integration of digital technologies in teaching and learning is increasingly important in higher education. Studies have highlighted the need for PD that equips instructors with the skills to effectively utilise digital tools and resources [15]. This is particularly relevant in light of the COVID-19 pandemic, which has accelerated the adoption of online learning and highlighted the disparities in digital access and competence among educators [16]. Ref. [17] further proffered the importance of blended learning in PD, highlighting the need for instructors to understand learners’ needs in a blended environment. These findings underscore the need for PD programmes that not only address traditional pedagogical approaches but also equip instructors with the skills necessary to navigate the digital landscape [18], advocating for targeted training that builds staff capabilities with new technologies.

The call for a validated instrument is further supported by findings from studies examining the factors influencing instructor engagement in PD. For example, ref. [19] found that PD needs differed significantly between less experienced and experienced teachers, indicating that a one-size-fits-all approach was insufficient. To this end, a measure such as the PDTLR would enable higher education institutions to provide targeted PD as instructors develop their self-reflexivity. In the same vein, without a robustly validated measure to identify PD needs in teaching and learning, institutions may struggle to implement effective PD strategies that resonate with their instructors. Evidently, the literature strongly suggests the need for a validated measure that can effectively identify the PD in teaching and learning needs of higher education instructors. Existing measures (e.g., questionnaires) often lack robust psychometric properties and, hence, fail to appropriately articulate instructors’ perceptions.

3. Methodology

RMT and Rasch analysis have emerged as critical methodologies in the development and validation of questionnaires across various disciplines, including psychology, education, and health sciences. As an example, ref. [20] evaluated the Pregnancy-Related Anxiety Scale–Screener using Rasch analysis and demonstrated how it could provide comprehensive insights into the psychometric properties of assessment tools. This rigorous validation process is essential for ensuring that questionnaires are not only reliable but also valid for the populations they intend to serve. The significance of RMT lies in its ability to transform ordinal data into interval-level measurements, thereby enhancing the precision and interpretability of assessments [2].

One of the primary strengths of Rasch analysis is its capacity to provide detailed insights into the dimensionality and measurement invariance of questionnaires. Moreover, Rasch analysis facilitates the identification of differential item functioning (DIF), which occurs when items perform differently across subgroups. By utilising RMT, researchers can ensure that the items in a questionnaire are equitable and fit-for-purpose for all respondents, thus enhancing score validity. This capability is crucial for developing instruments that are applicable across various demographic groups, as highlighted by the work of [21], who applied Rasch analysis to the WHO Disability Assessment Schedule (WHODAS) 2.0 scale in schizophrenia, demonstrating the model’s effectiveness in addressing DIF.

In addition to enhancing measurement precision, Rasch analysis contributes to the development of shorter, more efficient questionnaires without compromising psychometric integrity. Ref. [22] examined the Connor–Davidson Resilience Scale using Rasch analysis, providing evidence for the model’s ability to support the development of brief yet effective measures. This is particularly relevant in settings where lengthy assessments may deter participant engagement, a setting in which the PDTLR is likely to be administered. The ability to create concise instruments that maintain high levels of reliability and validity is a significant advantage of employing RMT.

To ascertain the psychometric properties of the PDTLR, six areas of the Rasch analysis were investigated via RUMM2030 (RUMM Laboratory Pty Ltd., Perth, Australia) for each salient category: (1) threshold order—this refers to the arrangement of response categories for items, ensuring that higher levels of the latent trait (i.e., endorsability of each salient category in terms of instructor confidence levels) correspond to higher response categories; properly ordered thresholds indicate that respondents’ probabilities of endorsing higher categories increase with their underlying ability [23]; (2) item fit—this assesses how well individual items conform to the expectations of the Rasch model; items that fit well contribute to the validity of the measurement, while misfitting items may indicate issues with the item or the model’s assumptions [23]; (3) response dependence—this highlights the potential correlation between responses to different items, which can violate the assumption of local independence in the Rasch model [2]; (4) dimensionality—this refers to the number of underlying traits that an assessment measures; Rasch analysis assumes unidimensionality, and violations can lead to misleading interpretations of the data [24]; (5) DIF—this occurs when different groups respond differently to an item, despite having the same level of the underlying trait; identifying DIF is essential for ensuring fairness and validity across diverse populations [25]; and (6) reliability and targeting—this evaluates the alignment between item difficulty and respondent ability. Adequate reliability and effective targeting ensure that items are appropriate for the intended population, enhancing the utility of the measure [2].

3.1. Sample and Data Collection

Availability sampling [26] was used for data collection, and participants (i.e., instructors from a state-funded university in Singapore) who agreed to take part in the study were given a cash voucher as a token of appreciation.

Responses of 245 participants, which were authorised by the institutional review board of the authors (approval of exemption number APL-0192-2023-EXP-01), provided the data for this study. Written informed consent was sought from all respondents via the PDTLR, and data was collected between September and December 2024.

Table 1. Demographics of participants.

	n	%
Gender	117	47.8
Female
Male	128	52.2
Years of teaching	129	52.7
Less than five years
Five to less than 10 years	59	24.1
10 or more years	57	23.3
Average instructor ratings from January 2022 to July 2023	24	9.8
Observations with at least one or more ratings missing
Poor	44	18.0
Fair	44	18.0
Good	45	18.4
Very good	44	18.0
Excellent	44	18.0
School participant taught with	31	12.7
Multiple schools
School N	31	12.7
School B	76	31.0
School H	53	21.6
School S	35	14.3
School C	19	7.8

details the participant demographics. The number of participants was considered adequate based on the recommendations of [27].

3.2. Instrumentation

The PDTLR is a 40-item self-report questionnaire developed upon approaches recommended by [28]. These included expert panel input, a literature review, and alignment to the prevailing PD in teaching and learning needs of the university. Based on the review, items were developed and clustered into four salient categories (instructional approaches [12 items], digital learning [10 items], assessment [15 items], and delivery modes and communication [3 items]). For each item, participants are required to select one of six options (1—not confident at all to 6—completely confident) that best describes their level of confidence corresponding to that item; such rating scales are not uncommon (e.g., the Lecturer Self-efficacy Questionnaire by [29] uses a not confident to completely confident 10-point scale).

4. Results and Discussion

This section presents the Rasch analyses output and interpretations of each salient category.

4.1. Instructional Approaches

The initial Rasch analysis of the instructional approaches category found item misfit to the Rasch model, given that the standard deviation of the item fit residual (M = 0.13, SD = 2.25) was not close to the theoretical one [2] and the item–trait interaction χ² statistic (χ² (36, n = 243) = 84.62, p < 0.05) was significant. Nonetheless, there was no evidence of disordered thresholds, response dependence (i.e., all item residual correlations were below the recommended 0.3), and DIF across the subgroups listed in Table 1 (i.e., gender, years of teaching, average instructor ratings, and school). As the two items that participants had least confidence in (i.e., item 13901 designing teaching and learning activities through generative AI (e.g., ChatGPT) using appropriate prompts, and item 13902 identifying ethical and privacy implications of using generative AI in tertiary education) presented misfit to the Rasch model (i.e., their fit residuals were in excess of 4.0), they were deleted and a second Rasch analysis was undertaken for this category.

The second Rasch analysis found that the remaining items presented a fit to the Rasch model based on the item fit residual values (M = 0.02, SD = 1.32) and the non-significant item–trait interaction χ² statistic (χ² (30, n = 242) = 30.89, p = 0.42). The items in this category also presented acceptable unidimensionality, as the t-test approach recommended by [30] indicated that the estimates from the two subscales of this category (i.e., items with negative item residual correlations and items with positive residual correlations) were different for 7.35% of the participants (see Figure 1); 7.35% is a modest deviation from the critical value of 5% [2]. Additionally, reliability and targeting were acceptable as the person separation index (PSI) was 0.93, which indicated good internal consistency and that participants could be reliably differentiated [2].

4.2. Digital Learning

Misfit to the Rasch model was detected in the initial analysis of the digital learning category based on the item fit residual values (M = 0.01, SD = 2.46), though the item–trait interaction χ² statistic (χ² (30, n = 240) = 42.43, p = 0.07) was non-significant. Items with high fit residual values (e.g., item 15201, designing impactful presentation slides for teaching) were deleted; these items were perceived as either too accessible or remotely related to the category of digital learning by the participants. This resulted in five items that presented a good fit to the Rasch model based on the item fit residual values (M = −0.19, SD = 1.17) and the non-significant item–trait interaction χ² statistic (χ² (15, n = 236) = 5.04, p = 0.99). There was also no evidence of disordered thresholds and response dependence, and the t-test indicated that this category was unidimensional, as the estimates from the two subscales of this category were different for 5.71% of the participants (see Figure 2). Reliability and targeting were also acceptable, as the PSI was 0.92. DIF was specific to item 15301 (designing technology-enabled activities to help students recall course materials) and was deemed benign [31] as it presented two findings. First, participants with five but less than ten years of teaching experience were more confident in the items related to digital learning (see Figure 3). This is explicable given that instructors with more than a decade of teaching experience may be less susceptible to changing their teaching practices, as suggested by [32], who found that teachers “perceive the cost and time it would take to learn to use digital tools to analyse data with their students as key barriers to adopting new tools” (p. 1180). Participants with less than five years of teaching experience were also less confident in the items related to digital learning, as newer instructors require more PD opportunities to be technically proficient to successfully incorporate technological resources into their courses [33,34]. Second, participants from school S were less confident in the items related to digital learning (see Figure 4). This is in line with expectations, as instructors in school S teach engineering and cognate subjects, and it has been suggested that the transition to digital learning can be challenging in the context of engineering education [35].

4.3. Assessment

The initial analysis of the items within the assessment category did not present evidence of threshold disordering, response dependence and DIF. However, the items presented evidence of non-unidimensionality as the t-test estimates from the two subscales of this category were different for 12.24% of the participants, indicating that the items were perceived as more than one dimension of assessment.

A closer examination of the items within the assessment category suggested that participants perceived assessment as three sub-categories (see Figure 5).

Such a perception is consistent with how university lecturers view assessment. As [36] suggested, lecturers often express the need to embrace assessments as integral components of learning rather than as mere evaluative measures. In this regard, lecturers, like the participants in this study, perceived assessment design and development as critical competencies, as they would pave the way for valid assessment scores and student engagement through feedback communication. Feedback is another critical aspect of assessment that university lecturers grapple with and, hence, find value in, ref. [37] noted that lecturers often struggle to define what effective feedback (e.g., feedforward) entails, and are increasingly recognising the need for clear, actionable feedback that empowers students to take ownership of their learning. In this regard, the participants in this study viewed student engagement and feedback communication as a sub-category within the assessment category, distinct from assessment design and development and the assessment evaluation and data interpretation sub-categories.

Based on these perceptions and iterative Rasch analyses, the items within the assessment category were recoded to reflect these sub-categories (see

Table 2. Recoded assessment items.

Item	Sub-Category
16101 Developing assignments and/or quizzes that inform student learning progress	1
16102 Applying principles of assessment (e.g., validity, reliability, fairness) to develop assignments and/or quizzes	1
16201 Designing a curriculum map that supports adaptive learning	4 a
16301 Writing different types of MCQs for different purposes	1
16302 Interpreting MCQ item data to inform student learning	2 b
16501 Developing scoring rubrics that support consistency in marking	2 b
16502 Developing scoring rubrics that inform student learning	3
16601 Creating an item bank through applying adaptive learning principles	4 a
16701 Giving feedback that students would find useful	3
16702 Responding to student feedback appropriately	3
16901 Applying classroom assessment strategies to assess students’ knowledge and skills	1
16902 Applying classroom assessment strategies to assess students’ attitudes, values and self-awareness	1
26101 Designing fit-for-purpose group-based assignments	1
26102 Facilitating student management of their own group processes	3
26201 Interpreting adaptive learning data to inform teaching and learning	4 a

^a Disregarded as the initial Rasch analysis found item misfit. ^b Rasch analysis was not performed for sub-category 2 as there were only two items that were subsequently disregarded.

).

Rasch analyses of sub-categories 1 and 3 were then performed. There was no evidence of threshold disordering, response dependence and DIF. Fit to the Rasch model was also acceptable based on the item fit residual values (M_{sub-category 1} = −0.29, SD_{sub-category 1} = 0.95; M_{sub-category 3} = 0.33, SD_{sub-category 3} = 1.60), as with the reliability and targeting (PSI_{sub-category 1} = 0.89; PSI_{sub-category 3} = 0.78). The item–trait interaction χ² statistic (χ² (18, n = 240)_{sub-category 1} = 10.32, p = 0.92; χ² (12, n = 229)_{sub-category 3} = 21.31, p = 0.05) for both sub-categories also suggested a fit to the Rasch model. The unidimensionality of both sub-categories was also acceptable given that the estimates from the two subscales of these sub-categories were different for 6.12% and 6.94% of the participants based on the t-test approach (see Figure 6 and Figure 7).

Based on the Rasch analyses, it could be taken that participants in this study were focused on sub-category 1 (i.e., assessment design and development) and sub-category 3 (i.e., student engagement and feedback communication). Due to the lack of items that measured sub-category 2, it may not be concluded that the participants were less keen on assessment evaluation and data interpretation.

4.4. Delivery Modes and Communication

The Rasch analysis found that the items in the delivery modes and communication category fit the model based on the item fit residual values (M = −0.03, SD = 0.46) and the non-significant item–trait interaction χ² statistic (χ² (9, n = 234) = 7.36, p = 0.60). Reliability and targeting were also adequate (PSI = 0.78), as there was a lack of evidence to suggest the presence of disordered thresholds, response dependence and DIF. The t-test approach indicated that this category was unidimensional, as the estimates from the two subscales of this category were different for 4.49% of the participants (see Figure 8).

Exploiting the affordances of RMT allows for the development of shorter questionnaires without compromising psychometric integrity. The final PDLTR (see Appendix A) comprised 28 items (i.e., 10 items on instructional approaches, five items on digital learning, 10 items on assessment, and three items on delivery modes and communication), down from the original 40 items. Given that the PSI across all categories was adequate and that there was no presence of disordered thresholds, it can be concluded that the six-option rating (1—not confident at all to 6—completely confident), of which participants had to select one that best described their level of confidence corresponding to that item, was appropriate. Taken together, the analyses presented four categories of PD in teaching and learning that can be reliably identified through the PDTLR (i.e., instructional approaches, digital learning, assessment (as two sub-categories), and delivery modes and communication). A subtest analysis was further performed in RUMM2030 to ascertain whether the 28 items were influenced by a general factor, as this analysis presents a bifactor equivalent to factor analytic procedures [38]. The items from each of the salient categories were subtested to create four subtests for the analysis, and the result was that the proportion of common variance retained was 0.99. This suggests that it would be feasible to consider all 28 items as manifest variables of a general factor, which could appropriately be labelled as teaching quality confidence given that the four salient categories are related to teaching in higher education.

5. Practical Significance and Limitations

Research has consistently demonstrated that PD is crucial for improving teaching practices and student outcomes. For instance, ref. [39] suggested that validated instruments can significantly enhance instructors’ confidence and awareness of their teaching roles if the instruments identify their PD in teaching and learning needs appropriately and the instructors attend targeted PD. These findings support the PDTLR as an instrument that can be practically deployed on a broad-scale basis to identify PD in teaching and learning needs of higher education instructors; the PDTLR is particularly useful as it identifies needs based on four salient categories relevant to higher education institutions. With the PDTLR, institutions would be better able to optimise resources and implement strategies that promote teaching excellence and student success more effectively. As for an instructor, the PDTLR could provide two valuable sources of information: (1) confidence levels based on each salient category for the purpose of identifying areas of PD in teaching and learning that would be more beneficial, and (2) an overall metric representing self-perceived teaching quality confidence.

Based on the Rasch analysis, validity evidence in terms of content, response processes and internal structure has been established for the PDTLR. However, a limitation of this study is that consequential validity [40] cannot be established until later. It remains to be seen how the PDTLR can influence instructors’ decisions to attend PD in teaching and learning. While the sample size for this study was deemed adequate based on [27] recommendations, obtaining more stable parameter estimates via a larger sample can enhance the generalisability of the results [41]. Further, more items could subsequently be developed and validated for the purpose of establishing assessment sub-category 2 (i.e., assessment evaluation and data interpretation). Similarly, as more PD related to class delivery modes and communication is developed, more items should be included in that category to better establish its internal structure.

In addition, the applicability of the PDTLR across different cultural and institutional contexts warrants further investigation. Institutions intending to implement the instrument are encouraged to undertake local validation procedures, including reviews of contextual relevance, linguistic appropriateness, and alignment with institutional teaching and learning practices. Such efforts would support the continued refinement and evolution of the PDTLR as a context-responsive instrument for identifying PD in teaching and learning needs.

6. Conclusions

The development and validation of the PDTLR underscored the importance of employing modern psychometric methodologies to enhance the identification of PD in teaching and learning needs for higher education instructors. By applying RMT, which is indispensable in the development and validation of questionnaires across various fields due to its ability to provide precise, reliable, and valid measurements for researchers seeking to understand complex constructs, this study addressed some limitations of existing measures and provided a reliable tool for identifying PD in teaching and learning needs. The PDTLR is also a response to [42], who highlighted the importance of evidence-based instructional practices for enhancing student learning outcomes. As institutions continue to navigate the complexities of teaching and learning in a rapidly changing educational landscape, the PDTLR stands poised to play a pivotal role in supporting instructors’ growth and enhancing the overall quality of higher education.