Feasibility of Script Concordance Test Development: A Qualitative Study of Medical Educators’ Experiences

Abstract

Script Concordance Tests (SCTs) assess clinical reasoning under uncertainty. While construction guidelines exist, the feasibility of collaborative development approaches and educators’ real-time experiences remain underreported. This feasibility study explores how medical educators construct SCTs collaboratively and their perceptions of the process. Four UK-based medical educators developed SCTs for prescribing in older adults during a three-hour workshop involving observation with a think-aloud approach, followed by a post-workshop focus group. Data were analysed using Braun and Clarke’s thematic analysis, Tuckman’s group development informed observation analysis, and an inductive approach for the focus group. Educators created seven vignettes (30 items) in 127 min, averaging 18 min per vignette. Observation revealed small-team role specification (lead, scribe, challenger) and three themes: content development, quality checks, and team dynamics. The team progressed rapidly through Tuckman’s stages, spending most of the time in the performing stage. Focus group analysis revealed four themes: design features, perceived utility, group dynamics and best-practice recommendations. This study demonstrates the feasibility of collaborative SCT development through structured teamwork. Educators perceived SCTs as practical, as valuing effective team dynamics and clear role distribution. Findings can provide practical insights for institutions implementing SCT development, emphasising practice sessions and appropriately sized collaborative teams.

1. Introduction

In medical practice, clinical reasoning and decision-making in uncertain situations are foundation skills [1]. Prescribing medication is a complex task that requires clinical reasoning and decision-making abilities [2]. Reflecting these competencies, existing research suggests that inadequate prescribing skills, such as inappropriate prescribing, irrational prescribing, or other forms of prescribing competency issues, can compromise patient safety [3,4]. Thus, there is a necessity for effective prescribing skills remedies through training and assessment in medical education and continuing professional development.

Every training or intervention aimed at improvement should be paired with appropriate assessment methods. Assessment utility depends on multiple factors including validity, reliability, educational impact, acceptability, feasibility, and cost-effectiveness [5].

Despite this framework, research has prioritised developing training tools over creating tailored assessments for prescribing competencies, particularly in uncertain clinical situations. A systematic review by Kamarudin and Penm (2013) reported various interventions but noted limited information on the validity and reliability of the associated assessment tools [4]. Similarly, Shi and Qin’s recent review (2023) highlighted the need for validated and reliable assessment measures in future studies [6].

Traditional assessment methods often fail to capture these nuanced cognitive processes involved in prescribing. Script Concordance Tests (SCTs) have emerged as a promising tool designed to evaluate reasoning skills under uncertainty [7].

Developed by Bernard Charlin and colleagues, building upon earlier work with Diagnostic Script Questionnaires [7,8] SCTs are validated assessment tools widely used across healthcare education [9,10,11]. A concise outline of the SCT structure is available in Supplementary File S1.

However, guidelines exist [12,13] and systematic reviews [10] provide theoretical frameworks without capturing actual construction processes. While many SCT studies examine validity and learner performance [9,10,11] or provide construction principles [7,12,13]; the practical experiences of educators during SCT development, i.e., their real-time decisions, collaborative negotiations, quality considerations, and perceptions, remain undocumented. This created three knowledge gaps: First, process transparency: no studies capture how educators actually translate guidelines into practice, including time requirements, decision points, and problem-solving strategies.

Second, collaborative evidence: guidelines recommend single or paired construction [12,13]. However, there is no evidence to support team-based approaches that institutions may need to adopt.

Third, practitioner insights: many studies validate SCT effectiveness [9,11], educators’ experiences, challenges, and lessons learned during construction remain absent from the literature.

Prior to allocating resources for widespread SCT implementation, evidence is required regarding the practical feasibility of collaborative development approaches. Feasibility studies evaluate whether an intervention or process can be successfully delivered with standard institutional limitations, assessing factors such as time, resource requirements, stakeholders’ acceptability, and preliminary evidence of utility [14]. This feasibility evidence is essential for informing decisions regarding their implementation on a large scale and identifying potential barriers or facilitators to adoption in educational settings.

This two-part qualitative feasibility study explores how one UK team of medical educators (developing an SCT for subsequent use in a randomised controlled trial RCT with final-year medical students) constructed SCTs for prescribing and experienced that process. Specifically, the study examined whether collaborative SCT development is feasible, acceptable, and efficient. We observed a live in-person SCT-building workshop using a think-aloud approach; followed by a post-workshop focus group. The study aims to provide practical design insights into team dynamics and design processes that can inform effective collaborative SCT development for assessing clinical reasoning under uncertainty.

2. Methods

This study employed a two-part qualitative feasibility design using methodological triangulation, consisting of an observational phase followed by a focus group interview. The primary objective was to observe real-time SCT construction processes, while the secondary objectives included exploring educators’ perceptions of the process, perceptions about SCTs and providing best practices for SCT development based on their experiences.

Reporting follows the Consolidated Criteria for Reporting Qualitative Research (COREQ) guideline [15].

2.1. Methodological Triangulation

This study employed methodological triangulation, systematically integrating three data sources: observational data (real-time behaviours), think-aloud protocols (cognitive processes), and focus group discussions (reflective perceptions). Following individual analysis, findings were compared using thematic mapping matrices. Convergent findings strengthened confidence, while divergent perspectives revealed process complexity. This approach captured the ‘what,’ ‘how,’ and ‘why’ of collaborative SCT development.

2.2. Participants and Sampling

Purposive sampling was used, as described by Palinkas et al. [16]. The inclusion criteria were: a minimum of 5 years of experience in both medical education and clinical experience relevant to medication prescribing, and previous experience in creating various types of assessment content both competencies were essential for developing educationally sound SCTs focused on prescribing decisions. Eligible educators and doctors from UK higher education and healthcare institutions were identified through internal networks (S.K.P., A.R.C.) as well as through professional recommendations (including limited snowball sampling).

Six individuals from the two universities and hospitals were invited by email; two withdrew due to scheduling conflicts, and four completed both parts. The four participants were all affiliated with the University of Birmingham, which resulted in pre-existing professional relationships among participants. As in a feasibility study, sample size was determined by the need to test whether a process can be implemented, rather than to achieve thematic saturation [14]. The final sample size of four was determined by practical feasibility constraints (two withdrawals due to scheduling conflicts) rather than priori sample size calculation. However, four participants forming a single collaborative team proved appropriate for examining the feasibility of team-based SCT development, which allows for intensive observation of the collaborative process, time requirements and assessment of acceptability. According to Vasileiou et al. [17] sample size adequacy should be evaluated according to study-specific factors such as research aims, scope and depth of data collected. The feasibility study was designed for intensive process observation rather than achieving thematic saturation or making population inference, prioritising depth of observation within practical constraints.

Professional background, experience, and prior SCT exposure are summarised in Supplementary File S2 Table S1. Participants’ sex, gender, race and ethnicity were not collected as they are not considered relevant to the study’s focus on SCT construction and medical educators’ perceptions of SCT and its development. Participants were purposefully selected from medical and pharmacy disciplines to reflect the interprofessional nature of prescribing practice. This diverse professional representation ensured that SCT development incorporated perspectives from key stakeholders involved in medication management decisions.

2.3. Part 1: The Observational Phase

2.3.1. Aim

To capture a deeper understanding of participants’ experiences and thought processes while formulating SCT, following an in-person workshop introducing core SCT principles and construction guidelines.

2.3.2. Data Collection

• Location and setting: In-person, a three-hour observational session was conducted at the University of Birmingham’s (UoB) medical school in a seminar room that had movable furniture arranged to facilitate participants’ interactions, with only the participants and the researcher present in that room.
• Context: The SCTs were developed for subsequent use in an RCT with final-year medical students following an eLearning module; no trial data are reported here.
• The workshop: The workshop was designed to facilitate the understanding of SCT principles essential for creating good test items. R.M.A. designed and facilitated the workshop following the guidelines described by Fournier, Demeester [12] and Lubarsky, Dory [13]

Structure: 30 min orientation, then around 120 min item writing (main-block), 10 min wrap-up. See Supplementary File S3 Table S2 for a phase-by-phase breakdown (purpose, activities, outputs, roles, and materials).

During the session, participants focused on developing SCTs related to older adult prescribing. They were already familiar with the intended learning outcomes for the module, which were also included among the workshop materials. Participants were tasked to create three vignettes with three items each within a three-hour time frame. Materials included SCT examples, a quality matrix, and templates. Observations were recorded, detailing the process and time spent on each SCT component.

• Think-Aloud Protocols as described by Ericsson [18], were utilised to understand the cognitive processes behind expert performance, where participants vocalised their thoughts during task execution. This method was chosen for its ability to provide real-time insights and facilitate observation of interactions, in contrast with retrospective protocols, where thoughts are reported after reviewing a performance recording. The researcher (R.M.A.) adopted an observer role, remaining seated to the side, seeking to observe without interacting or influencing the setting, taking field notes. The notes are to document the physical setting, participants’ interactions, communication patterns, team workflow, and group dynamics, complementing the audio recording.

2.3.3. Data Analysis

Data Preparation

The audio recording from the observation section was transcribed verbatim by R.M.A. and uploaded to NVivo 12 Plus (QSR International Pty Ltd., Burlington, MA, USA, 2018) for efficient, organised data coding and analysis. Field notes were documented by R.M.A., including the setting, participants’ interactions, and the workflow, which were integrated with the transcript data during analysis. The identities of participants were anonymised (PE1–PE4) in the transcripts to maintain confidentiality.

Thematic Analysis Approach

Analysis was performed by following Braun and Clarke’s thematic analysis framework [19,20], utilising deductive and inductive approaches to identify patterns within the observation data. Tuckman’s framework [21] was applied deductively, while additional themes related to SCTs construction process emerged inductively from the data.

Phase 1 data familiarisation: R.M.A. read the observation transcript multiple times to familiarise with the data. During the initial phase, preliminary notes were taken regarding recurring ideas, interactions, behaviours, vocalised thoughts and potential patterns relevant to the research focus.

Phase 2 initial coding: R.M.A. conducted initial coding on the complete dataset. S.K.P. independently coded a subset of 20% of the transcripts to enhance credibility; disagreements were resolved through discussion and codebook refinement. The codebook was updated systematically across all analytical phases, with coding decisions, disagreements, and resolutions documented in a decision log serving as our audit trail. This approach ensured framework evolution while maintaining consistency and transparency [22].

Tuckman’s stages of group development—forming, storming, norming, and performing—[21] provided a framework and was applied deductively to observational data. Additional inductive coding was conducted to capture emergent themes related to SCT development processes not accounted for Tuckman’s framework.

See

Table 1. Tuckman Model Stages and Characteristics.

Stage	Characteristics
Forming	Initial stage where group members get acquainted and establish ground rules.
Storming	Members experience conflicts and competition as they assert their opinions.
Norming	Group begins to establish norms and cohesion, resolving conflicts.
Performing	Group reaches optimal functioning, focusing and working effectively towards goals.

For a brief description of Tuckman’s Four stages of group development.

Phase 3 theme development: codes were collected and examined for patterns that aligned with Tuckman’s stages, as well as themes emerging inductively from the data related to SCTs item creation process and workflow within the team. All relevant codes extracted were then gathered within each optional theme, and relationships between themes were mapped to understand group dynamics and workflow throughout the observation period.

Phase 4 theme refinement: Each candidate theme was reviewed against the coded extracts and the entire dataset to ensure internal coherence and a clear distinction between them. Themes were assessed for their fit within Tuckman’s framework and their relevance to the research questions.

Phase 5 theme definition and naming: Each theme was refined through detailed analysis to identify its essence and scope. Clear definitions articulated what each theme captured regarding observed group behaviours, interaction and SCT development strategies, and how it related to the research objectives. Themes and subthemes were assigned descriptive names that conveyed their central concepts.

Phase 6 report production: the final analysis integrated analytical narrative with illustrative quotes from the observation transcript to demonstrate each theme, providing a coherent account of both group development patterns and the process involved in SCT construction.

2.4. Part 2: Post-Workshop Focus Group Interview

2.4.1. Aim

To gather a deeper insight from the perspectives of the same medical educators on the feasibility, acceptability, and experience of creating SCTs collaboratively and their experiences of the development process.

2.4.2. Data Collection

• Location and setting: The focus group was scheduled shortly after the workshop to ensure participants’ recollections and insights remained fresh and accurate, with only the participants and the researcher present during the focus group.
• Interview Guide: (R.M.A.) facilitated the focus group, adhering to the guidelines outlined by Jordan, Clarke [23] for conducting interviews in medical education research; a semi-structured interview format was employed, covering topics such as participants’ experiences, SCT comparisons with other assessments, best practices, and recommendations for future SCT development. The interview guide was reviewed by an expert and underwent refinement for content validity and clarity [24].

2.4.3. Data Analysis

Data Preparation

The 55 min session was recorded and transcribed by R.M.A. verbatim into a Microsoft Word document, which was then imported to NVivo 12 Plus. Participants reviewed the transcript and confirmed the accuracy, and comments were reviewed and clarified. Participants’ identities were anonymised, consistent with the observational transcript.

Thematic Analysis Approach

Data were analysed following Braun and Clarke’s thematic analysis [19,20], as recommended by Maguire and Delahunt [22]. This approach involves an inductive approach to identify common themes emerging from participants’ perspectives and experiences. Notably, specific themes was anticipated to align with Van Der Vleuten Formula [5]. Moreover, themes related to best practices and construction methods are expected to be identified.

Phase 1 data familiarisation: R.M.A. read the focus group transcript multiple times to familiarise with the data. Having conducted and transcribed the interview, the R.M.A. was already familiar with the data. To ensure accuracy, the transcript was compared against the original recording multiple times, with R.M.A. re-listening while reading on several occasions to verify transcription fidelity.

During the initial phase, preliminary notes were taken regarding recurring ideas, participants’ reflections on the development process and their impression about SCT [20].

Phase 2 initial coding: R.M.A. conducted initial coding of the complete transcript, systematically identifying features of the data relevant to the research aims. The code captured participants’ perspectives on SCT characteristics, designs, and their experiences in construction. S.K.P. independently coded a subset of 20% of the focus group transcripts to enhance credibility; disagreements were resolved through discussion and iterative codebook refinement [22]. Consistent with the observational analysis, coding decisions, disagreements, and resolutions were documented in our decision log audit trail. The approach ensured consistency between data sources while allowing appropriate framework evolution.

Phase 3 theme development: codes were collected and examined for patterns that could form potential themes. Related codes were grouped, and all relevant coded extracts were gathered within each provisional theme. Particular considerations for SCT development and group collaboration.

Phase 4 theme refinement: Each candidate theme was reviewed against the coded extracts and the entire dataset to ensure internal coherence and a clear distinction between them. Themes were assessed for their relevance to the research objectives concerning feasibility, acceptability and stakeholder experiences. Themes lacking sufficient data support were merged with related themes or discarded.

Phase 5 theme definition and naming: Each theme was refined through detailed analysis to identify its essence and scope. Clear definitions articulated what each theme captured regarding participants’ perceptions and experiences of SCT development. Themes and subthemes were assigned descriptive names that conveyed their central concepts.

Phase 6 report production: the final analysis integrated analytical narrative with illustrative quotes from the focus group transcript to demonstrate each theme, providing a coherent account of medical educators’ perspectives on the perceived feasibility, acceptability, and practical considerations of SCT development.

2.5. Researcher Positioning, Reflexivity and Trustworthiness

The lead author (R.M.A.), a PhD researcher, facilitated the workshop, conducted non-participant observation, and carried out the interviews. The researcher’s dual roles as both facilitator and observer created potential for influencing group dynamics and participants responses. To minimise observer effects, the researcher maintained a peripheral seating position during development activities and avoided directive interventions. Participants were explicitly informed that the researcher’s role was observational rather than evaluative. We acknowledge that complete elimination of observer influence is impossible, and to mitigate potential bias from R.M.A.’s dual role as facilitator and analyst, S.K.P. independently coded 20% of transcripts. Disagreements were resolved through consensus meetings. R.M.A. has completed training in qualitative research methods and holds a certification in Good Clinical Practice. The researcher had a mixed pre-existing relationship with participants: one established relationship, two minimal with professional contact, and one with no prior relationship. Participants in the focus group interview reviewed the transcript excerpts for accuracy; minor clarifications were incorporated.

2.6. Ethics

The study adhered to ethical principles, obtaining written informed consent and ensuring data confidentiality. Participation was voluntary; all participants provided written informed consent for participation, audio recording, and use of anonymised data. The University of Birmingham Ethics Committee approved the study protocol [ERN_21-1350] before data collection began.

3. Results

3.1. Sampling

Six individuals met the inclusion criteria and agreed to participate; two withdrew due to scheduling conflicts. Four completed both the SCT construction and the focus group session. The participants represented diverse disciplines: medicine (n = 1), clinical pharmacy (n = 2), and pharmacy (n = 1). Primary practice settings included university settings (n = 2) and tertiary hospital settings (n = 2). All participants had ≥10 years of clinical practice. Three had ≥ 10 years of education experience; one had 6–9 years. Prior SCT exposure was limited, with one having no previous exposure and three having limited exposure. Pre-existing relationships varied: three were departmental colleagues, and one was a result of a prior collaboration. (Supplementary File S2 Table S1).

3.2. Part 1: Observational Study Findings

3.2.1. Feasibility Outcomes

Task Achievement and Efficiency

Participants produced seven SCT vignettes within a three-hour time frame. Each vignette included three to five items, with a total of 30 items. The total construction time was 127 min, giving an average of 18 min per vignette and four items per vignette.

Table 2. Timing and outputs for each SCT development during the workshop.

SCT Component	No. of Items	Time (min)
Q1	5	21
Q2	5	28
Q3	3	16
Q4	5	19
Q5	5	20
Q6	4	12
Q7	3	11
Average	4	18

Note: Average is rounded to the nearest whole number.

summarises the timings and item counts during the development. A sample of one SCT that was produced by participants is used in Supplementary File S1 SCT Structure Example.

Beyond the development timings,

Table 3. Observed Feasibility Indicators.

Feasibility Indicators	Evidence from the Study	Assessment
Time Efficiency	127 min for 7 vignettes (18 min/vignette average)	Efficient development rate
Resource Requirements	4 participants, 3 h workshop, basic materials	Minimal institutional resource needs
Medical Educators Acceptability	Positive focus group responses, sustained engagement.	Participants acceptance overserved and reported
Workload Manageability	Collaborative approach reduced individual burden, natural role division.	Workload for participants appeared manageable.

summarises key feasibility indicators observed in our study. The evidence suggests that collaborative SCT development demonstrated feasibility across multiple dimensions within our study context. Time efficiency was evidenced through consistent development rates, while participant acceptance was observed through sustained engagement and positive feedback. Workload appeared manageable through collaborative processes, and resource requirements remained minimal.

3.2.2. Content Coverage

Cases addressed common and challenging issues in prescribing for older adults. Topics included delirium, falls, acute kidney injury, urinary tract infections, adverse drug reactions, drug interactions, postural hypotension, hospital discharges, medication reviews, and the assessment of treatment risks versus benefits.

3.2.3. Group Development

The team progressed rapidly through the group development process. Overview of key behaviours, interactions and decision-making observed at each of Tuckman’s stages of group development during SCT construction,

Table 4. Summary of observed group behaviours mapped to Tuckman’s stages.

Stage	Group Action	Dynamics
Forming	Rapid establishment of roles and task orientation.	- Focused immediately on task with minimal introductions. - Quick agreement on roles and responsibilities. - Participant 2 emerges as an informal leader.
Storming	Minor conflicts and collaborative resolution.	- Generally agreeable interactions with occasional disagreements. - Quick resolution of disagreements through open discussion. - Building on each other’s ideas.
Norming	Developed a shared understanding of the task, how to approach it and a collaborative rhythm.	- Mutual reinforcement of ideas and approaches. - Brainstormed and refined SCT questions collaboratively.
Performing	Demonstrated high productivity and adaptability in SCT case development.	- Adapted strategies to challenges in real-time. - Deep engagement with complex clinical scenarios. - Critical evaluation of their own work throughout the process

, highlights how role allocation, conflict management and collaborative problem-solving contribute to team function. Figure 1 represents the group’s trajectory progression through Tuckman’s stages, with the performing stage occupying the largest proportion of the timeline observed, followed by shorter periods of forming and storming.

3.2.4. Key Behavioural Patterns Observed

• Leadership and Roles:

All team members actively participated and played important roles in generating ideas. Roles emerged naturally and without conflict. Certain members assumed prominent responsibilities:

• ‑

  Participant 2 emerged as a content expert and primary idea generator, often leading discussions and providing clinical insights.

  ‑

  Participant 1 facilitated group discussions and served as the scribe, summarising ideas to ensure clarity and focus.

  ‑

  Participant 3 and Participant 4 provided validation and alternative perspectives.

• Communication patterns:

• The team exhibited effective communication strategies, including:

  ‑

  Actively listening to ensure all viewpoints were considered.

  ‑

  Supportive responses to uncertainty

  ‑

  Building on each other’s ideas

  ‑

  Constructively challenging ideas to refine SCT development.

  ‑

  Using humour to maintain a positive atmosphere.

3.2.5. Think-Aloud Analysis

Transcript analysis yielded three key themes related to SCT development and group dynamics:

1.

Content Development: This theme captures the foundational aspects of SCT item creation, which include two subthemes:

A.

Learning/Mastering SCT Format: This refers to checking question format and requirements. Participant 2 responds to the question about case linkages by stating, “Well, it doesn’t matter, because the format is that you’re not going to link them” (PE2). Another way the participants mastered the SCT was by working backwards, writing items starting from the outcomes and ending with writing the case.

B.

Developing Clinically Authentic Cases: included illustrations of incorporating realistic clinical scenarios, discussing the cases based on their clinical experience and mapping these cases to the task of older adult prescribing. Participant 1 reminded the others about the cases’ alignment with the module “With a past medical history of COPD, which is not in the older adult module.” (PE1).

2.

Assessment of Quality and Efficiency: This theme emphasises the refinement of SCT items in which participants, during the development process, aim to ensure high-quality and effective items. This theme is divided into three subthemes.

A.

Iterative Question Drafting Process: This process is marked by a dynamic, rapid generation of draft questions through continuous brainstorming within the team. Participants actively seek input from each other, where they specifically ask for feedback on various components, such as question scoring and continuous re-evaluation of the cases and items in anticipation of the expert panel review. As Participant 3 articulated a concern about unanimous expert responses, noting, “If all the experts answer the same, then it [the items] all goes.” (PE3), suggesting the potential impact of unanimous expert opinions.

B.

Balancing Question Difficulty and Relevance: This involves avoiding simple questions and incorporating nuanced clinical details to ensure clinical judgement is required. Participant 1 rejected an easy question by stating, “That’s too easy.” (PE1) The team discussed the complexities of a vitamin D deficiency scenario, highlighting the importance of understanding the implications of prescribing calciferol for calcium and vitamin D levels, thus preventing direct, predictable questions in a clinical context.

C.

Crafting Challenging Judgement-based Options: This subtheme focuses on achieving a balance of ambiguity to elicit a spread of responses. One recommends “make it a bit more ambiguous, not ambiguous a little less obvious,” (PE3), which is believed to lead to a variety of responses “that’s going to lead to a nice spread across experts.” (PE2). This approach aims to elicit nuanced and subtle judgments, as highlighted by observations such as “It’s quite a nuanced sort of decision” and “That’s very subtle, and it’s a judge.” (PE2)

3.

Team Dynamics

A.

Differing Perspectives: This subtheme captures instances where participants expressed opposing viewpoints, such as question difficulty, relevance, and scoring.

B.

Collaborative Process: This involves participants building on each other’s ideas and combining their clinical expertise to refine SCT items collaboratively. An example of this dynamic is when (PE2) suggests considering prior actions and implies resuming simvastatin, currently on hold due to clarithromycin treatment, (PE1) built on this idea considering re-prescription, which showcases effective teamwork in leveraging collective knowledge.

C.

Insightful Discussions: The team engaged in profound discussions about various topics. For example, they debated students’ understanding of questions, with Participant 1 noting that “Students may find a question obvious” (PE1) while Participant 3 suggested using STOPP/START criteria to clarify medication review questions. Additionally, they aimed to improve scenario relevance by proposing changes such as referencing “recurrent falls” in the case to justify medication reviews.

3.3. Part 2:Focus Group Interview Findings—Stakeholder Perceptions

Four major themes emerged from the focus group: (1) SCT Perceived Utility, (2) Design Features, (3) Group Dynamics, and (4) Best Practice Recommendations. Thematic analysis results are visually represented in Supplementary File S4 Figure S1.

(1)

SCT Perceived Utility

Participants viewed SCTs as feasible and practical for assessing clinical decision-making skills authentically. They highlighted perceived efficiencies in grading and the potential for online delivery and automation in high-stakes contexts. Some also felt that constructing SCT items could be less demanding than writing high-quality MCQs: “In some ways it’s easier than MCQs, because with an MCQ you have to find four distractors, and with these you do not, so I quite liked it personally.” (PE3) These reflections represent participants’ perceptions due to the workshop and insight while developing the SCT rather than outcomes measured in the present study. As one participant noted, “The marking can be really difficult…when you’re doing high stakes exams, you’re marking criteria can mean the difference between pass or fail” (PE1). Anticipated educational impact was also highlighted from participants’ perceptions, particularly for prescribing judgments, with SCTs viewed as prompting deeper engagement with reasoning: “They certainly will have an educational impact as people start to think of it [SCT] more about prescribing judgments.” (PE2).

(2)

Design Features

Participants highlighted two distinguishing design features of SCTs and compared them with their expertise with other assessment forms. First, design flexibility; unlike MCQs, SCT was perceived to simplify the construction and modification of items by eliminating the need for multiple distractors. They perceived that this allows the task to centre around nuanced clinical judgement rather than factual recall. Second, structure and format; while SCTs share some similarities with EMQs (i.e., core stems), participants noted that SCTs were distinguished by their focus on decision-making under uncertainty.

(3)

Group Dynamics

From their experience in the SCT development, participants described effective collaboration characteristics by natural role division of labour (e.g., assigning leaders, scribe and challenger). As one participant observed, “What I noticed about that session…people fell into roles.” (PE3). They also reflected on how team size shaped contribution and dynamics: “I think four is a perfect amount; with more than four you start to get people who do not contribute.” (PE2). The process was iterative, with ideas triggered from other ideas “What happened quite a lot. You say something that sparks? … I was thinking about something else. …, and then you go.” (PE3) and was supported by engaging in constructive debates to refine items. A “challenge culture” within the team, inviting peers to question assumptions and provide different input “, That’s what [multidisciplinary working] is about.” (PE2).

(4)

Best Practice Recommendations

Several best practice suggestions arose after the participants’ new experience. These suggestions included leveraging established tools, such as STOPP-START, building SCT item banks to support reuse and broader applicability, conducting practice sessions for new developers, and maintaining small team sizes (four to six members) to improve efficiency and productivity: “If you had 12 people, split down into two or three groups, then get back together.” (PE2).

4. Discussion

This feasibility study explored four medical educators’ experiences of developing SCTs for prescribing decisions in older adults for final-year medical students. Demonstrating that collaborative SCT development is feasible and acceptable within typical institutional constraints. The study provides insights into the test construction process, their perceptions of SCT utility, comparisons to other assessments, and recommendations for best practices.

4.1. Reflections on the SCT Development Process

Observations and focus groups revealed several critical aspects of the development. The efficiency and collaborative dynamics observed in the workshop suggest that SCTs can be developed in a time-effective manner when supported by a well-coordinated team. The collaborative dynamics seen during the workshop sparked a variety of ideas. Furthermore, the educators acknowledged the collaborative idea-building among themselves.

The iterative process of drafting, discussing both panel and test-taker perspectives, and refining SCT items accordingly facilitated the tailoring to common scenarios encountered in older adults. This approach is consistent with Johnson, Constantinou [25] findings, which emphasise the importance of incorporating examinee perspectives in question development to ensure clarity and relevance in test items.

Additionally, the participants engaged in the practice of creating items prior to the scenarios, a technique Johnson, Constantinou [25] describe as “reverse logic,” a process in which one starts with the intended outcome or answer and works backwards, which aids in crafting more effective test items.

Moreover, the novelty of SCTs necessitates that developers possess a thorough understanding of both the subject matter and the SCT methodology to create items that accurately evaluate clinical reasoning in ambiguous situations. Introducing practice sessions has proven essential in supporting the development of clear and well-aligned SCT components, as evidenced by think-aloud protocols and the focus interview; these sessions have significantly contributed to content development by enhancing understanding of the SCT format and the ability to design challenging judgement-based options and clinically authentic cases. The participants acknowledged this positive impact during the group interview.

The pace of SCT development during the observation period was rapid, which participants acknowledged was faster than anticipated, suggesting that the structured collaborative approaches combined with clear role distribution can enhance the efficiency in SCT construction. These findings support the feasibility of the collaborative development within typical time and resource constraints. The observed momentum gain indicated that the initial familiarisation with the SCT format, though requiring upfront time investment, accelerates subsequent item development once the participants master the assessment methodology.

4.2. Comparison of SCT with Other Assessments

Participants perceived SCTs as providing distinct advantages over MCQs and EMQs for assessing clinical decision-making.

Participants value SCTs for their focus on decision-making under uncertainty rather than factual recall [26,27]. One participant noted that SCTs “assesses something different to multiple choice question” (PE1), underscoring their perceived uniqueness in capturing nuanced aspects of decision-making, which aligns with contemporary demands for comprehensive and authentic assessment frameworks [28].

Participants acknowledged the structural similarities between SCTs and EMQs (both use a central stem or scenario to anchor questions). However, they distinguished SCTs by their emphasis on reasoning in ambiguous situations that expert clinicians recognise may reasonably differ in their approaches, whereas EMQs effectively evaluate pattern recognition and differential diagnosis skills through their thematic structure [29,30].

This position suggests that SCTs are more appropriate in cases where the educational goal is to assess clinical reasoning in ambiguous situations where multiple possible answers exist. This mirrors real-world clinical practices and aligns with recent trends that advocate for authentic assessments that reflect the complexity of clinical decision-making processes [31].

4.3. SCTs’ Utility

According to Van der Vleuten, C.P.M. (1996), the utility of an assessment tool is crucial to ensure that the assessments are not only accurate and reliable but also support learning, are resource-efficient, and acceptable to all stakeholders, thus optimising the educational impact [5].

SCTs have been demonstrated in previous studies to be a valid and reliable tool for assessing clinical reasoning in different fields [32,33]. In this study, participants have reported their perceptions of practicality and educational impact of SCTs, such as the practicality of construction, scoring, and delivery, which affirm their value in assessing clinical reasoning in prescribing.

Practicality in design and grading: SCTs have more straightforward grading criteria than other open-ended assessments, making them more suitable for high-stakes testing. The analysis revealed that participants valued SCTs for the design flexibility and simplicity in terms of the elimination of distractor constructions, which are required in MCQs. It was perceived as enhancing efficiency in item development.

Participants, based on their expertise in constructing different assessments, noted the possibilities of automated scoring and online delivery that could enhance SCT’s feasibility for large-scale assessments. While in this current study, these features were not evaluated, previous research has demonstrated the successful implementation of automated SCT scoring systems [34,35,36,37], suggesting this represents a practical direction for future development to promote broader SCT adoption.

The empirically demonstrated time efficiency of collaborative development, coupled with participants’ perceptions of SCT advantages based on their expertise in various assessment development, supports the potential institutional feasibility of SCT implementation.

Educational impact: The differentiation between SCTs and other assessments, such as MCQs and EMQs, further illustrates SCTs’ unique position in medical education. By demanding an evaluation of clinical scenarios against a backdrop of uncertainty, SCTs mimic the complexities of real-world clinical decision-making, thereby offering a more authentic assessment of clinical reasoning skills. These findings accord with Charlin, Roy [7] citing SCTs’ alignment with uncertain clinical reasoning. Unlike MCQs, SCTs offer a more nuanced assessment of decision-making, focusing on Judgment rather than recall. These findings agree with Lubarsky, Dory [13], who cited that SCTs are more of a knowledge application manifested in a form of decision-making. SCTs were seen as having potential educational impacts, particularly in influencing how prescribers’ approach clinical judgments. This suggests SCTs could play a valuable role in training. Moreover, SCT has proven to exert an educational impact on other fields. It is a reliable and standardised test of genetic counselling trainees’ clinical reasoning skills [38]. In a paediatric clerkship, SCTs were used for formative feedback and performance evaluation, indicating a correlation between SCT scores and other measures [39]. Anaesthesiologists employ SCTs to evaluate residents’ clinical reasoning for airway management, showing they can distinguish between clinical experience levels [40].

4.4. Development Guidelines and Recommendations

Recommendations were provided to optimise SCT development, including using existing resources as templates, building banks of items, having practice sessions, and managing team sizes. These suggestions could make development more efficient and also offer practical guidance for programmes that wish to move from the single four-person feasibility team towards broader implementation.

4.4.1. Using Existing Resources as Templates

Participants recommended leveraging established clinical tools and guidelines as templates to enhance scenario authenticity and clinical relevance, such as STOPP/START criteria (Screening Tool of Older Persons’ Prescriptions/Screening Tool to Alert to Right Treatment) to clarify medication review questions. STOPP/START represents evidence-based criteria for identifying potentially inappropriate prescribing patterns in older adults [41,42], which could inform future SCT development by providing structured frameworks for medication review scenarios. Such existing clinical resources serve as valuable templates for ensuring that SCT scenarios reflect real-world clinical challenges and support the development of clinically authentic assessment content.

4.4.2. Building SCT Bank

Participants recommended developing an SCT bank, as one of them felt that some questions constructed during the workshop could be adapted for use with populations beyond older adults. Establishing such a bank would be a practical step, particularly to support diverse contexts. We consider this a valuable recommendation, provided that the SCTs undergo a rigorous quality assurance process and are validated. This aligns with Aldekhayel, Alselaim [43], who highlighted the critical importance of ensuring the validity of SCT components to establish them as readily available resources for clinical reasoning assessments.

4.4.3. Practice Sessions

Training sections are significant for novice educators. In this study, the training session was conducted as a workshop, providing participants with the opportunity to address misconceptions regarding the SCT construction process. The session was found to be successful, with participants acknowledging its benefits. Similarly, a study utilised a workshop to train nursing educators in SCT design further emphasised the importance of such sessions in developing the necessary skills and understanding [44].

4.4.4. Optimising Team Sizes for SCT Development

The role of team size has been a topic of debate. Fournier, Demeester [12] advocate for smaller teams, suggesting that single-person efforts or two can boost efficiency and inspire creativity, a view also supported by Lubarsky, Dory [13]. However, findings from this study indicate that teams with more than two members can further accelerate the development process and enhance creativity. Focus group reflections and observations revealed that interactions within larger groups effectively balanced collaboration and productivity.

During the focus group, participants speculated that teams of four or up to six would foster productive collaboration. At the same time, they believe that teams larger than six risked reduced efficiency due to non-productive discussions, a concern also witnessed by Fournier, Demeester [12].

However, these recommendations about team size of more than four should be interpreted with caution, as they represent participants’ predictions rather than empirical observation. The data from this study support that a team size of four, with pre-existing professional relationships, worked well for this specific task. Their suggestion of a team of six members might be optimal warrants testing for future studies.

The participants recommended dividing larger groups into subgroups of four to six members, which could later reconvene to collaboratively refine SCT items. These reflections derive from one four-person team and warrant exploration in other settings.

4.4.5. Institutional Implementation Infrastructure

Translating these recommendations into routine practice requires basic institutional infrastructure. First, programmes need access to exemplar SCTs, clearly formatted templates for vignettes and items, and a shared repository for storing, versioning, and curating SCT vignettes, scoring keys, and supporting evidence (for example, an institutional SCT bank). Secondly, local leadership and facilitation capacity are needed to plan and deliver workshops, support development teams, and oversee quality assurance of newly created items. This includes ongoing faculty development to maintain expertise and support new team members as programmes expand. Thirdly, cross-institutional collaboration could enable partner institutions to co-develop and share SCT banks while still undertaking local review to ensure contextual and curricular fit. Together, these elements provide a pragmatic framework for programmes that wish to adopt and scale SCT development beyond the single four-person feasibility team described in this study.

4.5. The Role of Group Dynamics in SCT Development

The success of team collaboration plays a pivotal role in the effectiveness of SCT development. The team’s progression through Tuckman’s stages was evident, though the influence of the team’s pre-existing professional relationships and shared expertise [21].

Pre-existing professional relationships appeared to accelerate the team development. The compressed early forming–storming phases may be explained by reduced initial uncertainty and established trust, enabling participants to focus on task completion rather than relationship building [45,46]. This familiarity allowed for the natural adoption of roles without prolonged negotiation, facilitating a direct focus on the task. Evidence from healthcare teams indicates that increased interprofessional familiarity can improve collaborative performance [45]. Repeated experiences working together predict faster learning [46]. The observed rapid progression through stages may not be representative of newly formed teams without established working relationships. Teams composed of unfamiliar members would likely require extended forming and storming phase, potentially affecting development efficiency. Future development initiatives should consider whether to prioritise existing professional relationships for time efficiency or deliberately construct diverse unfamiliar teams to test broader applicability.

The study observation shows that most activity occurred in the performing stage, which demonstrates that well-coordinated teams with a shared professional background can maximise productive collaboration. However, newly formed teams without pre-existing relationships may require a longer forming and storming period [47] but could achieve effective collaboration through structured facilitation.

Several limitations should be acknowledged. First, while the study observed high SCT production rates, the quality of the SCT items has not yet been comprehensively evaluated. Second, all participants were from the same institution and had pre-existing relationships, which may have influenced group dynamics. This familiarity likely facilitated a shorter Forming and Storming stage in Tuckman’s model, potentially differing from dynamics observed in newly formed teams. Third, the final sample size was determined by practical constraints rather than planned design, as two initially invited participants withdrew due to scheduling conflicts. While the resulting team size of four proved effective for intensive process observation, this represents a feasibility constraint rather than an optimal, simple size determination. Fourth, this feasibility study used a small purposive sample (n = 4) to intensively observe in real-time in a single collaboration workshop as recommended [14], rather than for broader generalisation across multiple settings. While this approach facilitated the detailed capture of the process, it limits transferability to other institutional contexts. Fifth, methodological limitations include potential observer bias from the researcher’s presence during activities and dual facilitator-observer role, which may have influenced participant behaviour. These may affect interpretation of observed behaviours and reported perceptions. Future research should investigate SCT development with more diverse teams and incorporate robust quality assurance measures to evaluate the clarity and effectiveness of SCTs produced after team collaboration, ensuring their overall success.

This study provides evidence of the process feasibility of collaborative SCT development and illuminates participants’ perceptions of utility and acceptability. Our findings represent observed collaborative behaviours and participant-reported experiences, rather than validated performance outcomes or empirically tested effectiveness measures. The feasibility evidence reflects successful task completion within resource constraints and participants’ experience; however, long-term sustainability and effectiveness across diverse institutional contexts require further investigation.

In conclusion, the present study explored how one team, consisting of four medical educators with previous experience in developing various assessments, developed SCTs for prescribing assessments to be utilised in an RCT. This was executed through observation and focus group methods. Main findings suggest that SCTs can be efficiently created when teams have a structured process, allocated roles, and have established professional relationships. Regarding the perceived findings from the participants, they perceived SCTs as practical for the assessment of clinical reasoning under uncertainty and noted the SCTs’ perceived design advantage over MCQs.

However, this study has several limitations: the quality of SCT items was not evaluated, all participants were from a single institution with pre-existing relationships among them, and the perceptions of educational impact represent anticipated rather than measured outcomes.

Best practices recommendations that emerged from the single-team experience, such as conducting practice sessions, team size, leveraging existing clinical tools as templates and developing validated item banks. However, these recommendation needs empirical testing across different contexts and settings. Practical implication: For institutional consideration, this study suggests that collaborative approaches supported by appropriate facilitation and basic infrastructure showed potential for efficient development of SCT items in our study context. Future research should examine the quality of collaborative SCT development and compare outcomes across different team configurations, and compare SCTs produced by collaborative teams versus individual item writers as well as empirically test optimal development processes in various settings.