AI-Assisted Career Preparation and Skill Gap Awareness: A Retrospective Pretest-Posttest Study

Abstract

This study explores the effectiveness of an AI-enabled career preparation platform in enhancing undergraduate students’ awareness of their career readiness and skill development. The research was conducted within a localized context at a comprehensive university in Singapore, introduced as part of a career-preparation exercise for internship exploration and selection, allowing students to self-assess their current competencies and identify gaps vis-à-vis industry requirements. Students first evaluate their perceived knowledge of their skills and the deficiencies they need to address. This platform leverages artificial intelligence to help students profile their skills and discover tailored internship opportunities. By uploading their resumes, students receive a personalized skills profile identifying their relevant competencies. The platform then suggests potential career roles and automatically shows skills for development. Using a retrospective pretest-posttest survey with Likert-scale responses, statistical tests revealed significant improvements across all measured areas. The platform was further assessed across two constructs with high internal consistency, reflecting strong user engagement and satisfaction. Lastly, we highlight the potential of AI-driven tools in supporting student career preparedness and offer insights for further platform improvements. The findings from this study are not assumed to generalize directly to other institutional, cultural, or national settings, but instead offer initial context-specific indications of how such tools may support students’ perceived skill awareness and career planning.

1. Introduction

In an increasingly dynamic and competitive global economy, career preparation for university students has become a central concern in both higher education and career development research. The traditional trajectory from education to employment is evolving rapidly, with employers now seeking graduates who possess not only academic knowledge but also a range of practical, digital, and transferable skills (Knight & Yorke, 2003; Yorke, 2006). This shift is driven by structural changes in the global workforce, including automation, digital transformation, and the rise of hybrid work models, which have created new forms of skill mismatches and uncertainty for first-time job seekers (OECD, 2023; World Economic Forum, 2025). In this context, universities are increasingly expected to act not only as knowledge providers but also as career development agents, equipping students with the competencies needed to navigate diverse and rapidly changing career paths (Bridgstock, 2009). Effective career preparation, as discussed in vocational psychology and employability literature, involves a multidimensional approach that integrates technical skills, soft skills, and self-regulatory capacities such as adaptability and self-efficacy (Lent et al., 1994; Savickas, 2005).

The World Economic Forum’s Future of Jobs Report (World Economic Forum, 2025) projects that by 2030, 92 million jobs may be displaced, while 170 million new roles could emerge across industries as companies adopt artificial intelligence (AI), robotics, and automation technologies, leading to a net increase of 78 million jobs. This highlights the need for students to develop core competencies in their chosen fields and continually refine their skills to adapt to changing market needs. Consequently, it is increasingly important for universities to consider aligning their curricula and resources with evolving skill demands to support students’ readiness for employment. AI and machine learning (ML) are proving to be enabling tools in career preparation and workforce development (Lai et al., 2024, 2025). AI and ML have the potential to assist both educators and students by tracking real-time changes in job market demands, analyzing emerging trends, and identifying new skills that are gaining prominence across different sectors. For instance, recommender systems have the potential to personalize career guidance by assessing students’ current skill sets and identifying gaps relative to the evolving job market (Lahoud et al., 2022; Qamhieh et al., 2020). This enables students to make informed decisions about their professional development, focusing on the competencies they need to build or enhance.

By integrating AI and ML into key stages of the career preparation process—specifically internship selection, matching, and the discovery of first job opportunities—universities can help students better understand the job market and the paths they could potentially take. When effectively designed, these technologies may serve as personalized career advisors, assisting students to recognize the opportunities available to them and the steps they need to take to reach their goals. As the job market continues to evolve, tools like AI have the potential to make the process of career planning more informed, focused, and responsive to change (Chen et al., 2022; Lewton & Haddad, 2024; Moran & Ackerman, 2023). By incorporating AI-driven tools, universities may support students in taking a more active role in shaping their career trajectories.

This study investigates how undergraduate students’ perceptions of AI-enabled career preparation platforms can enhance their understanding of career readiness by focusing on skills development. We define ‘career readiness’ as a student’s awareness and preparedness to pursue internship or employment opportunities through the identification and development of relevant skills. This definition aligns with dimensions commonly cited in the literature, such as self-assessment, skills acquisition, and alignment with workforce expectations (Savickas, 2005; Yorke, 2006). The platform is introduced as a career-preparation exercise, allowing students to self-assess their current competencies and identify gaps between their existing skill sets and those required in their prospective career paths. Initially, students evaluate their knowledge of the skills they currently possess and their awareness of any deficiencies. Following the exercise, the same assessments are administered to measure shifts in their perceptions regarding their skills and knowledge gaps. In addition, students’ perceptions are evaluated across two key constructs of the platform: the quality of features and overall usability, and its usefulness for career preparation and progression. Although this study was conducted in a localized context, it involved participants from diverse nationalities, and the principles and insights derived may be applicable to a broader international audience.

In the subsequent sections, we perform a literature review in Section 2 on current methods for identifying existing skills in undergraduate students, skills gaps, particularly relating to career aspirations and approaches to skills development. Then, in Section 3, we outline the method used in this work to investigate students’ skill awareness and perception of an internship and skills identification platform that supports their skills development. Using a data-driven approach, we perform analytics on the survey responses in Section 4 to illuminate themes and techniques practitioners can adopt for advancing AI-driven career preparation. Lastly, we discuss future approaches in Section 5.

2. Literature Review

2.1. Skills, Skills Gaps, and Career Readiness

The conceptualization and operationalization of skills and the development of a standardized taxonomy of skills remain complex and ongoing endeavors. This complexity is primarily due to the varying perspectives of different stakeholders involved in defining and measuring skills. These stakeholders include educators, employers, policymakers, and students, who may interpret differently what constitutes a “skill” and how it should be assessed (Lim et al., 2024). Several studies have highlighted these discrepancies in definitions and the challenges that arise when attempting to create a unified framework for understanding skills within various contexts (Garcia-Esteban & Jahnke, 2020; Goulart et al., 2021; Jones et al., 2016; Mainga et al., 2022; McManus & Rook, 2019). This lack of consensus often leads to a gap in understanding the actual skill set of undergraduate students, contributing to what is commonly referred to as the skills gap.

The skills gap generally refers to the mismatch between the skills prospective employees possess and the skills employers require for effective performance in the workforce. This gap is often identified as a key challenge in bridging the transition from higher education to employment (Hora, 2016; Lise & Postel-Vinay, 2020; McGuinness et al., 2018). Central to the skills gap issue is the varying degrees of awareness students have regarding their skills. Many students may have limited insight into their capabilities, which might result in misalignment between their career aspirations, the skills they believe they have, and the demands of the labor market. Consequently, students’ understanding of their skill sets, especially concerning specific career paths, often varies. When students connect their skills, career choices, and professional goals with broader labor market trends, they may better understand the market demand for their competencies.

While a wide range of factors shape students’ transitions into employment, addressing the skills gap represents one actionable area where higher education institutions and employers can better align their efforts to support graduate preparedness. As part of this effort, higher education institutions aim to help students identify their existing competencies and any gaps that may hinder their future employability or workplace performance. By providing students with opportunities to enhance their skills in alignment with market needs, educational institutions can contribute to developing a more employable graduate workforce (Kang, 2023; Nagarajan & Edwards, 2015; Rêgo et al., 2023). One strategy these institutions employ is encouraging students to engage in career preparation activities before graduation, such as internships, job shadowing, and skills identification workshops, like resume writing, which help students build practical, market-relevant competencies. Furthermore, aligning educational offerings with the evolving demands of the job market is seen as critical for improving the employability of graduates in an increasingly competitive global economy.

Employers are another key stakeholder in this process, as they ultimately define the skills deemed essential for prospective employees. Employers identify the skill sets necessary for job performance and play a vital role in shaping the skills landscape by highlighting gaps in the existing talent pool. This is particularly important in the context of rapid global economic changes, including those brought about by digital transformation and the rise of new industries. As businesses adapt to these changes, the demand for certain skill sets shifts accordingly. Employers are thus invested in understanding and aligning their hiring processes with the skills currently in demand and expected to be essential in the near future (Li et al., 2021; Mainga et al., 2022). In this regard, collaboration between educational institutions and employers is critical to ensure that the skills taught in higher education correspond to the needs of the labor market.

2.2. Approaches to Identifying Skills and Gaps

As such, institutions have used various methods to identify students’ skills and skills gaps. Quantitative methods often rely on tools like Likert scales to measure stakeholders’ perceptions of the importance of various skills (Nghia, 2018). However, quantitative approaches may fail to capture the nuanced and technical differences across multiple industry sectors. As such, qualitative methods are frequently employed in studies that aim to uncover deeper insights into industry skills and potential skill gaps. In addition to qualitative methods, mixed-method approaches have been increasingly adopted. These approaches combine qualitative techniques, such as in-depth interviews and focus groups, with quantitative methods, such as closed-ended surveys, to provide a more comprehensive view of students’ skills across various contexts (Abbas & Sagsan, 2019; Akhtar et al., 2024).

More recently, AI and ML techniques have gained significant traction in identifying and analyzing skills. These advanced technologies, particularly Natural Language Processing (NLP), are employed to mine and process large volumes of text data, such as job descriptions, resumes, and online job postings. Through NLP, algorithms can extract relevant skills from these texts and map them to market demands, helping to identify skills gaps more effectively. Furthermore, classification techniques within ML models can be used to categorize and match these skills with specific job roles, allowing for a more nuanced understanding of the evolving needs within the labor market and development of a skills taxonomy (Karimi & Pina, 2021; Siswipraptini et al., 2024).

With the continued digital transformation of the workforce, the demand for new and transformed job roles is increasing rapidly. This shift presents a challenge for traditional methods of skills gap identification, which can be slow to adapt to these changes. ML techniques, by contrast, can analyze vast amounts of real-time data from online job platforms, social media, and recruitment websites. These data sources offer a more dynamic and timely snapshot of the skills currently in demand. By leveraging this data, ML models can more quickly identify emerging skills and analyze trends in job market shifts, addressing the time lag that often limits the effectiveness of traditional skills gap assessments. Additionally, ML models can account for regional variations and sector-specific trends, further enhancing the precision of skills gap analysis in the context of the rapidly evolving global labor market (Braun et al., 2024).

2.3. Career Development Theory

Ginzberg et al.’s career development theory suggests that young individuals, especially those in higher education settings, are often in the critical phase of their career development. This stage is characterized by refining career choices through a series of phases: exploration, crystallization, and specification. Each successive stage narrows an individual’s focus and strengthens their commitment to a particular career path as they gain more insight into their interests, skills, and market demands (Ginzberg et al., 1951; Kim, 2021). In the context of AI-driven recommender systems, the use of technology can facilitate this process by engaging in the different phases of Ginzberg’s theory. Students could engage in personalized exploration based on factors such as personal interests, motivations, beliefs and social support with the platform. Potentially, this could lead to crystallization of their aspirations toward a specific professional trajectory (Jiang et al., 2019).

2.4. Stakeholder Roles in Career Preparation

Contemporary career development literature recognizes that effective career preparation is a shared responsibility involving students, educators, institutional career services, and employers (Hooley et al., 2011). Each stakeholder plays a role in shaping students’ career trajectories through self-directed learning, institutional support, and alignment with labor market needs. The literature commonly identifies universities as key players in fostering career development by providing resources, guidance, and opportunities for students to engage in career-related activities (Nagarajan & Edwards, 2015). However, some studies suggest that institutions may not always adequately prepare students for the demands of the workforce. For instance, university curricula are sometimes criticized for misaligning the skills and attributes employers expect from new graduates. This disconnect may result from overemphasizing academic knowledge rather than practical, work-ready competencies (Goulart et al., 2021). Additionally, a lack of collaboration between educational institutions and other stakeholders, such as employers and professional organizations, may hinder students’ exposure to real-world industry expectations (Kleckner & Butz, 2020; McManus & Rook, 2019; Rêgo et al., 2023).

Effective collaboration between educational institutions and employers is essential for aligning academic learning with workplace needs. Such partnerships help foster a mutual understanding of the skills required by employers and enable the design of educational strategies responsive to the dynamic nature of the job market. This helps ensure alignment between higher education institutions and the workforce for first-time workers and in the context of lifelong learning (Lim et al., 2023a, 2023b). Ultimately, while collaboration between educational institutions and employers is crucial in shaping skill development strategies, it is equally important to empower students to take an active role in their career preparedness. The role of educators, and career services should be to empower students with the information and tools they need to make informed decisions about their existing skills and future development pathways. This fosters an environment where self-determination flourishes, empowering individuals to stay motivated and take charge of their own development.

2.5. Gap and Research Questions

A significant gap exists in the practical application of AI-driven platforms that allow students to self-assess their skills in real time, compare their competencies against evolving industry requirements, and receive dynamic career recommendations based on real-world and real-time data. Existing platforms tend to provide static career guidance without integrating automated skills gap analysis or real-time labor market insights. This study addresses this gap by introducing an exploratory AI-enabled career preparation platform that not only facilitates personalized career exploration but also helps students critically reflect on their skill development through interactive feedback and real-time industry alignment. Investigating how students perceive and engage with such platforms is essential to understanding their potential in enhancing career readiness and bridging the gap between education and employment.

The literature suggests that career readiness involves students’ awareness of required skills, their recognition of personal skill gaps, and their ability to plan for further development. It also indicates that AI-enabled career tools may support reflection, guidance, and decision-making, although evidence remains limited regarding how students perceive and respond to such platforms in practice. In light of this gap, the present study addresses the following research questions: To what extent does participation in an AI-enabled career preparation exercise relate to changes in students’ self-perceived knowledge of industry-relevant skills, awareness of personal skill gaps, and ability to prioritise future skill development?

3. Methodology

This study was at Nanyang Technological University (NTU), Singapore, focusing on students currently seeking internship placements, which typically happens around one and a half to two years before graduation. These students are in the phase of exploration as described by Ginzberg, making them an appropriate group to study the processes involved in getting ready for their professional lives (Osipow, 1968). Since they are gaining practical experience, their views on the skills and abilities needed in the workplace are likely to be fresh and insightful. The timing of their academic progress further justifies their selection as participants in this research. We expect their feedback to differ from students earlier in their university educational journeys. Investigating this group can provide essential insights into the success of current career preparation programs and help pinpoint areas that need improvement to prepare students for their future careers better.

Traditional career preparation programs and platforms typically require applicants to browse internship portals, identify opportunities based on job descriptions, and submit applications. If shortlisted, candidates attend interviews, during which they may receive feedback. However, this approach relies heavily on self-assessment of suitability based on job descriptions and feedback from interviews, which can be challenging for individuals lacking experience in evaluating their own skills against job requirements (Cappelli, 2015).

To address this gap and personalize the process, the university enhanced its internship portal by integrating it with an AI-driven career preparation platform. This exercise aims to provide applicants with tailored insights into their suitability for specific roles and to identify skill gaps. These are actions in line with the first two phases of Ginzberg’s theory. In the initial exploration stage, students begin forming broad career preferences by assessing their perceived competencies and career interests. At this phase, they may not yet have a clear understanding of their strengths or how their skills align with specific professions. The AI-enabled platform facilitates this process by allowing students to self-assess their skills and gaps relative to industry requirements. As students progress to the crystallization stage, they start narrowing down their career choices based on a better understanding of their capabilities and aspirations. The AI-driven platform supports this phase by offering personalized skills profiling and career role suggestions based on the student’s resume and self-assessment data.

3.1. Career-Preparation Exercise

The AI-driven career preparation platform used in this study is a commercially licensed, third-party Software-as-a-Service (SaaS) tool adopted by the institution. As such, the authors do not have access to the proprietary algorithm, training data, or internal logic used to categorize skills, match resumes to competencies, or generate career pathway recommendations. This study does not evaluate the technical performance or accuracy of the underlying AI model. Rather, the focus is on exploring students’ perceptions of their own skill awareness and the perceived usefulness of the platform in supporting their career planning. While the platform includes functionalities such as real-time job data scraping, resume parsing, and skill-role alignment, the technical specifics of these processes lie outside the scope of this research. The tool was institutionally approved for educational use, informed by internal reviews. For prior work assessing the output quality of the platform through expert evaluation, see [redacted]. The app is designed to help students streamline their career development and explore tailored internship opportunities by leveraging their skills and career aspirations through generative AI. Here are the key features of the AI career preparation platform:

• Resume Upload for Skill Profiling: Students upload their resumes, which the platform uses to analyze and map out their skill sets (see Figure 1). This is the primary data source from prospective applicants. This process creates a personalized skills profile that identifies relevant strengths and competencies through AI methods. Students can add additional skills via a search bar or from recommended options.
• Pathway Exploration and Role Addition: Once the platform has created a skills profile, it presents students with possible career pathways to explore. Students also have the flexibility to manually add desired roles from a drop-down bar or choose from suggested roles.
• Dynamic Skill Development Suggestions and Profile: Based on the student’s profile, the platform automatically identifies essential skills for development. Students can access a full overview of their current skills and see which ones they want to develop (see Figure 2). This provides a holistic understanding of where they stand and what areas need further growth.
• Tailored Internship Recommendations: The platform then recommends internship opportunities based on students’ current skill sets and their aspired roles. These recommendations are categorized into buckets that reflect how well the student’s skills align with various roles. A strong fit suggests that most of the student’s skills can be transferred to the role, making it a good match. These skills are extracted by proprietary AI technology that tags job descriptions, provided by employers, with appropriate skills.
• Detailed Internship Insights: Students can click on any internship tile to view comprehensive details about the opportunity. This includes a breakdown of the skills required for the internship and highlights (in green) the skills the student already possesses, allowing them to see their strengths relative to the role requirements clearly (see Figure 3).

The AI career preparation platform offers an interactive, personalized platform that empowers students to map their career goals, develop essential skills, and discover suitable internships. It helps them bridge the gap between their current skill set and future aspirations.

3.2. Survey-Based Data Collection

All participants of this study underwent a short-term study spanning three months, comprising the following activities:

• Pre-exercise survey;
• Career preparation exercise with a customized career portal;
• Post-exercise survey.

The three-month interval between the pre-exercise and post-exercise surveys was to allocate ample time for participants to explore the options provided by the customized career portal fully and also to explore their skills profile. The post-exercise retrospective pretest survey was conducted before students participated in their chosen internship.

The design of the questionnaire was informed by the literature on career readiness, employability, and educational technology adoption. Constructs related to students’ skill awareness and perceived career preparedness draw conceptually from career self-efficacy theory (Lent et al., 1994). Constructs on user experience and platform features were influenced by principles from the Technology Acceptance Model (Davis, 1989).

In particular, a retrospective pretest methodology was employed in the post-exercise survey. Also known as a post-then-pre survey, this survey method required participants to assess their past state (pre-exercise) and their current state (post-exercise) in that order after the exercise. In the retrospective pretest method, respondents reflect on how they felt or perceived their knowledge or abilities before the exercise while evaluating their current status. One of the key advantages of the retrospective pretest design is that it can help reduce response shift bias (Pratt et al., 2000). This type of bias occurs when participants’ understanding of key concepts or their own abilities changes during the exercise. In a traditional pretest-posttest design, participants may not fully understand their skills or knowledge before the exercise begins. As they learn or experience the exercise, their baseline perceptions may shift, making posttest responses challenging to interpret in relation to the pretest (Lam & Bengo, 2003; Little et al., 2019).

Despite its strengths, the retrospective pretest method has several limitations, including the potential for recall bias. Because participants are asked to recall their pre-exercise state retrospectively, their memories may be inaccurate or influenced by their current experiences. Another drawback is the risk of social desirability bias, where participants may exaggerate the effectiveness of the exercise by reporting a more significant improvement. This can lead to distorted responses, where participants either overestimate or underestimate how they felt or performed before the exercise (Howard & Dailey, 1979; Howard et al., 1979). To counter this, we administered a pre-exercise survey containing similar questions on some key indicators to attain a baseline sentiment. Their responses to these question items in the pre-exercise survey will be compared to their responses to the retrospective pretest survey. By comparing both datasets, we can better control response variability and ensure that our findings are grounded in more than participants’ retrospective impressions by reducing distortions when participants try to recall past perceptions.

The post-exercise survey was also designed to collect user perceptions regarding the exercise across two distinct constructs: (1) the features and usability and (2) the perceived usefulness of the career preparation platform. Each construct was assessed using a series of statements rated on a 5-point Likert scale. The survey included 12 items, seven targeting the first construct and five evaluating the second construct.

3.3. Data Analysis

For this study, the primary data analysis will focus on the responses from a Likert scale survey, which consists of ordinal data. Given the ordinal nature of the data, where the distances between response categories cannot be assumed to be equal, it is essential to use non-parametric statistical methods that do not rely on the assumption of normality. Non-parametric tests are well-suited for this type of data as they compare the ranks of responses rather than the raw data, allowing for a more accurate analysis of trends and changes in ordinal responses. This approach ensures that the analysis remains robust, even in cases where the assumptions of parametric tests, such as normal distribution, cannot be met. In this study, we utilize the Wilcoxon test (Wilcoxon, 1945) for the comparative post-exercise retrospective survey and the Mann–Whitney U test (Mann & Whitney, 1947) for the construct study. For each, we measure its significance with the p-value and its effect with Cliff’s delta (Cliff, 1993) and Mann–Whitney effect size, respectively.

Although the study primarily relied on quantitative analysis, we also analyzed participants’ open-ended responses to gain additional insight into their perceptions of the AI-enabled career preparation platform. These responses were reviewed using an exploratory thematic analysis approach, with comments grouped according to recurring patterns and issues raised by participants. Because the qualitative component was intended to supplement and contextualize the survey findings rather than to serve as a standalone qualitative inquiry, coding was conducted by the research team, and no formal inter-rater reliability coefficient was calculated. We also did not seek thematic saturation, as the purpose of this analysis was descriptive and illustrative rather than exhaustive theory development. The resulting themes were used to identify common areas for platform improvement and to provide contextual support for the quantitative results.

4. Results & Discussion

The pre-exercise survey received responses from 154 participants, and we received 109 participant responses from the post-exercise survey. A total of 100 participants provided complete data for pre- and post-exercise surveys and confirmed using the exercise app. Our study has 50 participants from the Science, Technology, Engineering, and Mathematics (STEM) fields and 50 from the Social Sciences, Humanities, and the Arts for People and the Economy (SHAPE) fields. A total of 32 participants provided usable qualitative comments in response to the open-ended questions on platform features and perceived usefulness for career preparation.

4.1. Pre-Exercise vs. Post-Exercise Comparative Responses

In both the pre-exercise and post-exercise surveys, we asked participants three similar questions to investigate the variance in responses due to potential response shift bias and the effect of the exercise. For the pre-exercise survey, participants were asked to indicate their agreement to a set of statements on a 5-point Likert scale (1 = Strongly Disagree to 5 = Strongly Agree) covering:

• Knowledge of skills required for one’s desired industry;
• Knowledge of skills gap to one’s desired industry;
• Plan to prioritize the skills one should develop.

Similarly, for the post-exercise survey, participants were asked to indicate their agreement with the same statements in a post-then-pre format: first, where they felt they had been previously, and then where they were at the time of the survey. These three items were intentionally designed to capture distinct but complementary aspects of career preparation: knowledge of industry-relevant skills, awareness of one’s own skill gaps, and ability to prioritise future skill development. They were not intended to function as a formal unidimensional scale, and no composite score was calculated. Accordingly, each item was analysed and interpreted on its own terms. This design allows for a more fine-grained view of where perceived change occurred, but it also means that the findings should not be interpreted as evidence of change in a single underlying construct of ‘career readiness’ or compared directly to studies using validated multi-item scales. The responses to these questions are compiled in Figure 4.

In the pre-exercise survey, most participants agreed that they understood the skills required for their desired industry. However, in the post-exercise retrospective pretest, many participants reassessed their earlier confidence, with an increase in neutral and disagreeing responses. This response shift indicates that the exercise encouraged participants to critically reflect on their prior self-assessments. As a result of the career preparation exercise, a significant proportion of participants reported agreement concerning their knowledge of required skills, suggesting successful improved understanding. A similar pattern emerged when analyzing participants’ awareness of their skills gaps and when asked if they have a skills development plan. Pre-exercise responses indicated that many participants believed they were aware of their shortcomings and had a plan, respectively. However, the retrospective pretest responses showed a more critical reflection, with some participants indicating that they previously overestimated their understanding of their skills gaps and direction in planning their skills development, respectively.

This comparative study illustrates the importance of educational interventions in enhancing individuals’ self-assessment and planning capabilities. Across all three areas, participants initially demonstrated confidence in their abilities. However, the post-then-pre format revealed that many participants had overestimated their initial understanding, a phenomenon consistent with response shift bias. By encouraging critical reflection, the career preparation exercise corrected these initial over-estimations and provided participants with the tools and knowledge necessary to bridge the gap between their perceived and actual competencies. The improvements in participants’ post-exercise retrospective responses across all three items suggest that the exercise fostered greater perceived clarity about required skills, personal skill gaps, and skill development planning.

4.2. Post-Exercise Retrospective Pretest Responses

In our post-exercise retrospective pretest, we included additional statements to further explore changes in participants’ responses. We apply the Wilcoxon signed-rank test to assess changes to these responses, a non-parametric test used to evaluate paired data. The Wilcoxon test compares repeated measurements on a single sample to assess whether their population mean ranks differ with no required assumptions of normality. A significant result, $p<0.05$, suggests that the medians of the two distributions are significantly different, indicating a change between the two related samples. In addition to the Wilcoxon test, we will calculate Cliff’s delta, a non-parametric measure of effect size, to quantify the magnitude and direction of the observed changes. This approach allowed us to detect significant shifts in responses and understand the practical significance of the observed effects across the sets of statements. The Wilcoxon statistic, W, p-value and $\delta$ are compiled in

Table 1. Wilcoxon statistic W, p-value and associated Cliff’s $\delta$ for participant responses to the post-exercise retrospective pretest survey questions.

Code †	W-Statistic	p-Value	$\mathit{\delta }$
RET.A	$84.5$	<$0.001$	$-0.282$
RET.B	$133.5$	<$0.001$	$-0.355$
RET.C	$205.0$	<$0.001$	$-0.345$
RET.D	$175.0$	<$0.001$	$-0.404$
RET.E	$172.5$	<$0.001$	$-0.366$
RET.F	$614.5$	<$0.001$	$-0.215$
RET.G	$279.5$	<$0.001$	$-0.284$
RET.H	$279.0$	<$0.001$	$-0.340$
RET.I	$179.0$	<$0.001$	$-0.274$
RET.J	$165.5$	<$0.001$	$-0.349$

^† Statements are found in Table A1 of the Appendix A.

.

In analyzing data from a Likert 5-point scale, the null hypothesis, in this case, states that there is no difference in the median responses between each participant’s responses to the same statement, meaning that any observed changes in the Likert scale responses are due to random variation. Consequently, for all statements, we observe that the p-values are of several orders of magnitude less than $0.05$. This low p-value implies the rejection of the null hypothesis; thus, the differences between the responses before and after the exercise are statistically significant. Furthermore, a negative $\delta$ value indicates that the agreement among participants to the statements in the retrospective pretest survey statements after the exercise is generally higher than where they felt they were before the exercise.

The varying effect sizes provide insight into which aspects of the exercise were most effective and highlight opportunities for further enhancement to maximize its impact. While the magnitude of effect sizes generally falls in the small to medium range ($0.215$ to $0.404$), the changes are still practically significant, especially in areas such as understanding career options and prioritizing skill development. The exercise appears to have been particularly effective in enhancing participants’ knowledge of their skills and career paths, as well as their confidence in navigating their career development and future transitions. These findings suggest that such exercises successfully empowered participants by providing them with the tools and knowledge they need to plan for their careers, identify skill gaps, and manage their career trajectories with greater confidence and clarity. Smaller effect sizes in developing action plans for courses and knowledge of resources indicate potential areas where the exercise could be enhanced to provide more support.

4.3. Construct Perception Study

The Mann–Whitney U test was performed to assess the effect of the exercise and whether user perceptions for each construct were significantly different from a neutral value (the midpoint of 3 on the Likert scale). The null hypothesis assumes no significant difference exists between the observed distribution of responses and the neutral value. In contrast, the alternative hypothesis posited that the respondents’ perceptions significantly differ from neutrality. The Mann–Whitney U test was further performed on the combined set of responses within each construct. Similarly, all statistical tests were conducted at a 95% confidence, with significance determined using a p-value threshold of $0.05$.

Additionally, the reliability of the survey items within each construct was evaluated using the ordinal alpha, $\alpha$, a measure of internal consistency. This variant of Cronbach’s alpha is designed specifically for ordinal data (Cho, 2016; Cronbach, 1951; Gadermann et al., 2023). It adjusts for the ordinal nature of the data and provides a more appropriate reliability estimate for non-parametric data. This test was conducted for each construct separately to ensure that the items within the construct consistently measured the same underlying concept. The quantitative results are summarized in

Table 2. Descriptive statistics, Mann–Whitney U statistic, p-value, and effect size r for participant responses to each post-exercise construct questions.

Code ‡	Median	Mean	Std Dev	U-Statistic	p-Value	r
FEA.A	4	3.56	0.891	7500	<$0.001$	0.432
FEA.B	4	3.70	0.847	8050	<$0.001$	0.527
FEA.C	4	3.53	0.937	7350	<$0.001$	0.406
FEA.D	4	3.70	0.948	7900	<$0.001$	0.501
FEA.E	4	3.75	0.744	8300	<$0.001$	0.570
FEA.F	4	3.39	0.920	6850	<$0.001$	0.320
FEA.G	4	3.44	0.925	7050	<$0.001$	0.354
CAR.A	4	3.39	0.994	6800	<$0.001$	0.311
CAR.B	4	3.81	0.907	8200	<$0.001$	0.553
CAR.C	4	3.48	1.01	6850	<$0.001$	0.320
CAR.D	4	3.52	1.02	7100	<$0.001$	0.363
CAR.E	4	3.79	0.946	8000	<$0.001$	0.518

^‡ Statements are found in Table A2 of Appendix A.

and

Table 3. Ordinal $\alpha$, Mann–Whitney U statistic, p-value, and effect size r for participant responses to the post-exercise construct questions, grouped by construct.

Construct	$\mathit{\alpha }$	U-Statistic	p-Value	r
FEA: Features and usability	0.860	371,000	<$0.001$	0.445
CAR: Usefulness for career preparation	0.827	184,750	<$0.001$	0.414

.

For Construct 1, which focused on the AI career preparation platform’s usability, participants generally found it to be user-friendly and intuitive, as indicated by mean scores of $3.56$ for FEA.A (User-friendliness) and $3.70$ for FEA.B (Ease of navigation). The low standard deviations ($0.891$ for FEA.A and $0.847$ for FEA.B) show strong agreement among respondents on these usability features. However, items related to perceived trust in its analytical capabilities, specifically, its ability to analyze skills using AI tagging (FEA.F, mean $3.39$) and evaluate job market requirements (FEA.G, mean $3.44$) received slightly lower scores and more moderate effect sizes ($0.320$ and $0.354$, respectively). This suggests some uncertainty or variability in how participants perceived these analytical aspects, though overall responses remained positive. Notably, FEA.E (Overall satisfaction with usability) showed the highest effect size ($0.570$) within this construct, indicating that the participants perceived that the AI career preparation platform and its perceived functionality was easy to use. Similarly, FEA.B (Ease of navigation) had a large effect size ($0.527$), underscoring its intuitive design as a key driver of participant satisfaction.

In Construct 2, which examined the AI career preparation platform’s usefulness for career-related decisions, participants rated several functions highly. They found it particularly valuable for making internship decisions (CAR.B, mean $3.81$, effect size $0.553$) and for recommending it to others (CAR.E, mean $3.79$, effect size $0.518$). These strong ratings, coupled with large effect sizes, suggest that participants regarded it as an effective tool for both personal career advancement and peer recommendations. However, its usefulness in helping participants make course selection decisions (CAR.A) received somewhat lower scores (mean $3.39$, effect size $0.311$). A plausible explanation for this outcome is that the platform was designed to provide internship recommendations rather than specific course recommendations. Consequently, participants were required to independently match the identified skill gaps with appropriate university courses. This additional step likely contributed to the lower scores observed for this item.

The post-exercise survey results offer valuable insights into how participants perceived the AI career preparation platform’s features and usability (Construct 1) as well as its usefulness for career preparation and progression (Construct 2). Across both constructs, participants consistently reported positive experiences, reflected in the median response of 4 and mean scores ranging from $3.39$ to $3.81$. Overall, the highly significant p-values (<$0.001$ for all items) and effect sizes ranging from $0.311$ to $0.570$ highlight that participants made clear distinctions between the survey items, with the majority rating the AI career preparation platform favorably across both usability and career-preparation dimensions.

Table 3 presents the collective results of the Mann–Whitney U test and ordinal $\alpha$ values to assess the exercise’s impact on user perceptions for each construct. For the first construct, an ordinal $\alpha =0.860$ indicates good internal consistency among the items, meaning the questions reliably measured users’ perceptions of the app’s functionality. The p-value (<$0.001$) suggests a statistically significant difference in user ratings from a neutral point, showing that the exercise had a meaningful effect on user perceptions. Additionally, the effect size $r=0.445$ indicates a medium-to-large effect, further emphasizing that participants responded positively to the app’s features and usability. This implies that the app’s user interface, functionality, and overall design significantly influenced the users’ experience.

For the second construct, an ordinal $\alpha =0.827$ similarly demonstrates good reliability, confirming that the items measuring the app’s usefulness in career preparation were consistent. The p-value (<$0.001$) again confirms a significant deviation from neutral responses, suggesting that participants found the app useful for career-related decision-making. The effect size $r=0.414$ indicates a medium effect, showing that the app was perceived as moderately effective in helping users with career preparation and progression. This finding reinforces the notion that the app provided valuable support in assisting participants with decisions related to internships, skills gap analysis, and career recommendations.

Overall, the ordinal $\alpha$ values across both constructs indicate strong internal consistency, affirming that the survey items were appropriate for measuring user perceptions of the AI career preparation platform. The statistically significant p-values across both constructs demonstrate that respondents rated the attributes favorably, with both its features and career-related utility scoring above the neutral value. The effect sizes reveal important distinctions, with the AI career preparation platform’s usability having a slightly larger impact on user experience than its career preparation features. Nevertheless, both constructs were positively received, indicating that the AI career preparation platform is effective in both functionality and career support.

In an effort to elicit feedback for improving on the two constructs, we also gathered qualitative responses for useful features and how it can help with career preparation, in our case for finding relevant internships. A common concern revolves around navigation, particularly regarding the search functionality. One user remarked, “the search function was not very helpful and did not turn up relevant results”, underscoring the need for improvements in this area. There is also a clear desire for more customizable features tailored to users’ desire for internship pathways. For instance, some participants only wanted to be recommended internships within their specific academic domain (e.g., electrical engineering) and not from general domains (e.g., engineering), even though they fit the skills profile. This was suggested by a participant’s suggestion to “allow the feature to select [internships for specific majors], and provide the list of companies that are related to the industry”. In contrast, another user pointed out, “I cannot [gain] access to internships that are not meant to be [sic] for my major. I found this unnecessary because I [may] also wish to apply to many other job descriptions,” emphasizing the need for broader access to opportunities and not be restricted to one’s academic domain.

Performance enhancements were another notable request, with one user stating, “the loading was slow”, suggesting that faster response times could significantly improve user experience. With users also calling for better collaboration among career services in the institution, with one remarking, “It would be great if everyone involved in career stuff at NTU collaborated effectively to put everything in one place”. This suggests that integrating resources, such as specific course recommendations, could streamline users’ experience and access to information. Lastly, the need for more accurate and real-time job or internship listings was highlighted, with one user suggesting, “more accurate job postings”, pointing to the importance of reliable and up-to-date information as some job postings were already filled, but this information was not reflected in a timely manner. These insights collectively illustrate users’ diverse expectations for the app to effectively support their career development and progression.

5. Implications and Future Work

Based on these findings, we propose a set of actionable recommendations aimed at enhancing the platform. These recommendations directly address the key areas of improvement identified in our study, ensuring that the platform can better align with user needs. Furthermore, these insights have broader implications, as they can inform the development of other AI-enabled career preparation and recommendation platforms. Furthermore, there is a third and final phase in Ginzberg’s theory that is yet to be realized in the existing platform. We envisage that at this stage, the platform should automatically suggest skill-building and job opportunities. By linking students with relevant career development resources, the platform acts as a bridge between education and employment, enhancing students’ preparedness for the workforce. Below are the actionable recommendations for future work that can be applied to other career preparation platforms as well.

5.1. Enhancing Career Platform Features

Although the AI career preparation platform is generally rated well among survey participants, future work could focus on improving how information and resources are delivered to users, making it more actionable and relevant to individual needs. Specifically, the following are some improvements that could be implemented:

• By analyzing user behavior and feedback trace data, AI-driven personalization features can offer targeted recommendations that are more likely to influence decision-making.
• Adding features like decision trees, skill gap analysis tools, or interactive questionnaires could make the information more engaging and actionable. These tools could guide users through a structured process of identifying their needs and help them make informed decisions about their career development.
• Providing users with real-time updates on institution-wide academic and co-curricular activities and external courses classified by skill types relevant to their professional growth could further enhance the utility of career preparation platforms.
• Using ML algorithms, career preparation platforms could predict future skill gaps based on industry trends and user behavior, offering proactive recommendations to users on courses or skills they should focus on before these gaps become critical in the job market.

5.2. Improve Data-Driven Insights

Career preparation platforms could also be enhanced by incorporating more sophisticated data analytics features, allowing users to make data-driven decisions based on their past usage and progress. For example, providing users with detailed personalized progress reports based on their activity within the platform could offer valuable insights into their learning journey. These reports could include statistics on skills developed, areas of improvement, and personalized recommendations for future courses or activities. Additionally, using ML, the platform could predict future skill gaps based on industry trends and user behavior, offering proactive recommendations to users on courses or skills they should focus on before these gaps become critical in the job market.

5.3. Longitudinal and Regional Studies and Feedback Loops

On the experimental front, future work should include more longitudinal studies to track user engagement and outcomes over a longer period of time. This would provide insights into how effectively such platforms support career development beyond initial interactions. Specifically, implementing regular feedback loops within the platform to ask users about their evolving needs and experiences could help keep the information such platforms provide aligned with user expectations at various points in their careers. Tracking user engagement over time and correlating it with real-world outcomes (e.g., job placements, promotions, or skill certifications) could provide valuable insights into how effectively career preparation platforms support users’ long-term goals.

5.4. Limitations

An important limitation of this study is that the reported outcomes are based entirely on students’ self-perceptions of their skills, skill gaps, and career preparedness following the exercise. While these self-reports are valuable for understanding how students interpret and experience the platform, they do not provide objective evidence that students’ actual competencies or career-related behaviours improved. We did not validate these perceptions against external indicators such as supervisor assessments, academic coach evaluations, internship performance, or academic outcomes in this study. A larger-scale follow-up analysis drawing on these indicators will be rolled out in stages, but within the scope of this current study, the findings should be interpreted only as evidence.

Another limitation relates to the measurement design used for the three pre-post comparison items. Although the items addressed related aspects of students’ career preparation, they were developed as separate indicators rather than as a validated scale representing a single latent construct. For this reason, the observed changes should be interpreted as item-specific perceptual shifts rather than as a unified change in overall career readiness.

A further limitation concerns the scope of the sample. The study involved 100 students from a single comprehensive university in Singapore, with balanced representation from STEM and SHAPE disciplines, and the findings should therefore be interpreted within this institutional context. Although this setting provided a useful test bed for examining students’ perceptions of the platform, the results may not transfer directly to students in other higher education systems, labour markets, disciplinary mixes, or cultural settings. Students’ interpretations of career readiness, skill gaps, and AI-generated guidance are likely shaped by local institutional practices, internship structures, and national employment conditions. Accordingly, our findings should be understood as context-specific evidence of perceived short-term changes in skill awareness and career preparation, rather than as evidence of broad cross-context effectiveness.

While our study focused on the user experience and perceived impact of the platform, we recognize that its broader adoption may require further consideration of implementation factors across diverse institutional contexts. For example, we do not explore ethical considerations in depth; we acknowledge the relevance of ongoing discussions around data use, accessibility, and inclusion in the context of AI-powered educational technologies.

6. Conclusions

In conclusion, although our study specifically focuses on a particular AI career preparation platform as the exercise, the findings extend beyond the platform itself. They highlight the broader need for such platforms to support students’ skill awareness, development, and preparation for careers in AI. The study underscores key features critical for the effectiveness of these platforms, offering insights into how similar tools can be designed to better meet the needs of students. Thus, while the exercise studied is one example, the lessons learnt apply to developing future platforms to enhance career readiness in AI and other related fields. These results can inform the design and evaluation of similar interventions, but further multi-institutional and cross-cultural research is needed before stronger claims can be made about effectiveness across higher education contexts.

This study highlights the potential impact of an AI-enabled career preparation platform in helping students assess their skills, identify gaps, and enhance their readiness for the job market. After using the platform, our statistical tests showed statistically significant improvements in participants’ self-perceived knowledge, confidence, and career planning abilities, with p-values consistently of several orders of magnitude below 0.05. Although the Cliff’s delta effect sizes ranged from small to medium, indicating moderate changes in participants’ responses, the exercise led to meaningful gains in areas such as career knowledge, action planning, and confidence in career-related research.

Our platform also aims to set itself apart through the two key constructs. A perception study from participants demonstrated strong performance across these constructs with high internal consistency ($\alpha =0.860$ and $0.827$, respectively). Participants showed high engagement and satisfaction levels, focusing on the information provided by the AI career preparation platform and its influence on skills development for career preparation and progression. While the platform effectively supports career development, it can potentially improve its ability to propose follow-up activities, such as course or program recommendations, for users to take action.

The promising results of this study suggest that AI-enabled career preparation platforms have the potential to support students’ skills awareness and perceived career readiness. Future research may build on these findings by incorporating broader item sets or longitudinal measures to assess the durability and behavioral impact of such changes over time. However, one must recognize that such tools are most effective when implemented within a broader institutional support system. In particular, university career counselors and advisors play a critical role in guiding students through the interpretation of platform outputs, contextualizing skill assessments, and helping students translate recommendations into meaningful career actions. Rather than replacing traditional support services, the platform should be viewed as a complementary resource that enhances existing career guidance by offering personalized, data-informed insights.