Integrating Large Language Models with near Real-Time Web Crawling for Enhanced Job Recommendation Systems

Abstract

This study addresses the limitations of traditional job recommendation systems that rely on static datasets, making them less responsive to dynamic job market changes. While existing job platforms address job search with an untransparent logic following their business goals, job seekers may benefit from a solution actively and dynamically crawling and evaluating job offers from a variety of sites according to their objectives. To address this gap, a hybrid system was developed that integrates large language models (LLMs) for semantic analysis with near real-time data acquisition through web crawling. The system extracts and ranks job-specific keywords from user inputs, such as resumes, while dynamically retrieving job listings from online platforms. User evaluations indicated strong performance in keyword extraction and system usability but revealed challenges in web crawler performance, affecting recommendation accuracy. Compared with a state-of-the-art commercial tool, user tests indicate a smaller accuracy of our prototype but a higher functionality satisfaction. Test users highlighted its great potential for further development. The results highlight the benefits of combining LLMs and web crawling while emphasizing the need for improved near real-time data handling to enhance recommendation precision and user satisfaction.

1. Introduction

The rapid digitalization of recruitment processes has transformed the way individuals search for jobs and organizations identify potential candidates. Job recommendation systems play an important role in streamlining this process by matching job seekers with relevant opportunities, leveraging advancements in artificial intelligence (AI) and machine learning (ML). Traditional systems, while effective in using static datasets, often fail to adapt to the dynamic nature of the job market, where postings and requirements can change rapidly. This limitation highlights the need for new systems that integrate near real-time data acquisition with advanced language processing capabilities.

Studies such as those by Pei et al. [1] and Wu et al. [2] have demonstrated the potential of LLMs in enhancing recommendation accuracy and user satisfaction. These models use sophisticated techniques, such as negative feedback generation and behavior graphs, to refine the recommendations. However, their reliance on static datasets limits their responsiveness to fast changes in the job market. On the other hand, web crawling and scraping technologies, as explored by Kumar et al. [3], provide a mechanism for collecting up-to-date job listings from diverse sources. However, these systems lack the semantic and contextual understanding required to generate personalized and adaptive recommendations.

This article proposes a job recommendation system that bridges this gap by integrating near real-time data acquisition through web crawling with LLM-powered semantic analysis. By combining these technologies, the proposed system aims to provide dynamic, personalized, and relevant job recommendations that adapt to changing market trends and user needs. The research explores the feasibility and effectiveness of this hybrid approach, focusing on the system’s ability to address the limitations of traditional models and improve the overall job-seeking experience.

The article is structured as follows: Section 1 outlines the problem statement, research goal, and research questions that underpin this study. Section 2 explores the relevant literature, highlighting advancements in job recommendation systems and identifying research gaps addressed by the proposed solution. Section 3 details the research design and methodology, focusing on the development and evaluation of the hybrid system. Section 4 describes the prototype solution, including its architecture, implementation, configurations, functionalities, and limitations. Section 5 examines the user testing and evaluation process, combining quantitative and qualitative analyses and offering suggestions for improvement. In addition, Section 6 elaborates on the qualitative user testing with acquired knowledge during development. Finally, Section 7 concludes by summarizing the key findings, discussing the implications, and proposing directions for future research. In the next subsections, we first present the problem statement and then discuss our research goal and four research questions.

1.1. Problem Statement

Current job recommendation systems have advanced by using natural language processing (NLP), LLMs and machine learning to improve matching accuracy and personalize user experiences. However, they predominantly rely on static data, limiting their responsiveness to fast job market changes. In addition, commercial job platforms follow a logic untransparent to job seekers and may be influenced by specific business goals.

While systems such as those developed in [1,2] use LLMs to refine recommendations through mechanisms like negative feedback and behavior graphs, they lack integration with near real-time data, leaving recommendations potentially outdated. In contrast, models incorporating web crawling, such as those of [3], capture diverse job listings but lack LLM capabilities for dynamic, continuous analysis. This gap points to the need for a system that combines LLMs with web crawling and scraping, enabling job recommendations that continuously adapt to new listings and market trends.

1.2. Research Goal and Questions

The goal of our study is to analyze the following hypothesis: A recommender system based on the combination of web crawling and LLMs can provide personalized, accurate job recommendations based on user input, comparable to traditional recommendation systems in terms of relevance and efficiency. To investigate this, four research questions (RQs) were formulated:

• What are the key advantages of using LLMs to process job-related data in near real-time recommendations?
• How can web crawling technologies be integrated into job recommendation systems to utilize near real-time data collection from online sources?
• How can the effectiveness of a job recommendation system that uses these technologies be evaluated?
• How does the proposed system compare with traditional job recommendation systems in terms of accuracy and relevance?

By addressing these questions, our study provides a guideline on how recommender systems for jobs based on near real-time data from a wide variety of internet sources can be set up and provides insights into their effectiveness.

2. Literature Review

In the following we summarize the relevant literature to provide a theoretical foundation for the study, evaluate the current advancements, and identify any relevant research gaps. To ensure a comprehensive review, a thorough literature search was conducted utilizing multiple academic databases, including Google Scholar, Elicit, and Semantic Scholar. The search employed a variety of combined keywords relevant to our research focus, specifically “Recommender System,” “Recommendation System,” “Large Language Models,” “Web Crawling,” “Web Scraping,” and “Job.”

2.1. Related Work

In the following, we review various approaches related to job recommendation systems, including negative feedback mechanisms, behavior graphs, and machine learning techniques, that aim to refine job matching processes. By synthesizing findings from notable studies, this section illustrates different approaches to job recommendation strategies and underscores the importance of integrating advanced technologies to improve the overall effectiveness of job search solutions.

2.1.1. Approaches Involving Large Language Models

Pei et al. [1] developed a new method to improve job recommendations on platforms like Indeed. They used Large Language Models (LLMs) to create explicit negative feedback labels, which help solve the common problem of “bad matches” in recommendation systems. Their process involves using human evaluators to validate LLM-generated labels, which are then used to train machine learning models. These models can then filter out irrelevant recommendations in almost real-time. This has been shown to improve user satisfaction, increase application rates and reduce subscription cancelations. The study also found that a fine-tuned GPT-3.5 model offered a great balance of performance and cost, nearly matching GPT-4 but with lower latency. This hybrid system, which combines LLMs with traditional machine learning, provides a scalable way to enhance recommendation relevance, especially where negative feedback is hard to come by.

In contrast to Ref. [1], Zheng et al. [4] introduced a novel generative job recommendation system named GIRL (Generative job Recommendation based on Large language models). This system addresses the limitations of traditional recommendation methods, which often operate as “black box” models lacking explainability and adaptability [4]. The authors employ a three-step training methodology to refine their LLM generator: first, through Supervised Fine-Tuning (SFT) to generate job descriptions (JDs) from the Curriculum Vitae (CV) of job seekers (which may provide better matching possibilities with descriptions of offered jobs); second, by developing a reward model trained on recruiter feedback to assess the degree of matching between CVs and JDs; and third, using a Reinforcement Learning (RL) approach to align the generator outputs with recruiter preferences [4]. The experimental results demonstrate that the GIRL model significantly enhances the job recommendation process, compared to baseline models [4], including an improved explainability of job recommendations and effectively bridging the semantic gap between CVs and JDs, thereby providing a more personalized job-seeking experience [4]. This research presents a paradigm shift in job recommendation systems, suggesting that generative models can play a crucial role in tailoring job recommendations to individual users [4].

Ghosh and Sadaphal [5] compared four job recommendation approaches: a deterministic content-based model, two LLM-based approaches (guided and unguided), and a hybrid model combining both deterministic and LLM techniques. The deterministic model uses structured attributes such as skills and experience but fails to capture qualitative job aspects such as work environment and growth potential due to the limitations of the structured data. In contrast, the LLM-based methods analyze unstructured data directly from resumes and job descriptions, with the guided model operating within set criteria and the unguided model functioning freely to assess suitability based on job content [5]. Finally, the hybrid model first filters jobs using deterministic methods before applying LLM analysis to enhance the recommendation accuracy with qualitative insights. Experiments on both synthetic and real-world data demonstrate that the hybrid approach offers a high recommendation quality while remaining cost-effective and scalable compared to purely LLM-based methods [5].

Wu et al. [2] investigated the potential of LLMs for job recommendation systems by leveraging behavior graphs. The authors propose a novel framework, GLRec, which integrates a prompt constructor designed to encode interaction information from heterogeneous behavior graphs into natural language prompts [2]. This approach enables the LLM for a better matching of job seekers and job postings [2]. Experimental evaluations reveal that GLRec significantly outperforms existing job recommendation models across various scenarios [2].

2.1.2. Approaches Involving Web Crawling

Musale et al. [6] focused on campus recruitment, employing a combination of semantic matching and tree-based knowledge matching methods, which consider various attributes of both students and companies to determine the degree of profile similarity. They also provided a keyword-based web crawling approach. The results indicate that the developed system can significantly reduce recruitment time while improving the precision of job matches [6].

Mankawade et al. [7] incorporated machine learning techniques together with NLP techniques to improve the job matching process by first identifying skills from resumes and then predicting the most suitable professions using a Naïve Bayes classifier. This information is then matched with scraped job listings using cosine similarity.

Kumar et al. [3] adopted a hybrid model that merged content-based filtering with collaborative filtering. The system utilized web crawling and APIs to aggregate job listings from various company websites together with intelligent filtering algorithms to prioritize the most appropriate job matches. The results indicated that the proposed system outperformed existing job recommendation models, demonstrating the effectiveness of using user profile data and automated data collection methods to enhance job recommendations [3].

Another recent study on exploring a job recommender system based on scrapped internet data is given by Yörük [8]. In addition to job recommendations, the suggested solution also considers the automatic personalized generation of cover letters and application emails. During an evaluation, an 89% match precision is reported for the suggested solution. In another study by Gore et al. [9] a solution for automatic job offer extraction from websites together with automated profile matching and application support is suggested. Based on tests with 3 job platforms and average user relevance of 86% for recommendations is reported together with a reduction in time to application to 10 min. In the study by Akhtar et al. [10] a similar solution is suggested which uses in addition ontology information for improving the matching. However, this study does not include an evaluation of the proposed tool.

Some recent NLP-based approaches with neither using LLMs nor web crawling are the following: Gugnani and Misra [11] present a job recommender system aimed at improving the match between candidate resumes and job descriptions by using NLP techniques for extracting explicit and implicit skills from job descriptions. Their approach includes using a Doc2Vec model trained on over a million job descriptions to identify also skills which are not explicitly listed. Bansal et al. [12] employed topic modeling techniques for job recommendations together with Latent Dirichlet Allocation (LDA) showing good results. A recent review of job recommender systems considering the results from 76 publications was provide by Çelik Ertuğrul and Bitirim [13].

2.2. Research Gap

The reviewed literature highlights significant advancements in job recommendation systems, particularly through the integration of LLMs and various machine learning approaches. These studies illustrate the potential of these techniques to make job recommendations more precise and adaptive.

However, a significant research gap remains in exploring how LLMs can be integrated with web crawling technologies to create adaptive, near real-time job recommendation systems. Most current approaches rely on static datasets, limiting the models’ adaptability to shifts in labor market demands and user preferences. These models could be significantly enhanced by incorporating near real-time data retrieval through web crawling and scraping, which would enable them to continuously update recommendations based on the latest job postings and market trends. Some other studies such as [3] leveraged web crawling to aggregate job listings from multiple sources, but they did not integrate LLMs to interpret or analyze this data dynamically, resulting in a missed opportunity to combine near real-time data acquisition with deep language understanding.

Addressing this gap involves developing a system that unifies LLM capabilities with web-based data retrieval, allowing job recommendation systems to adapt quickly to market changes (see Section 1.2). By integrating near real-time data from web crawling with an LLMs’ ability to process and interpret unstructured data, such a system could improve recommendation relevance, update job matches dynamically, and offer more personalized and timely job search solutions.

3. Research Design

In the following, we discuss the research design employed to develop and evaluate an innovative job recommendation system that integrates near real-time web crawling with semantic processing powered by LLMs. While two established methodologies, experimental research and design science research (DSR) [14], are considered, the research ultimately adopts the DSR methodology to address the problem of outdated and irrelevant job recommendations through the creation and evaluation of a functional artifact.

The subsections that follow focus on the application of the DSR methodology, detailing the tools and technologies employed, such as LangChain, FastAPI, PostgreSQL, and Scrapy, and the rationale for their selection. A structured workflow of the recommender system is presented, demonstrating the steps from user input to the generation of personalized job matches. The subsections also discuss the evaluation phase, where user feedback is collected through surveys and usability tests to refine and validate the system’s accuracy in meeting the needs of users.

3.1. Research Strategy

The proposed research aligns with a combination of two established research strategies: experimental research and the DSR methodology. These approaches are selected for comparison to ensure that the research not only addresses the problem of outdated and irrelevant job recommendations but also contributes to the development of an innovative and practical system for near real-time, personalized job matching. The chosen design is structured to directly answer the research questions outlined earlier and substantiate the research hypothesis statement that integrating LLMs with near real-time web crawling technologies can significantly improve the job recommendation process.

The DSR methodology is concerned with the creation and evaluation of artifacts designed to solve practical problems. In the context of this research, the artifact is a job recommendation system that integrates web crawling and LLMs. The goal of DSR is to create and assess an innovative solution to the problem of outdated or irrelevant job recommendations, ensuring that the system addresses the practical needs of job seekers and recruiters [15].

The research process in this methodology follows an iterative pattern of design, evaluation, and refinement. First, the system is designed and developed to combine near real-time web data acquisition through web crawling with semantic job matching capabilities using LLMs. Then, experimental research is used to assess the effectiveness of an intervention, in this case, the integration of LLMs and web crawling technologies to enhance job recommendation systems. This approach allows for a controlled examination of how specific components of the mentioned potential system affect the quality and relevance of job recommendations.

The primary objective of applying an experimental approach is to evaluate the performance of such a hybrid job recommendation system. However, to conduct an experiment, a working prototype of the system is required. As this is not the case for our study, we rather follow the DSR methodology, which focuses both on the creation and evaluation of innovative artifacts to solve practical problems. In the context of this research, the artifact is the hybrid job recommendation system that integrates web crawling with LLMs. The goal is to not only develop a functional prototype but also to rigorously evaluate the system to ensure it addresses the needs of job seekers and recruiters.

3.2. Prototype and Used Technologies

The design and development of the prototype itself follow a typical cyclical sequence of design, evaluation, and refinement. All these stages will be conducted using an agile approach. The artifact of this study incorporates web crawling for near real-time job data collection and LLMs for semantic job matching. The key aspects of the system include:

• Web crawling
• LLM-based keyword extraction
• LLM-based ranking of job postings

For the implementation, development and distribution of the prototype, the following technologies were utilized (with versions current as of beginning of 2025):

LangChain [16]: A powerful framework for developing applications that involve LLMs. It is designed to make working with LLMs more flexible by providing tools for chaining multiple LLMs together, creating agents, managing prompts, and interacting with various data sources. This makes LangChain particularly useful for developing the core functionality of the job recommendation system, where natural language understanding and response generation are essential. This allows the system to process both structured and unstructured data seamlessly and build dynamic workflows tailored to the job recommendation process.

FastAPI [17]: An API framework that provides high performance, ease of use, and support for asynchronous programming, making it ideal for handling high-volume, real-time web requests. This is especially important for a job recommendation system that may need to respond quickly to a variety of user inputs and interface with external services (e.g., the Gemini API, databases, etc.). FastAPI also provides automatic OpenAPI documentation, which facilitates development and ensures that the system is easily scalable and maintainable.

PostgreSQL [18]: Chosen for its robustness, scalability, and support for complex queries. PostgreSQL efficiently handles both structured and semi-structured data, making it ideal for managing the job-related data and the configuration data of the prototype.

Gemini [19]: The Gemini Application Programming Interface (API) was selected because it is a powerful tool for enhancing the NLP capabilities of the system. Gemini’s ability to handle complex NLP tasks, such as extracting meaning from large text corpora, allows the system to refine its job recommendations by processing job descriptions, user profiles, and other relevant textual data. Integrating Gemini’s API helps improve the quality of the system’s language understanding and can be crucial for processing the vast amounts of unstructured data from job listings. During the evaluation Gemini 1.5 was used.

Scrapy [20]: Scrapy (Version 2.13) is a Python (Version 3.13) library for web crawling that was chosen for its efficiency in near real-time data extraction. It enables continuous web scraping from job boards and other platforms, ensuring that the system stays up to date with fresh job listings.

Next.js [21]: Next.js was selected for its seamless integration with React, an open-source front-end JavaScript library for building user interfaces. Next.js can provide a simple, responsive, and interactive user experience, which is essential for the fast delivery of job recommendations.

Docker [22]: This technology was chosen for the containerization of the entire system because it allows for consistent, reproducible environments across different stages of development, testing, and deployment. Using Docker ensures that all the components (LangChain, FastAPI, PostgreSQL, Scrapy, etc.) run in isolated environments with the exact configurations needed, making it easier to deploy and scale the system. This is especially important for managing dependencies, as different parts of the system might require specific versions of libraries or tools. Docker simplifies system updates and scalability, making it easy to handle production-level traffic and integrate with other cloud services if needed.

Python Poetry [23]: This tool was selected for managing the dependencies and packaging of Python components within the system due to its simplicity and efficiency in handling complex dependency trees. Poetry provides a consistent environment for Python projects by locking dependencies to specific versions, ensuring that the system runs reliably across different stages of development.

The workflow of the job recommendation system prototype was designed to integrate user inputs and relevance-based ranking to deliver personalized job opportunities. This system operates in four interconnected stages: input analysis, web crawling, result ranking, and output generation. Each stage contributes an important function to ensure that the recommendations are precise and tailored to the user’s preferences and qualifications.

The process begins with user input, which can be carried out in two ways: text descriptions or a CV upload. In the case of textual input, the user provides a summary of their job preferences, such as desired roles, locations, or specific skills. Alternatively, if the user uploads a CV, the system uses the implemented LLM to extract keywords. These keywords may include job titles, skills, locations, years of experience, and other specific preferences such as “remote work” or “part-time roles.” This stage is particularly valuable for unstructured CV data, where the LLM can identify implicit information, such as skills (e.g., programming) or job roles (e.g., developer).

Once the keywords are extracted, they serve as input parameters for the web crawling component, powered by Scrapy. This stage involves dynamic searches through job listing platforms, such as Jobs.ch or Jobscout24.ch, using the extracted keywords. The crawler will be configured dynamically to navigate these platforms and retrieve job listings matching the user’s preferences. In addition, the scrapping process observes the relevant terms of service of the target website as specified in the robots.txt file. For each job listing, details such as the job title, company name, location, required skills, and job description are extracted.

After collecting relevant job listings, the system transitions to the ranking phase, where the LLM evaluates the relevance of each listing against the user’s preferences. The LLM compares the attributes of the job listings with the user input. Key factors such as skill matches, location alignment, and job title relevance were weighted to determine an overall relevance score. Additionally, the model can incorporate contextual factors, such as prioritizing listings explicitly marked as remote work or matching industry-specific terms. The result is a ranked list of job recommendations, ordered by their degree of alignment with the user’s criteria.

Finally, the system generates a user-readable output, presenting the ranked job recommendations in a comprehensible format. This output may include a simple list with key details such as job titles, company names, locations, and application links, or a more detailed format incorporating reasoning for each ranking. The system ensures that the recommendations are actionable, allowing users to easily navigate to the relevant job postings.

3.3. Evaluation with User Tests

The evaluation focuses on the accuracy of the recommendations and user satisfaction, combining quantitative metrics with qualitative feedback for actionable insights. The following section outlines the rationale behind the chosen evaluation methods, the implementation of usability testing, and the integration of user feedback to compare the prototype’s performance to traditional job recommendation platforms.

Shaikym [24] outlined metrics for assessing recommender systems that employ machine learning algorithms, content-based filtering, and collaborative filtering. Metrics such as precision and recall are well-suited for these hybrid machine learning systems. However, they are less applicable to this prototype, which relies on LLMs for extracting and analyzing job-related data and web crawling for retrieving job postings. The output of the prototype is customized to the user’s unstructured input and requires personal interpretation from the user to be evaluated. Given the differences in the outputs of these systems, the evaluation of the prototype is emphasized on user-centric metrics such as perceived accuracy and satisfaction, rather than algorithmic performance metrics. This ensures that the evaluation captures actionable insights into the system’s usability and its ability to meet user expectations.

The usability testing approach for this prototype is partially inspired by the study conducted by Zins et al. [25], which provides a comprehensive framework for evaluating recommender systems. Zins et al. combined subjective user feedback, collected through structured questionnaires, with objective interaction data to assess system design, functionality, and user satisfaction. This mixed-method approach integrates quantitative metrics with qualitative insights, offering a holistic evaluation of system performance and user experience. Similarly, the usability testing of the job recommendation prototype involves users performing key tasks, such as uploading their CVs and reviewing job recommendations. Participants provided quantitative ratings on aspects such as output accuracy and functionality, alongside qualitative feedback about their experiences. This dual approach identifies the strengths and weaknesses of the prototype, ensuring a good understanding of its usability.

The evaluation framework also aligns with the user-centric evaluation model proposed by Pu et al. [26], which emphasizes the assessment of perceived system qualities, user beliefs, and subjective attitudes. This approach provides a way to understand user experiences in recommender systems, further strengthening the evaluation’s validity.

4. Prototype Solution

In this section, the architecture and implementation of the job recommendation prototype are detailed, highlighting its modular design and key components. Furthermore, this section outlines the configurations for keyword extraction and the web crawler, emphasizing their roles in processing user inputs and retrieving relevant job postings. Additionally, limitations related to the prototype’s user interface (UI), crawler performance, and scope are highlighted.

4.1. Architecture and Implementation

The prototype for the job recommendation system was developed as a Python-based application with a modular architecture comprising a LangChain-powered backend and a frontend built using Next.js. The entire system is containerized using Docker to ensure consistency across the development and testing of environments, which allows each component (backend, frontend, and the database) to run in isolated environments with predefined configurations. Dependency management is handled via Poetry for reproducibility of the application.

The foundation of the prototype is based on an open-source application published on GitHub, which was specifically designed for extracting information from PDF documents using LLMs. This application is made publicly available with the intention of serving as a “jumping point” for developers aiming to build their own LLM-powered solutions. By providing a well-structured framework and key functionalities, such as seamless integration with LangChain and support for LLM connections, the original application offers a robust starting point for customization and further development. Using this foundation, the system was significantly expanded to incorporate the workflow described in Section 3.2, which introduced additional components and features tailored for job recommendations.

The architecture displayed in Figure 1 aims to address the research objective of improving the relevance and adaptability of job recommendations by combining LLMs and near real-time web crawling. The modular design supports the methodology described in Section 3.2.

The backend is responsible for handling core functionalities, including data management, keyword extraction, and the interaction with the LLM. These services are accessible through various endpoints created with FastAPI. A database is implemented to store various potential keyword extraction configurations. These different configurations were only used during the development of the application for fine-tuning. The keyword extraction module uses only one keyword extraction configuration, which is used by the LLM, to process user-provided CVs and descriptions for keyword extraction. The extracted keywords serve as input for the web crawler, which performs near real-time searches by accessing pre-defined job boards and career portals. All outputs scraped by the web crawler are used as inputs for the LLM module, which sends a query request to Gemini 1.5 for the ranking of jobs. The frontend, built with Next.js, serves as a simple client interface, offering the possibility to upload resumes (or other PDF documents) or manually input a description of their career to receive job recommendations. This interface communicates via the mentioned API endpoints with the backend to retrieve responses from the LLM and display them in an organized and simple format.

4.2. Configurations

The subsequent subsections provide an overview of the configurations for the keyword extraction component and the web crawler in the prototype.

4.2.1. Keyword Extraction Configuration

The keyword extraction component of the prototype was configured to operate using a predefined and static configuration stored in the database. As mentioned in Section 4.1, the database itself does not store the extracted keywords or the web crawler results. It is used solely to store and manage the keyword extraction configuration.

This configuration adheres to a specific JavaScript Object Notation (JSON) schema that defines the structure of the expected extracted keywords. The schema includes fields for skills, location, preferences, and experience. The skills field is structured as an array of strings, designed to capture both explicitly mentioned and inferred technical, soft, and domain-specific skills. Additionally, the location field extracts the geographical location mentioned in the CV or career description. The preferences field is an array that lists user-defined preferences, such as job types or work conditions, while the experience field captures relevant career experience, focusing on the last three years of professional history.

To ensure a valuable extraction process, the extraction query for the LLM is configured using a Chain of Thought (CoT) approach. According to [27], this method can yield more accurate and comprehensive results compared with traditional prompting approaches. By explicitly guiding the LLM through intermediate reasoning steps, the CoT approach improves the LLM’s ability to infer contextually relevant details, ensuring a richer and more precise extraction of the required schema elements.

Consequently, to indicate that the configuration does not change dynamically during runtime, the extraction process is described as a “predefined static extractor”. The LLM was provided with a specific instruction to follow the mentioned schema precisely: “Extract using the static schema. The list of skills at the schema property skills extracted from the text should include technical, soft, and domain-specific skills explicitly mentioned, as well as those inferred from the candidate’s career path and experiences from the last 3 years.” This instruction guided the LLM to perform a detailed analysis of the input data and ensure that all relevant attributes were captured in alignment with the predefined schema.

4.2.2. Web Crawler

To find job descriptions for the user, a web crawler capable of extracting and mapping specific information from job postings based on the previously extracted keywords (skills, experiences, preferences) was configured. The crawler is customized to the target data, including the job title, job description, and job location within various HTML elements of the crawled pages. Consequently, different html elements of each job description page are being focused on [28]. The crawler is set to compare the user-specific keywords with job titles, textual descriptions, and location information. Job postings that do not match any user-specific keywords are excluded, while the relevant jobs are saved in a JSON file with the following attributes:

• Job Title (String)
• Description (String)
• Location (String)
• Link (URL)

Scrapy provides configurable global settings that allow us to adjust the number of pages to process. Increasing the number of pages enables the crawler to process a larger dataset, retrieving a broader range of job postings and potentially improving the comprehensiveness of the results. However, this comes with a significant trade-off, as a higher page limit also increases the processing time, leading to slower overall performance and potential delays in delivering results. To strike an optimal balance between these factors, the crawler is configured to process up to 100 pages.

4.3. Functionalities

The primary functionality of the prototype is the recommendation of user-specific job listings based on career-related input. The system processes user-provided data and returns a list of three ranked job opportunities, with each result comprising a job title and a link to the corresponding job description. This number can be easily adjusted based on future adaptations. Both the extraction of relevant information from the input data and the ranking of potential job matches are performed using an LLM. The user interface explicitly displays which LLM is being used. The matching and ranking of job descriptions and user-specific keywords extracted from a CV is carried out with the following query: “I have extracted keywords out of a CV, which I have provided you in the list called Keywords. Additionally, I provided you a list of jobs including its title, location, and a link to its job description from the corresponding companies. Can you rank the jobs according to the keywords and provide me with the top 3 matching jobs that would fit to the CV-extracted keywords? Use the provided links to get more information on the specific job if needed for the ranking. Only respond with a Python list of objects that includes the title and link of the corresponding 3 best matching jobs without any further text. Only respond with that list of ranked results without any other text. The output should have the following structure: … ”. The process leverages a ranking algorithm to identify the top three job matches from a provided list of job titles, locations, and links to detailed job descriptions.

Users can interact with the prototype through a simple and intuitive starting page. The system offers two primary input methods: users can either upload their CV or resume as a PDF document, or they can manually enter a textual description of their career background, preferences, and relevant information. After processing the input, the system generates personalized job recommendations. These results are displayed in a clear and accessible format, allowing users to review the top-ranked job opportunities based on their profile.

4.4. Prototype Limitations

The UI of the prototype is minimalistic, with a simple structure that avoids complex formatting. Advanced design approaches, such as color theory or other detailed visual enhancements, are neglected. This design choice reflects the primary focus of the study, which is centered on evaluating the prototype’s performance, particularly the role of the LLM in supporting the recommendation process. The emphasis is placed on the functionality and effectiveness of the system rather than its visual presentation.

Regarding web crawler limitations, let us mention that the primary issue of the used web crawler is the incomplete and inconsistent mapping of job-related information. In numerous instances, the crawler fails to capture or accurately extract keywords from the website information. This inconsistency stems from the limitations in the design and execution of the implemented mapping logic. Moreover, the web crawler, with its current configuration, is unable to combine extracted keywords for a more specific understanding of what content should be crawled and scraped. For example, generic terms without further context, such as “Project Management”, introduce ambiguity for the crawler. These terms are applicable across multiple industries, including banking and construction. This ambiguity causes the crawler to retrieve job postings from a wide range of domains, many of which are irrelevant to the user’s intended field of interest, thereby reducing the accuracy and relevance of the results.

Another significant limitation is associated with the mapping of the job location data. The crawler frequently fails to extract location data from job postings, likely due to variations in the structure and formatting of the source webpages. While the LLM correctly extracted the location data from the user’s input, errors during the crawler’s mapping process resulted in inaccurate or incomplete location-based categorization. The time constraints of this project precluded further debugging and fine-tuning of the location-mapping process, leaving this aspect unresolved.

In addition to these issues, the speed of the crawler emerges as a major bottleneck. To retrieve relevant results, the crawler must navigate through numerous webpages to locate accurate job postings [28]. This process is time-intensive, and the slow performance of the crawler significantly impacts the overall efficiency of the application. The time required to gather and process the data leads to large delays, making the system impractical for real-time or large-scale usage. These limitations significantly impact the reliability and usability of the web crawler, as well as the overall application.

Consequently, the web crawler used in the prototype is limited to operating within two predefined job boards:

• Jobscout24 (www.jobscout24.ch)
• Jobs.ch (www.jobs.ch)

This restriction is necessary to maintain control over the crawler’s scope and runtime by implementing stopping criteria that limit the duration and breadth of the crawling process. Without such criteria, it would be difficult to manage how much data the crawler processes and how long it continues scraping [28]. This design decision also addresses the risk of the crawler encountering non-compatible webpage structures and mitigates the previously identified issue of slow crawling performance.

As a result, the prototype is currently optimized only for job recommendations relevant to the Swiss job market, which was considered during the evaluation phase. This limitation is solely due to the performance constraints of the web crawler and is not related to the functionality of the LLM. The LLM remains fully capable of processing and ranking data regardless of geographic constraints. Consequently, the scope of the job search can be expanded and refined in future iterations by improving and scaling the crawler to include additional platforms and regions.

5. User Tests and Evaluation

This section provides an analysis of the user tests conducted to assess the job recommendation prototype. It highlights both the strengths and weaknesses of the prototype by examining user feedback, comparing its performance to that of JobScout24.ch, and identifying actionable areas for improvement. The evaluation is divided into several sections, focusing on the structure of the tests, the questions asked, the quantitative results, and the qualitative insights gained during the testing process.

5.1. Structure of the Tests and Limitations

To evaluate and improve the job recommendation application prototype, user tests were conducted with voluntary participants. A total of 13 participants were found for the study. The participants were informed about the procedure of the user tests, the purpose of the study, and related aspects. Data from them was only kept in anonymized form. The tests were carried out remotely using Microsoft Teams. The primary focus of these tests was to evaluate the accuracy and relevance of the job recommendations rather than the speed of the prototype’s output or its user interface. A structured testing approach was followed to ensure consistency and gather comprehensive feedback.

The participants were contacted via WhatsApp, where the test schedules were coordinated and confirmed. Before the tests, participants were asked to share their CVs in advance. At the start of each session, participants were asked about their job interests independently of the information provided in their CVs. This step helped us understand their personal preferences without being influenced by their experience and current skills.

During the test, participants first manually searched for jobs on JobScout24.ch using their stated job interest. This process established a baseline experience of using an existing, well-known platform. Afterward, they used the prototype, which generated job recommendations based on their CVs. This setup enabled a direct comparison between the results from the prototype and those from JobScout24.ch. At the end of each session, participants provided feedback by answering questions about their experience in a User Test Form, and the results were summarized in a Microsoft Excel document.

The evaluation was divided into three main parts: a quantitative assessment of the prototype, a quantitative evaluation of JobScout24.ch, and a qualitative feedback section. This combination ensured that both measurable and subjective feedback was collected for the evaluation of the prototype and its comparison with JobScout24.ch.

In total, 13 user tests were successfully conducted, with each session lasting on average approximately 20 min. While this is a sufficient number of tests to gain qualitative insights, a quantitative evaluation including significance testing requires a larger number of tests, which is suggested for future work. This structured process provided valuable insights into the strengths and weaknesses of the prototype, helping guide future improvements.

Let us mention the following limitations of the evaluation setup resulting in possible threats to validity: There may be a selection bias for the rather small and not randomly selected study participants. In addition, the geographic focus both for study participants and resulting job offers (due to platform selection and proximity consideration for recommendations) to Switzerland may restrict external validity. The two-phase test may involve learning effects for the participants limiting the comparability of results for the two job recommendation solutions. Finally, as we analyze the comprehensive application setup, specific aspects (such as latency discussed further below) may have a strong impact on overall perceived quality, which cannot be isolated from the individual quality aspects.

5.2. User Test Questions

To evaluate the usability and performance of the prototype compared to JobScout24.ch, participants were asked a series of structured questions. These were divided into quantitative and qualitative categories to ensure a comprehensive analysis of both systems. The quantitative questions provided measurable insights into performance, while the qualitative questions offered actionable feedback for improvement. Together, these questions formed the foundation for understanding user expectations, identifying areas for refinement, and guiding the further development of the prototype.

5.2.1. Quantitative Evaluation: Prototype

Participants were asked to rate the prototype’s output on a numerical scale from 1 to 10, addressing the key aspects of its functionality and usability. This helped in objectively assessing the prototype’s capabilities. The following three aspects were evaluated by user questions:

• How accurate was the output? This question evaluated the relevance of the job recommendations generated by the prototype. Participants rated how well the recommended jobs matched their personal job interests. A high score indicated that the prototype successfully identified relevant job listings, while a low score highlighted potential shortcomings in the CV analysis, web crawling, or recommendation.
• How satisfied were you with the functionalities of the prototype? Participants rated their satisfaction with the prototype’s features and usability. This included aspects such as the ease of uploading a CV and the intuitiveness of the interface. The goal was to understand how effectively the prototype met user expectations in terms of functionality and ease of use.
• How satisfied were you with the prototype in general? This question provided a holistic view of the participant’s overall experience with the prototype. It combined their perceptions of recommendation accuracy, usability, and general satisfaction to provide a broader understanding of the prototype’s effectiveness.

5.2.2. Quantitative Evaluation: JobScout24.ch

The participants also evaluated the output of JobScout24.ch, a well-established job recommendation platform, using the same set of questions on a scale from 1 to 10 for benchmarking purposes. This allowed for a direct evaluation of how the prototype compared with an industry standard.

• How accurate was the output? Participants rated the precision of the job recommendations provided by JobScout24.ch. This served as a baseline for comparing the relevance and quality of the recommendations from the prototype, highlighting areas where the prototype either excelled or needed improvement.
• How satisfied were you with the functionalities of JobScout24.ch? This question focused on the usability of JobScout24.ch, including features such as search filters and overall interaction. Participants rated how effectively these functionalities supported their job search experience.
• How satisfied were you with JobScout24.ch in general? This question captured the participants’ overall impressions of JobScout24.ch. By comparing these scores with those of the prototype, it became possible to assess which system provided a better overall user experience.

5.2.3. Qualitative Evaluation

In addition to the numerical ratings, open-ended qualitative questions were included to capture feedback. These questions provided insights into user experiences, preferences, and suggestions:

• What did you like about the prototype? This question focused on identifying the strengths of the prototype. Participants shared what they found most appealing. Positive feedback helped highlight the aspects of the prototype that resonated well with users and should be retained or further developed.
• What did you not like about the prototype? Participants were asked to pinpoint specific issues or shortcomings. Identifying these areas provided a clear direction for improvement.
• What could we improve in the prototype? This question encouraged participants to suggest improvements. These suggestions offer valuable guidance for future iterations of the prototype.
• Did you prefer the output from the prototype or JobScout24.ch more and why? Participants were asked to state their preference between the two systems and justify their choice. This question provided insights into the prototype’s competitive strengths and weaknesses compared to JobScout24.ch. Understanding user preferences helped pinpoint areas where the prototype could better meet or exceed the benchmark set by JobScout24.ch.

5.3. Evaluation Results

This section provides an evaluation of the user test results for the job recommendation prototype, focusing on both quantitative and qualitative feedback. By integrating measurable data and user insights, the analysis aims to identify the prototype’s strengths and weaknesses, compare its performance to JobScout24.ch, and suggest actionable improvements. The findings not only highlight how well the prototype meets user expectations but also shed light on the reasons behind the observed outcomes.

5.3.1. Quantitative Results

The quantitative evaluation was conducted by calculating the rounded average responses to a series of questions posed to users. These questions assessed three key dimensions: output accuracy, functionality satisfaction, and general satisfaction.

Table 1. Quantitative Evaluation of the Prototype and Jobscouts24.ch.

		Prototype			Jobscouts24.ch
	Accuracy	Functionality Satisfaction	General Satisfaction	Accuracy	Functionality Satisfaction	General Satisfaction
Tester 1	4	9	6	7	7	7
Tester 2	5	9	6	7	9	8
Tester 3	6	10	7	8	7	7
Tester 4	2	8	4	8	7	7
Tester 5	2	8	5	9	9	9
Tester 6	2	7	4	7	7	7
Tester 7	4	9	6	8	6	7
Tester 8	1	6	3	5	8	7
Tester 9	2	6	4	6	8	7
Tester 10	2	10	5	9	8	8
Tester 11	5	9	7	7	7	7
Tester 12	6	9	8	8	8	8
Tester 13	2	8	5	9	8	8
Average	3.31	8.31	5.38	7.54	7.62	7.46

illustrates the quantitative evaluation of the prototype, showing the ratings based on user feedback together with the evaluation results for JobScout24.ch, providing a direct comparison of user satisfaction across the same dimensions.

The quantitative analysis revealed notable differences in user perceptions. As shown in Table 1, the overall general satisfaction with the prototype was rated at a rounded average of 5.38, whereas JobScout24.ch received a higher average score of 7.46.

Regarding output accuracy, the prototype was rated significantly lower with a rounded average score of 3.31, while JobScout24.ch achieved a rating of 7.54. This highlights that users found the output of JobScout24.ch to be more accurate compared to the prototype.

On the other hand, the prototype outperformed JobScout24.ch in the functionality satisfaction category, receiving a higher rounded average score of 8.31 compared to 7.62 for JobScout24.ch. This suggests that despite its lower accuracy, the prototype offered functionalities that were perceived as more satisfactory by users.

5.3.2. Qualitative Results

The qualitative evaluation of the user tests provided insights into the participants’ experiences, preferences, and criticisms of the prototype. This section synthesizes the verbal feedback, written responses, and observed interactions during the tests, offering valuable context to understand the numerical results and guiding areas for improvement.

Participants generally appreciated the innovative approach of the prototype. Many users highlighted that the ability to upload a CV and generate personalized job recommendations felt unique and user-friendly. Tester 3 also noted the option to either upload a CV or directly input details into a text field, emphasizing the flexibility of the prototype. The simplicity of the CV upload process was particularly well-received, with Tester 4 stating it was intuitive.

Overall, the concept itself was viewed as a novel and practical tool for job seekers, especially those seeking a quick and automated way to identify relevant opportunities. The innovative nature of the prototype was frequently acknowledged as something Tester 10 had “never seen before.” This aspect was cited as one of the key reasons why the prototype captured their interest, setting it apart from traditional platforms.

Despite its positive reception, the prototype faced significant criticism regarding its recommendation accuracy. Many participants stated that the job recommendations generated by the application were often irrelevant or only loosely aligned with their stated job interests and qualifications. Examples included recommendations for roles unrelated to the participants’ professional background or recommendations that focused solely on past experiences instead of aligning with their aspirations (Tester 5). This inconsistency led to frustration and reduced confidence in the system’s ability to provide meaningful results.

Another recurring issue was the lack of transparency in how recommendations were generated (Tester 11). Participants expressed a desire to understand why certain jobs were recommended and felt that the absence of this feature diminished their trust in the prototype. Transparency in the logic behind recommendations was a key area where users believed that improvements were necessary.

The lack of customization options was also a major point of dissatisfaction (Tester 11 and 12). Participants noted that the prototype did not allow them to refine or adjust their search parameters, such as filtering recommendations by location, industry, or specific job criteria. This limitation made the system feel rigid and less adaptable to individual needs, especially compared to more established platforms such as JobScout24.ch.

Privacy concerns were another issue raised during the tests. Some participants felt uneasy about uploading their CVs without a clear assurance of how their data would be handled. The absence of visible privacy policies or data protection measures further amplified these concerns (Tester 6 and 9).

When comparing the prototype to JobScout24.ch, most participants preferred the established platform. JobScout24.ch was praised for its accuracy, transparency, and customization options, which made it a more reliable tool for job searching. Tester 3 highlighted that JobScout24.ch’s ability to filter results and provide more relevant recommendations gave it a significant advantage over the prototype. Additionally, Tester 1 mentioned the presence of clear and comprehensive features, such as filtering and search customization, which made JobScout24.ch feel more user-centric and trustworthy.

However, users acknowledged the potential of the prototype, particularly its innovative approach. They believed that with significant improvements in accuracy, transparency, and customization, the prototype could become a valuable alternative to traditional platforms.

5.3.3. User Suggestions for Improvement

The qualitative feedback from the user tests revealed several areas where the prototype could be improved to better meet user expectations and compete with established platforms such as JobScout24.ch. The most pressing issue highlighted by the participants was the need to enhance the accuracy of job recommendations. The system should focus on refining the Web crawling to ensure that the recommended jobs align more closely with users’ skills, experiences, and career aspirations. By addressing this core functionality, the prototype can significantly improve its perceived value.

Another critical area for improvement is transparency. Users expressed a strong desire to understand why specific job recommendations were generated. Integrating a feature that explains the matching logic—such as highlighting relevant skills, keywords, or experiences from the CV—would build trust and encourage greater user engagement. Transparency is essential to make the system feel reliable and credible.

The participants also emphasized the need for customization options within the prototype. Unlike JobScout24.ch, which offers filtering tools, the prototype currently lacks the flexibility to refine search results. Introducing filters for location, industry, and job entry level would make the system more adaptable to individual preferences. This improvement would allow users to tailor their search experience, making the prototype feel more user-centric.

Privacy concerns were another significant issue raised during the tests. To address this, the prototype must incorporate a clear and visible privacy policy that reassures users about how their data is handled. Transparency around data security and privacy practices would mitigate user apprehensions and foster trust in the system.

Additionally, participants suggested customizing the output to improve usability. Allowing users to control the number of recommendations displayed or prioritize specific roles based on their preferences would enhance satisfaction and provide a better experience. This feature could be particularly valuable for users who feel overwhelmed by irrelevant or excessive recommendations.

Finally, while the CV upload feature was widely praised for its simplicity, participants noted that usability beyond uploading could be improved. Optimizing other aspects of the user interface to make the system more intuitive and user-friendly would further enhance the overall experience. By focusing on these areas, the prototype can address its current shortcomings and establish itself as a more competitive and reliable tool for job seekers.

6. Discussion

6.1. Keyword Extraction

LLMs excel at extracting contextually relevant keywords from unstructured data, such as CVs or user-provided descriptions. This capability allows the system to identify not only explicit attributes such as job titles and skills but also implicit preferences such as career aspirations or desired work conditions. During the development and programmatic analysis of the user tests, the prototype demonstrated effective keyword extraction, consistently identifying relevant keywords from the corresponding tested CVs. This reinforced the reliability of the LLM’s ability to accurately capture meaningful information, even in complex or unstructured formats.

6.2. Ranking and Contextual Matching

LLMs showcase excellent proficiency in semantic understanding, enabling them to evaluate and rank job listings based on their relevance to user input. By comparing job descriptions with extracted keywords and user preferences, the LLM effectively prioritizes listings that align closely with the candidate’s profile. Despite issues with the crawled data, which provided suboptimal and poorly matched outputs based on keywords, the LLM still managed to rank the best matching options at the top. This robustness was evident during development, where manual comparisons between unranked and ranked results highlighted the LLM’s ability to refine and elevate relevant listings, even when initial data inputs were imperfect.

6.3. Recommendation Accuracy and User Satisfaction

The prototype was primarily assessed through user feedback based on questionnaire responses. However, during user tests, it became evident that testers were dissatisfied with the long waiting times associated with the crawler despite being informed about this issue. This is understood to have negatively influenced their perception of the recommendation results. However, the testers appreciated the simplicity of the prototype’s structure and the straightforward integration of the LLM. This realization indicates that the primary cause of user dissatisfaction stemmed from the crawler’s performance rather than the LLM or the prototype design. For a future practical usage of this solution the web crawling and subsequent data analysis can be decoupled from user access so that a user can access any time the recommendations prepared in the background.

The integration of LLMs and web crawling technologies represents a promising approach to enhancing job recommendation systems. The study demonstrated the advantages of LLMs in keyword extraction and ranking, as well as the value of web crawling for real-time data acquisition. However, the challenges of accuracy, transparency, and performance must be addressed to unlock the full potential of this hybrid system.

7. Conclusions

The evaluation revealed that the prototype demonstrated strengths in user interaction and functionality, particularly regarding the CV upload feature and semantic extraction of key information. Through the LLM, the prototype was able to effectively identify relevant skills, job titles, and preferences from user inputs, showcasing its capacity for contextual understanding and accurate keyword extraction. Additionally, the ranking function based on the requested LLM provided job listings with personalized prioritization.

However, the conducted user tests identified significant performance limitations, particularly in the recommendation accuracy. The prototype frequently returned job listings that diverged from the users’ stated preferences, primarily due to inaccuracy and inefficiencies in the web crawling module. This limitation was reflected in the user ratings, where the prototype scored lower in accuracy compared to established platforms like JobScout24.ch.

The findings partially support the research hypothesis. While the LLM component significantly supports the extraction and contextual analysis of job-related data, the integration of web crawling for near real-time data collection proved challenging, requiring extensive fine-tuning to meet performance benchmarks. The overall system did not measure up to the benchmark platform in terms of accuracy, primarily due to web crawler limitations. Thus, while the concept holds theoretical promise, the prototype’s implementation fell short in practice and therefore the research hypothesis statement cannot be affirmed.

The performance of the prototype can be significantly enhanced by addressing several key areas. First, the web crawling capabilities need to be strengthened. This includes implementing more robust frameworks and refining the logic used for extracting relevant information from job listings. Optimizing how the crawler handles varying webpage structures, such as different HTML formats for job descriptions, will ensure that the collected data is consistent and relevant.

Additionally, the introduction of pre-crawled datasets for frequently updated job boards could improve system responsiveness. By periodically updating job data in the background, the system can reduce the latency users experience during real-time queries, leading to faster recommendations without sacrificing accuracy and concurrent changes in the job market.

By focusing on these improvements, future iterations of the system can become more accurate, efficient, and user-friendly, ultimately fulfilling the potential of LLM-based job recommendation systems. In future work, such a solution should also be evaluated in more comprehensive user tests which allow for statistical evaluations (significance testing).