An Enhanced Reference Paper Collection System with Generative AI for Paper Relevance Assessment

Abstract

Nowadays, academic paper search systems help researchers find related studies from a large number of publications. However, assessing whether each paper matches the user search query remains very difficult. Previously, we proposed a reference paper collection system, which assesses paper relevance using sentence embedding similarity. Unfortunately, the results are not always agree with human expert judgments because the system relies only on semantic similarity, whereas human experts considered predefined relevance criteria when evaluating papers. In this paper, we propose an enhanced reference paper collection system by integrating Generative Artificial Intelligence (GenAI) into the relevance assessment process. Candidate papers are retrieved from multiple academic sources and selected through a two-stage retrieval process before being assessed by the Mistral-Small-24B-Instruct-2501, a Large Language Model (LLM), using their titles, abstracts, disciplines, and keywords. Using predefined relevance criteria derived from human expert evaluations and a few-shot prompting approach, the LLM assigns one of four relevance categories and generates a short explanation for each assessment. The enhanced system was evaluated using 20 queries from five academic fields with human expert annotations as the reference standard. Compared with the results of the previous system, it improved relevance assessment agreement by 31.77% in Gwet’s AC2 and paper ranking quality by 11.27% in NDCG. In addition, the LLM was instructed to generate explanations based on the provided paper metadata to reduce unsupported statements. The generated explanations were evaluated using the RAGAS Faithfulness metric (0–1 scale) and an AspectCritic-based explanation relevance metric (1–3 scale), achieving average scores of 0.701 and 2.73 respectively, indicating that the generated explanations were generally supported by the provided paper metadata and appropriately described the relationship between the user query and retrieved papers. The average inference cost was approximately $0.000277 for each paper relevance assessment request. This enhanced system shows promising results in providing relevance assessments with explanations in our experiment across five academic fields.

1. Introduction

Nowadays, academic paper search systems are used to reduce the effort required to find related studies by researchers. With the increasing volume of academic publications, these systems help them obtain reference papers efficiently [1]. To offer proper references, the systems generally collect papers from available sources, estimate their relevance to the search query, and present the search results to the user [2]. However, determining whether a paper is relevant to the user’s research remains challenging because the relevance depends on user’s research interests and required information [3,4,5].

Recent studies have introduced Artificial Intelligence (AI)-based tools for scientific literature processing, particularly for article screening and systematic review assistance. For example, the ASReview framework applies machine learning for systematic review screening [6], while Large Language Models (LLMs) such as ChatGPT based on GPT-3.5-Turbo and GPT-4, have been investigated to support study selection activities in systematic reviews [7,8]. These approaches reduce the manual effort in processing large collections of publications. However, these existing approaches mainly focus on systematic literature review workflows, such as article selection based on predefined inclusion and exclusion criteria. They are not primarily designed to assess different relevance levels of retrieved reference papers according to a user’s research query or provide explanations that justify each relevance decision.

Previously, we proposed a reference paper collection system, which retrieves candidate papers from three academic paper sites and uses web scraping to obtain accessible papers with paper metadata. To assess the paper relevance, the system uses sentence embeddings to capture the semantic information of the search query and paper content, including the title, abstract, and keywords, and compares their similarity. The resulting similarity scores are used to rank the candidate papers and classify them into categories from high relevance to low relevance [9].

However, the relevance assessment results of the previous system sometimes did not align with human expert judgments. In some cases, papers classified as highly relevant by the system were considered less relevant by human experts. The system could rank less relevant papers higher than more relevant papers. This problem occurred because the previous system assessed relevance only from semantic similarity scores. It did not check whether each paper satisfied the relevance criteria related to the user search query.

Recent advances in Generative Artificial Intelligence (GenAI), particularly LLMs, have improved the analysis of scientific text [10,11]. Unlike sentence embedding methods comparing semantic similarity scores, LLMs use attention-based contextual processing to examine the relationship among the search query, the paper content, and the predefined relevance criteria. Some previous studies have applied LLMs to title and abstract screening to allow paper screening rules to be customized to specific review needs [12,13,14]. These studies indicate that LLMs have potential to support context-aware evaluation of scientific papers.

To address the limitations of our previous system, this paper proposes an enhanced reference paper collection system by integrating GenAI into the paper relevance assessment process. The enhanced system applies a two-stage candidate paper selection process, where the BM25 method is used for initial filtering and the MPNet sentence embedding model is used for semantic re-ranking before relevance assessment. The selected papers are then assessed using Mistral-Small-24B-Instruct-2501, an open-weight, instruction-tuned LLM with 24 billion parameters. This model was selected because it can follow predefined assessment criteria and generate structured JSON responses through the OpenRouter Chat Completion API without additional fine-tuning. Using a prompt containing relevance criteria, category assignment rules derived from human expert evaluations in our previous system, and few-shot examples, the model assigns each paper to one of four relevance categories and generates a short explanation for each assessment.

To evaluate the effectiveness of the enhanced system, this study focuses on three aspects: relevance assessment agreement, paper ranking quality, and generated explanation quality. Previous studies have demonstrated the potential of LLM-based paper screening and hybrid retrieval methods for assessing paper relevance and improving retrieval quality. These findings, together with previous research on evaluating generated explanations against source information, provide the basis for three hypotheses: the enhanced system improves the agreement with human relevance judgments (H1), improves the ranking of relevant papers (H2), and generates explanations that are supported by paper metadata and provide sufficient justification for the assigned relevance categories (H3). Since users typically focus on the highest-ranked search results, only the top-10 ranked papers for each query were evaluated. Human expert annotations were prepared by assigning one of the four relevance categories to each paper and were used as the ground truth for validation.

Gwet’s Agreement Coefficient 2 (Gwet’s AC2) [15] was used to evaluate H1 by measuring how consistently each system assigned the same relevance categories as the human experts, while Normalized Discounted Cumulative Gain (NDCG) [16] was used to evaluate H2 by measuring how effectively each system ranked highly relevant papers above less relevant ones. The results show relative improvements of 31.77% in Gwet’s AC2 and 11.27% in NDCG compared with our previous system.

In addition, H3 was examined by evaluating the explanations generated by the LLM using the Faithfulness and an AspectCritic-based explanation relevance metric provided by the RAGAS framework [17]. The enhanced system achieved average scores of 0.701 (0–1 scale) and 2.73 (1–3 scale), respectively, showing that the generated explanations were generally supported by the provided paper metadata and described the relationship between the user query and retrieved papers.

Additional analyses, including an ablation study and a comparison with a standard neural cross-encoder model, were conducted to analyze the contribution of each system component and examine the ranking performance against an alternative neural reranking approach.

The paper is structured as follows: Section 2 introduces related works in the literature. Section 3 presents our previous system. Section 4 presents the design of an enhanced reference paper collection system with generative AI. Section 5 evaluates the performance of the proposal. Section 6 concludes this paper with brief future works.

2. Related Works

In this section, we review relevant previous studies related to the proposed reference paper collection system. These papers were found by the proposed system.

2.1. Academic Paper Sources for Metadata Collection

First, we discuss academic paper sources for collecting scientific publications and related metadata. These sources can generally be categorized into preprint repositories, open bibliographic catalogs, and curated bibliographic databases. Each source provides different levels of coverage, metadata availability, and accessibility, which affect their suitability for reference paper collection.

Preprint repositories provide early access to scientific publications and emerging research topics in specific subject fields. In [18], Bloch et al. used the arXiv API to retrieve preprint papers covering physics, computer science, mathematics, statistics, and economics. In another study, detailed in [19], Eckmann et al. used the bioRxiv and medRxiv APIs for biological and biomedical preprints. In [20], Teixeira da Silva also discussed domain-specific repositories, including PsyArXiv, SocArXiv, and EdArXiv, for psychology, social science, and education. These repositories are useful for obtaining recent research outputs; however, their coverage is often limited to specific disciplines, publication metadata may be incomplete, preprints may not have undergone formal peer review, and links between preprint versions and published journal articles may not always be available.

Open bibliographic catalogs provide publication metadata compiled from multiple sources. In [21], Turner et al. used the Crossref API to improve paper retrieval and metadata completeness, together with Unpaywall, to obtain freely accessible PDF files. In [22], Haunschild et al. used OpenAlex, which provides publication metadata along with citation, research collaboration, and topic hierarchy information. Compared with preprint repositories, these catalogues provide broader publication coverage and additional metadata fields. However, the completeness and consistency of metadata may vary because the information is collected from different sources.

Curated bibliographic databases provide high-quality publication metadata maintained through curation processes. In [23], Rochim et al. used the Scopus API to collect structured publication, citation, and author data from author profiles. In [24], Haron et al. used the Web of Science (WoS) API to develop a journal recommendation system using publication metadata, citation indexing, and citation analysis. These databases provide structured and reliable metadata; however, access limitations and subscription requirements may restrict their use in open reference paper collection systems.

The reviewed academic paper sources provide publications and metadata with different characteristics. However, no single source may provide sufficient coverage and metadata availability for retrieving all papers relevant to a search query. Therefore, integrating multiple academic paper sources can provide a broader candidate paper pool for a reference paper collection system.

2.2. Retrieval Approaches for Candidate Paper Selection

Second, we review studies on retrieval approaches for candidate paper selection. This process is important for identifying papers that are potentially relevant to a search query. Previous studies have explored several retrieval approaches, including traditional keyword-based matching, sentence embedding models, and combinations of both methods.

In [25], Thakur et al. compared several traditional keyword-based matching methods, including TF-IDF, PL2, BM25, and BM25F. These methods estimate paper relevance by matching the keywords in a search query with the words appearing in a paper. Their results showed that BM25 and BM25F provide effective retrieval performance while maintaining computational efficiency. However, keyword-based methods mainly depend on matching terms and may miss semantically related papers that use different words. Therefore, keyword-based retrieval is effective for efficiently reducing large document collections, but additional semantic analysis may be required to identify papers with related research concepts.

In [26], Mori et al. compared sentence embedding models based on all-MiniLM-L6-v2 with BM25. Their results showed that sentence embeddings can better identify semantically related documents than keyword-based retrieval methods. However, applying sentence embedding models to large collections requires higher computational resources because each document needs to be represented as a vector before similarity calculation. Therefore, sentence embedding models are suitable for improving semantic retrieval quality, but their application to large-scale paper collections requires an efficient candidate selection strategy.

In [27], Chakraborty et al. proposed a hybrid retrieval approach using BM25 and BERT. First, BM25 retrieves candidate documents based on keyword matching. Then, BERT re-ranks the retrieved documents using semantic relevance scores. Their results showed that combining these two methods improves retrieval quality while maintaining retrieval efficiency compared with using a single retrieval method. This finding indicates that keyword-based retrieval and semantic retrieval methods have complementary roles, where keyword matching efficiently reduces the candidate space and semantic models improve the identification of conceptually related documents.

Motivated by these studies, we adopt a two-stage candidate paper selection that combines keyword-based matching and sentence embeddings. Since the candidate paper pool may contain thousands of papers, balancing retrieval efficiency and semantic relevance is important for selecting better candidate papers. These findings provide the basis for H2, which predicts that the enhanced system will improve the paper ranking quality compared with our previous system.

2.3. Generative AI for Paper Relevance Assessment

Finally, we review studies on the use of GenAI in the paper relevance assessment and the evaluation of generated explanations. Previous studies have used LLMs to assess paper relevance based on predefined assessment criteria in the prompt. Commonly, they use the inclusion criteria to describe which papers should be selected and the exclusion criteria to describe which papers should be removed from a literature review. Other studies have investigated the quality of LLM-generated explanations by examining their consistency with source information.

In [28], Dennstädt et al. used several LLMs, including FlanT5, OpenHermes-NeuralChat, Mixtral, and Platypus 2, for title and abstract screening in biomedical literature reviews. The models assessed paper relevance using the predefined inclusion and exclusion criteria, and represented the assessment results with Likert-scale relevance scores. The papers with scores of 3 or higher were considered relevant for further screening. Their results showed that LLMs can support paper relevance assessment while reducing manual screening effort.

In [29], López-Pineda et al. investigated the use of Llama 3 and ChatGPT-4o mini for title and abstract screening in biomedical systematic reviews. The models evaluated whether a paper matched the review objectives and eligibility requirements before generating included or excluded decisions. The resulting decisions were then compared with those of human reviewers. Their results suggested that LLMs can support the screening process and reduce reviewer workload, although manual validation is still needed to avoid overlooking relevant papers.

In [30], Tran et al. investigated the use of GPT-3.5-Turbo for title and abstract screening in systematic reviews and meta-analyses. The model applied predefined screening criteria in medical research, such as the treatment being studied, the comparison group, and the expected study outcomes, to determine whether a paper should be excluded before manual review. They demonstrated that LLMs can help identify which papers should be reviewed by human reviewers, reducing screening workload.

In addition to relevance assessment, evaluating the explanations generated by LLMs is also important. In [31], Cassola-Bacallao et al. evaluated LLMs as a judge for information extraction using several criteria, including Faithfulness to Source. They also examined the quality of generated justifications using the Explanatory Utility Index (EUI), which considers their accuracy and usefulness. Their results showed high agreement in assessing source faithfulness and high EUI scores, indicating that the evaluated models could provide consistent assessments and useful justifications.

The reviewed studies have investigated the use of LLMs for literature screening based on predefined criteria and the evaluation of generated explanations against source information. However, existing LLM-based screening approaches mainly focus on systematic review workflows, where the objective is to determine whether papers satisfy inclusion or exclusion requirements. In contrast, the enhanced reference paper collection system requires a different assessment setting, where retrieved papers are classified into multiple relevance levels and accompanied by explanations describing their relationship with the user query.

Inspired by these studies, we integrate GenAI into the enhanced system for paper relevance assessment. The assessment criteria and category assignment rules are defined in the prompt based on human expert evaluations to support graded relevance assessment, which addresses the need to distinguish different levels of paper relevance beyond binary inclusion or exclusion decisions. These findings provide the basis for H1, which predicts that the enhanced system will achieve better agreement with human relevance judgments than our previous system, and H3, which examines whether the enhanced system can generate explanations supported by paper metadata that justify the assigned relevance categories.

3. Review of Previous Reference Paper Collection System

In this section, we briefly review our previous reference paper collection system and discuss its limitations.

3.1. Overview of the Previous System

Our previous reference paper collection system retrieves papers from Semantic Scholar, DOAJ, and PubMed; removes duplicate records; and ranks the papers using sentence embedding similarity between the query and paper metadata. Multi-threading supports metadata retrieval, while web scraping is used to locate accessible PDF files, as shown in Figure 1 [32].

Figure 1. Overview of previous reference paper collection system.

The previous system therefore relied on semantic similarity scores to retrieve and rank candidate papers from multiple academic sources.

3.2. Human Expert Evaluation of Our Previous System

In the evaluation of this previous system, human experts reviewed each retrieved paper using the search query, title, abstract, and keywords. They assigned one of four relevance categories: highly relevant, moderately relevant, somewhat relevant, or low relevance. The expert annotations were used as the reference standard for evaluating the system results. Table 1 presents an example of the system results and the corresponding human expert annotations for the query “Learning System for Flutter Programming” together with the additional keywords.

Table 1. Top-5 retrieved papers for an example query.

Ranking	Paper Title	Similarity Score	Relevance Category	Human Expert Annotation
1	Implementation of Flutter-Based Learning Management System (LMS) at Universitas Andi Djemma Palopo	0.83	Highly Relevant	Somewhat Relevant
2	A Study of Learning Environment for Initiating Flutter App Development using Docker	0.65	Moderately Relevant	Moderately Relevant
3	An Independent Learning System for Flutter Cross-Platform Mobile Programming with Code Modification Problems	0.64	Moderately Relevant	Highly Relevant
4	Development of a Mobile Learning Management System using User-Centered Design (UCD) and the Flutter Framework	0.63	Moderately Relevant	Somewhat Relevant
5	Development of A Mobile Application for Individual English Language Learning using Flutter	0.62	Moderately Relevant	Somewhat Relevant

The example shows that higher similarity scores did not always become higher relevance according to the human experts. Similar problems were also observed in other queries. Across all evaluation queries, our previous system achieved 0.56 in agreement with human expert annotations and 0.76 in paper ranking quality on average [9]. These results suggest that semantic similarity alone was not sufficient to represent paper relevance according to the user search query.

3.3. Findings from the Human Expert Evaluation

To understand the limitations of semantic similarity-based relevance assessment, we analyzed the differences between the system results and the human expert annotations. Figure 2 summarizes representative findings obtained from this analysis and their implications for the design of the enhanced system.

Figure 2. Findings from the human expert evaluation and their implications for prompt design.

The analysis revealed several limitations of using semantic similarity alone for paper relevance assessment. Papers containing similar technology keywords could receive high similarity scores even when their relevance to the user search query was limited. Semantic similarity also did not fully capture pedagogical aspects considered by the human experts. In addition, papers with similar topics could belong to different relevance categories because of differences in their research focus.

These findings indicate that paper relevance depends on several factors beyond semantic similarity. During the assessment process, the human experts commonly considered five relevance criteria. Table 2 summarizes the identified criteria and their definitions.

Table 2. Relevance criteria identified from the human expert evaluation.

Criterion	Definition	Example
Research Domain	Determine whether the query and the paper belong to the same broad area of study.	Physical Sciences, Social Sciences, Health Sciences, Life Sciences, Multidisciplinary
Subject Field	Determine whether the query and the paper belong to the same academic discipline within the research domain.	Computer Science, Education, Psychology, Economics, Physics, Mathematics
Research Topic	Determine whether both studies investigate the same or closely related subject.	Programming Learning Assistant System, Brain Tumor Segmentation, Renewable Energy Forecasting
Research Focus	Determine whether both studies address the same objective, problem, or research question.	Improve prediction accuracy, evaluate model performance, optimize computational efficiency, conduct a systematic literature review
Research Method	Determine whether both studies use similar methodologies, analytical approaches, algorithms, models, or experimental procedures.	Transformer-based models, Large Language Models (LLMs), Graph Neural Networks (GNNs), Support Vector Machines (SVMs)

These criteria were frequently used by the human experts when giving relevance categories. Therefore, they were later incorporated into the LLM prompt to guide paper relevance assessment in the enhanced system.

4. Design of Enhanced Reference Paper Collection System

In this section, we present the design and workflow of the enhanced reference paper collection system.

4.1. Overview of Enhanced System

The enhanced system extends our previous system through several improvements in paper source integration, candidate paper selection, and relevance assessment. The detailed differences between the previous and enhanced systems are summarized in Table 3.

Table 3. Comparison between the previous system and the enhanced system.

System Component	Our Previous System	Enhanced System
Paper sources	Semantic Scholar, DOAJ, and PubMed	Integration of academic sources used in our previous work with additional sources, including preprint repositories, open bibliographic catalogues, and curated bibliographic databases
User input	API source selection, title query, and keywords	API source selection, research domain, subject field, title query, and keywords
Candidate paper selection	Direct ranking of retrieved papers based on sentence embedding similarity	Two-stage candidate selection using BM25 filtering and MPNet-based semantic ranking
Number of candidates before final assessment	Ranked retrieved papers directly	Top-100 after BM25 filtering and top-50 after MPNet ranking
Relevance assessment	Semantic similarity score-based classification using threshold values	LLM-based relevance assessment using predefined criteria and category assignment rules
Explanation generation	Not available	Generated explanations describing the relationship between the query and retrieved papers

Note: Italics indicate the names of academic sources, algorithms, models, and technical labels used in the system.

The improvements summarized in Table 3 are reflected in the overall workflow of the enhanced system, as presented in Figure 3.

Figure 3. Overview of enhanced reference paper collection system.

The enhanced system consists of three main stages: metadata collection, candidate paper selection, and GenAI-based relevance assessment. The details of each stage are described below.

First, the paper metadata is collected from multiple academic paper sources through their available APIs. The retrieved records are checked for PDF accessibility, duplicate records are removed, and the remaining metadata are stored in a unified database. This stage produces a candidate pool containing approximately 2700–3400 papers for each query.

Second, the candidate pool is reduced through two selection stages. The BM25 filter first selects the papers based on keyword matches with the user query. Then, the MPNet sentence embedding model reorders the filtered papers according to semantic similarity. The top-50 papers are selected because sending all retrieved metadata to the LLM would exceed its context window and increase processing cost.

Third, the top-50 papers are then assessed using the LLM together with a relevance assessment prompt based on the expert evaluation of our previous system. The model assigns each paper to one of four relevance categories: highly relevant, moderately relevant, somewhat relevant, or low relevance. It also generates a short explanation for each assessment. The papers are then sorted according to their assigned categories and presented to the user.

Through these three stages, the system separates large-scale paper retrieval from detailed relevance assessment. BM25 and MPNet reduce the large candidate pool to a manageable set, while LLM evaluates the selected papers using criteria that reflect human expert judgments. Therefore, the system can process papers from multiple sources efficiently while providing relevance categories and explanations that better represent the user search query.

4.2. Paper Metadata Retrieval from Multiple Sources

The enhanced system retrieves paper metadata from multiple academic sources through their available APIs. Using several sources can increase publication coverage and provide more complete bibliographic information. The retrieved metadata are processed and stored in the internal database. The sources, metadata fields, and their purposes are summarized in Table 4.

4.2.1. Academic Paper Sources

The listed sources provide different types of publication data with different levels of coverage and access conditions. Preprint repositories provide early publications, open catalogues supply wider metadata coverage, and curated databases provide structured citation and journal information.

These sources have different levels of publication curation, and the enhanced system does not assume that all retrieved papers have the same publication quality level. Preprint repositories are included to provide access to recent studies that may not yet be available in journal databases, although these papers may not have undergone formal peer review. To address the issue of multiple preprint versions, the system selects the latest available version of each paper for metadata collection. The source information is stored together with the collected metadata to maintain the origin of each retrieved paper. The current system focuses on assessing paper relevance based on the user query and does not perform an independent assessment of publication curation levels.

Regarding the use of different academic paper sources, the enhanced system does not assign different weights based on the database from which a paper is retrieved. Journal quartile rankings, such as Scopus Q1, are not directly used in the scoring calculation. Instead, journal impact factors are obtained using the impact-factor Python library [33] by matching journal information, such as ISSN, when available. Therefore, papers retrieved from different databases can receive publication reputation scores based on their citation counts and journal impact factors, regardless of their retrieval sources.

Table 4. List of academic paper sources, API endpoints, and retrieved metadata.

Source Type	Source	API Endpoint	Parameters	Example of Retrieved Metadata	Primary Purpose
Preprint Repository	arXiv [34]	http://export.arxiv.org/api/query (accessed on 10 March 2026)	search_query, start, max_results	Title, abstract, authors, subject categories, publication date, DOI, PDF URL, main URL	Early access to research in computer science, mathematics, physics, and related fields
Preprint Repository	bioRxiv [35], medRxiv [36]	https://api.biorxiv.org/details/biorxiv/, https://api.medrxiv.org/ (accessed on 10 March 2026)	server, start_date, end_date	Title, abstract, authors, publication date, subject areas, DOI, PDF URL, main URL	Early access to biological and biomedical research
Preprint Repository	PsyArXiv, SocArXiv, EdArXiv [37]	https://api.osf.io/v2/preprints/ (accessed on 10 March 2026)	filter[provider]	Title, abstract, authors, subjects, publication date, keywords, DOI, PDF URL, main URL	Coverage of psychology, social science, and education research
Open Bibliographic Catalog	Crossref [38]	https://api.crossref.org/works (accessed on 10 March 2026)	query.title, rows, cursor	Title, abstract, authors, journal name, publication date, DOI, citation counts, subject areas, journal volume, pages, PDF URL, main URL	Metadata enrichment and DOI, as well as resolution
Open Bibliographic Catalog	OpenAlex [39]	https://api.openalex.org/works (accessed on 10 March 2026)	search, page, per_page	Title, abstract, authors, concepts, citation counts, DOI, journal name, subject classification, PDF URL, main URL	Citation analysis and topic classification
Open Bibliographic Catalog	Unpaywall [40]	https://api.unpaywall.org/v2/ (accessed on 10 March 2026)	doi	DOI, open access status, PDF URL	Verification of open access PDF availability
Curated Bibliographic Database	Scopus [41]	https://api.elsevier.com/content/search/scopus (accessed on 10 March 2026)	query, count, start	Title, authors, publication date, citation counts, journal name, journal volume, pages, DOI, open access status, PDF URL, main URL	High-quality publication and citation metadata
Curated Bibliographic Database	Web of Science [42]	https://api.clarivate.com/apis/wos-starter/v1 (accessed on 10 March 2026)	q, db, limit, page	Title, abstract, authors, publication type, journal information, citation counts, subject areas, keywords, DOI, PDF URL, main URL	Curated citation indexing and journal information

Note: Italics indicate the names of academic sources, API parameters, and technical metadata identifiers.

Table 4. List of academic paper sources, API endpoints, and retrieved metadata.

Source Type	Source	API Endpoint	Parameters	Example of Retrieved Metadata	Primary Purpose
Preprint Repository	arXiv [34]	http://export.arxiv.org/api/query (accessed on 10 March 2026)	search_query, start, max_results	Title, abstract, authors, subject categories, publication date, DOI, PDF URL, main URL	Early access to research in computer science, mathematics, physics, and related fields
Preprint Repository	bioRxiv [35], medRxiv [36]	https://api.biorxiv.org/details/biorxiv/, https://api.medrxiv.org/ (accessed on 10 March 2026)	server, start_date, end_date	Title, abstract, authors, publication date, subject areas, DOI, PDF URL, main URL	Early access to biological and biomedical research
Preprint Repository	PsyArXiv, SocArXiv, EdArXiv [37]	https://api.osf.io/v2/preprints/ (accessed on 10 March 2026)	filter[provider]	Title, abstract, authors, subjects, publication date, keywords, DOI, PDF URL, main URL	Coverage of psychology, social science, and education research
Open Bibliographic Catalog	Crossref [38]	https://api.crossref.org/works (accessed on 10 March 2026)	query.title, rows, cursor	Title, abstract, authors, journal name, publication date, DOI, citation counts, subject areas, journal volume, pages, PDF URL, main URL	Metadata enrichment and DOI, as well as resolution
Open Bibliographic Catalog	OpenAlex [39]	https://api.openalex.org/works (accessed on 10 March 2026)	search, page, per_page	Title, abstract, authors, concepts, citation counts, DOI, journal name, subject classification, PDF URL, main URL	Citation analysis and topic classification
Open Bibliographic Catalog	Unpaywall [40]	https://api.unpaywall.org/v2/ (accessed on 10 March 2026)	doi	DOI, open access status, PDF URL	Verification of open access PDF availability
Curated Bibliographic Database	Scopus [41]	https://api.elsevier.com/content/search/scopus (accessed on 10 March 2026)	query, count, start	Title, authors, publication date, citation counts, journal name, journal volume, pages, DOI, open access status, PDF URL, main URL	High-quality publication and citation metadata
Curated Bibliographic Database	Web of Science [42]	https://api.clarivate.com/apis/wos-starter/v1 (accessed on 10 March 2026)	q, db, limit, page	Title, abstract, authors, publication type, journal information, citation counts, subject areas, keywords, DOI, PDF URL, main URL	Curated citation indexing and journal information

Note: Italics indicate the names of academic sources, API parameters, and technical metadata identifiers.

Regarding the metadata retrieval constraints, the process is performed dynamically during paper collection based on user queries. The retrieval process is limited to 30 s to maintain a practical retrieval time when collecting metadata from multiple API sources. For bioRxiv and medRxiv, which provide date-based retrieval endpoints, the system collects records published during the previous one-month period from the retrieval time. This period is selected to keep the number of retrieved records manageable under the same collection time constraint.

Access to subscription-based databases, including Scopus and Web of Science, depends on the available API access permissions. Although these databases provide high-quality publication metadata, some fields, such as abstracts, subject classifications, and PDF URLs, may not always be available depending on the provided access level. When some metadata fields are unavailable, the system performs paper relevance assessment using the available metadata, with the paper title serving as the primary field when the abstract information is not provided.

4.2.2. Metadata Integration

The retrieved metadata fields are not identical across all sources because each API provides different information according to its data structure and access conditions. Some sources provide detailed bibliographic information, while others provide only basic metadata fields. Therefore, the system collects all available metadata from each source and converts the records into a common schema before further processing. Missing fields are retained as unavailable information when they cannot be obtained from the original source or an additional enrichment process.

The paper metadata retrieval and integration workflow is illustrated in Figure 4.

Figure 4. Paper metadata retrieval and integration process.

The process begins when a user submits a search query through the web interface. The query is sent to the selected academic paper sources through their APIs. Each source returns metadata in either JSON or XML format. Since the returned structures differ, the system converts the records into a common schema containing 24 standardized metadata fields, including paper identifiers, bibliographic information, publication details, citation information, subject information, access information, and source information.

Then, duplicate papers are identified using DOI matching when available. For papers without DOI, title matching combined with other available metadata, such as the author information and publication details, is used to identify possible duplicate records. When several records refer to the same paper, the system keeps the most complete record and fills its missing fields using obtained data from the other records. The system also checks PDF accessibility. When a PDF URL or metadata field remains unavailable after enrichment using Crossref, Unpaywall, and web scraping, it is marked as “N/A”.

4.2.3. Database Structure

The cleaned and standardized metadata are stored in a relational MySQL database. In addition to storing paper metadata, the database manages other information required throughout the reference paper collection process, including query history, LLM-based relevance assessment results, human expert evaluations, and system performance records. The entity relationship diagram (ERD) of the database structure is presented in Figure 5.

Figure 5. Entity relationship diagram of the MySQL database used in the enhanced system.

The database consists of several related tables. The tb_papers table stores the collected paper metadata, such as title, abstract, authors, citation information, source information, PDF availability status, and other standardized metadata fields. The tb_query_history table records user query information, while tb_ai_rating table stores relevance assessment results generated by the LLM-based assessment process, including assigned categories and generated explanations. The tb_human_rating and tb_system_performance tables store human expert evaluations and system evaluation results, respectively.

Through this metadata retrieval stage, the system produces a unified paper metadata collection with fewer duplicate records and more complete information. The resulting collection is then used as the input for the subsequent candidate selection process.

4.3. Two-Stage Candidate Paper Selection

The two-stage candidate paper selection reduces the large paper collection into a manageable one before GenAI-based relevance assessment. The objective is to retain potentially relevant papers while removing less relevant candidates. The process consists of BM25-based filtering, MPNet-based semantic reranking, and mapping of the resulting similarity scores to initial relevance categories. The overall workflow is shown in Figure 6.

Figure 6. Two-stage candidate paper selection using BM25-based filtering and MPNet-based semantic reranking.

4.3.1. Stage 1: BM25-Based Filtering

In the first stage, BM25-based filtering is applied to the large collection of retrieved papers. This stage provides a fast screening process before the more costly semantic comparison. In this study, BM25 estimates the relevance score between the user query information and the retrieved papers based on term matching between the query and paper fields. The method adopts the Okapi BM25 formulation [43] and is implemented using the BM25Okapi algorithm provided by the rank-bm25 library [44].

Before applying BM25, the query and paper fields (title and abstract) are converted into lowercase and tokenized into individual terms. The system does not apply stemming, lemmatization, or stopword removal during preprocessing to preserve the original terms from different academic fields. The paper title and abstract are processed separately as document representations. The title-based score is obtained by comparing the query title with the paper title, while the abstract-based score is obtained by comparing the expanded query containing the query title, keywords, domain, and subject field with the paper abstract. The BM25 algorithm constructs corpus indexes based on term frequency, inverse document frequency, document length, and average document length to compute the relevance score of each paper.

The BM25 score of paper D for query Q is calculated using Equation (1):

$$BM25(Q,D)=\sum _{t\in Q}IDF\left(t\right)·{\displaystyle \frac{f(t,D)(k_1+1)}{f(t,D)+k_1\left(1-b+b\frac{\left|D\right|}{avgdl}\right)}}$$

(1)

where $f(t,D)$ denotes the frequency of term t in paper D, $IDF\left(t\right)$ is its inverse document frequency, $\left|D\right|$ is the paper length, and $avgdl$ is the average paper length. The parameters $k_1$ and b control term frequency saturation and document length normalization, respectively. The parameter $ϵ$ is used for IDF smoothing in the rank-bm25 implementation. In this study, the default parameters of the rank-bm25 library are used, where $k_1=1.5$, $b=0.75$, and $ϵ=0.25$. The final BM25 score is obtained by combining the title-based score and abstract-based score, as defined in Equation (2):

$$BM{25}_{total}=BM{25}_{title}+BM{25}_{abstract}$$

(2)

where $BM{25}_{title}$ and $BM{25}_{abstract}$ denote the scores obtained from the title and abstract matching processes, respectively.

Then, the papers are ordered by their final BM25 scores, and the top-100 papers are retained. The papers with lower BM25 scores are removed at this stage, while papers containing terms related to the user query remain available for further assessment. The top-100 limit is selected as a practical choice to reduce the amount of data processed by the sentence embedding model while maintaining a sufficient number of candidates for the subsequent semantic ranking. Since the sentence embedding process requires more memory and a longer processing time compared with BM25-based filtering, this setting provides a balance between the number of retained candidates and computational cost. Thus, the first stage provides a smaller candidate set for the next stage.

4.3.2. Stage 2: MPNet-Based Reranking

In the second stage, the top-100 papers are reordered according to the semantic similarity score. This stage can identify related papers even when they use terms that differ from those in the user query. To estimate the semantic similarity score, sentence embeddings are generated for the user query and paper metadata using the all-mpnet-base-v2 model based on the sentence-transformer framework [45]. The implementation uses the publicly available model checkpoint from Hugging Face [46]. The similarity between the query and paper representations is then measured using Cosine Similarity.

The semantic similarity score is calculated from four comparison components: (1) the similarity between the query title and paper title; (2) the similarity between the expanded query containing the query title, keywords, domain, and subject field with the paper abstract; (3) the similarity between query keywords and paper keywords; and (4) the similarity between query domain and subject fields and paper disciplines. These components are combined using predefined heuristic weights, where the title similarity, abstract similarity, keyword similarity, and discipline similarity are assigned weights of 0.55, 0.35, 0.05, and 0.05, respectively. The title similarity receives the highest weight because the title directly represents the main topic of the paper, while the abstract provides additional research context. Keyword and discipline similarities are assigned lower weights because they provide supporting information for the relevance assessment.

The semantic similarity formulation is given in Equations (3) and (4):

$$STS={\displaystyle \sum _{i=1}^4}{w}_i·CosEmb(Q_i,D_i)$$

(3)

$$\begin{array}{cc}\hfill STS=& 0.55·CosEmb(Q_{\mathrm{title}},D_{\mathrm{title}})\hfill \\ & +0.35·CosEmb(Q_{\mathrm{expanded}},D_{\mathrm{abstract}})\hfill \\ & +0.05·CosEmb(Q_{\mathrm{keywords}},D_{\mathrm{keywords}})\hfill \\ & +0.05·CosEmb(Q_{\mathrm{domain}+\mathrm{subject\; fields}},D_{\mathrm{discipline}})\hfill \end{array}$$

(4)

where $w_i$ represents the heuristic weighting parameter assigned to each similarity component. $Q_i$ and $D_i$ denote the i-th query attribute and the corresponding paper attribute, respectively. $CosEmb(.)$ refers to cosine similarity between embedding vectors.

Subsequently, the semantic similarity score is combined with publication reputation information based on available citation counts and journal impact factors. The reputation information is used as an additional factor rather than the main relevance measure. This allows papers with similar semantic similarity scores to be ranked differently according to their publication reputations. However, these indicators do not directly measure the methodological quality of individual studies, such as the appropriateness of their research design and data analysis or the validity of their findings.

The reputation score R is calculated based on the normalized citation count and normalized journal impact factor, as shown in Equation (5):

$$R=0.6C_{\mathrm{norm}}+0.4JI{F}_{\mathrm{norm}}$$

(5)

where $C_{\mathrm{norm}}$ denotes the normalized citation count and $JI{F}_{\mathrm{norm}}$ represents the normalized journal impact factor. Both values are normalized into the range of 0–1 using min–max normalization, as shown in Equation (6):

$$x_{\mathrm{norm}}={\displaystyle \frac{x-x_{\min }}{x_{\max }-x_{\min }}}$$

(6)

where x represents the original value, while $x_{\min }$ and $x_{\max }$ denote the minimum and maximum values in the corresponding dataset.

The citation count is assigned a higher weight because it reflects the influence of an individual paper, while the journal impact factor represents the general reputation of the publication venue.

The final score is calculated using Equation (7):

$$S=STS+\alpha R$$

(7)

where S is the final score, $STS$ is the semantic similarity score produced by MPNet, R is the publication reputation score, and $\alpha$ controls the contribution of the reputation score.

In this study, $\alpha =0.15$ is used as a heuristic weighting parameter. This setting keeps semantic similarity as the main component while allowing publication reputation to provide additional ranking information.

Since the combined score may exceed the range of 0–1 after adding the reputation component, the score is normalized before assigning the initial relevance categories. The normalization is performed using min–max normalization, as shown in Equation (8):

$$S_{norm}={\displaystyle \frac{S-S_{min}}{S_{max}-S_{min}}}$$

(8)

Table 5 shows how the semantic similarity and publication reputation scores are combined to calculate the final ranking scores for the query “Protein Phosphatase 2A in Osteoblast Differentiation” in the MNS (Dental (Oral) Morphology) field. Papers were retrieved from different academic sources and evaluated using the same scoring method.

Table 5. An example of combining semantic similarity and publication reputation scores across academic paper sources.

Ranking	Paper	Source	DOI	ISSN	Citations	JIF	STS	R	S
1	Role of Protein Phosphatase 2A in Osteoblast Differentiation and Function	Semantic Scholar	10.3390/jcm6030023	N/A	21	0	0.981	0.075	0.992
2	Protein Phosphatase 2A C$\alpha$ Regulates Osteoblast Differentiation and The Expressions of Bone Sialoprotein and Osteocalcin via Osterix Transcription Factor	PubMed	10.1002/jcp.24250	0021-9541	13	4.5	0.861	0.237	0.896
3	Reduction of Protein Phosphatase 2A C$\alpha$ Enhances Bone Formation and Osteoblast Differentiation Through The Expression of Bone-Specific Transcription Factor Osterix	PubMed	10.1016/j.bone.2011.06.004	N/A	14	0	0.851	0.049	0.858
4	High Glucose Inhibits O-GlcNAc Transferase Translocation in Early Osteoblast Differentiation by Altering Protein Phosphatase 2A Activity	PubMed	10.1002/jcp.31524	0021-9541	1	4.5	0.681	0.191	0.710
5	Reduction of Protein Phosphatase 2A C$\alpha$ Promotes In Vivo Bone Formation And Adipocyte Differentiation	Semantic Scholar	10.1016/j.mce. 2017.11.005	N/A	8	0	0.695	0.026	0.699
…	…	…	…	…	…	…	…	…	…
12	Progressive Changes in The Protein Composition of The Nuclear Matrix During Rat Osteoblast Differentiation	Crossref	10.1073/pnas.87.12.4605	0027-8424	142	9.4	0.530	0.929	0.669

Note: Citation counts and journal impact factors are normalized using the minimum and maximum values among the 12 papers. Unavailable JIF values are treated as zero in the calculation.

The papers are reordered according to the normalized final score. Papers with higher scores are placed at higher positions and are then mapped to initial relevance categories. This stage narrows the candidate set from 100 to the top-50 papers based on the normalized final score.

The top-50 setting is selected as a practical choice to reduce the number of papers forwarded to the LLM-based relevance assessment stage while maintaining sufficient highly ranked candidates. Since a larger candidate set increases the number of tokens processed by the LLM and increases the response time of the LLM API, this setting helps maintain a practical response time during relevance assessment.

4.3.3. Initial Relevance Category

After reranking, each paper is assigned an initial relevance category based on its final score. This category provides an initial relevance indicator that is later included in the LLM prompt. The initial label does not determine the final category because the LLM may retain or revise it according to the predefined assessment criteria.

To define these categories, the thresholds shown in Table 6 were determined based on the interpretation of semantic similarity scores, where higher scores indicate stronger alignment between textual representations. Semantic similarity can be measured using embedding-based approaches such as Sentence-BERT with cosine similarity [45], while the score ranges were adopted from practical interpretations of semantic similarity scores [47]. These categories were used only as preliminary indicators for the LLM-based relevance assessment and did not determine the final relevance labels.

Table 6. Score ranges for initial relevance categories.

Score Range	Category
0.80 ≤ score ≤ 1.00	Highly Relevant
0.60 ≤ score < 0.80	Moderately Relevant
0.40 ≤ score < 0.60	Somewhat Relevant
score < 0.40	Low Relevance

The assigned relevance categories and ranking results are stored in the database. The top-50 papers are then sent to the LLM-based relevance assessment stage. This reduction keeps the strongest candidates while limiting the amount of metadata processed by the model. It also separates the initial score-based selection from the final criteria-based relevance assessment.

4.4. Generative AI-Based Relevance Assessment

The third stage applies LLM for relevance assessment of the top-50 papers retained after the two-stage candidate paper selection process. By incorporating the predefined relevance criteria into the prompt, this stage evaluates paper relevance beyond semantic similarity and considers the factors used by human experts during assessment. This stage consists of prompt construction, relevance assessment, and explanation generation. The workflow of this stage is presented in Figure 7.

Figure 7. LLM-based relevance assessment and explanation generation.

4.4.1. Prompt Construction

The relevance assessment prompt adopts a few-shot prompting strategy. It consists of four main components: relevance criteria, category assignment rules, few-shot examples, and assessment inputs containing the user query and paper metadata. The relevance criteria represent the aspects considered during relevance assessment, while the category assignment rules define the conditions for assigning each relevance category based on the relationship between the query and paper metadata. The few-shot examples provide example papers with initial relevance categories to illustrate the relevance category patterns for the LLM.

The relevance criteria and category assignment rules were formulated based on the human expert evaluation of the previous system, which was conducted using the same set of research queries in five academic fields. The evaluation results were analyzed to identify the aspects considered by experts during relevance assessment and their relationship with relevance categories. These findings were converted into relevance criteria and category assignment rules for the LLM prompt. These components are used by the LLM as guidelines for assessing paper relevance rather than as direct relevance labels for the evaluated papers. For generating the few-shot examples, the previous MPNet-based similarity assessment process was used to provide initial relevance category references.

As shown in Figure 8, the relevance criteria and category assignment rules remain fixed, while the few-shot examples and assessment inputs are replaced during execution according to the retrieved papers and user query. To provide few-shot examples for the LLM, the top-three papers from each initial relevance category are selected based on the highest semantic similarity scores among the retrieved papers in each category. This selection provides representative examples for the LLM while maintaining a reasonable prompt size.

Figure 8. Structure of relevance assessment prompt.

During the relevance assessment, the prompt does not use the full-text paper content considering the computational cost and context requirements. Instead, the LLM receives the paper metadata including the title, abstract, discipline or subject, and extracted keywords. Although the initial relevance categories in the few-shot examples are provided as references, the LLM determines the final relevance category according to the provided relevance criteria and category assignment rules, by assessing the relationship between the query and paper metadata. The prompt instructs the LLM to generate a short explanation in the specified output format. The complete prompt is provided in Appendix A.

4.4.2. Relevance Assessment and Explanation Generation

The constructed prompt is sent to Mistral-Small-24B-Instruct-2501 [48] through the OpenRouter Chat Completion API. This model was selected based on its instruction-following capability, general knowledge, support for structured JSON outputs, and a context window of 32,768 tokens. Its performance on the Massive Multitask Language Understanding-Pro (MMLU-Pro) and Graduate-Level Google-Proof Q&A (GPQA) benchmarks indicates its ability to handle knowledge and reasoning tasks across academic fields. These capabilities support the use of predefined relevance criteria and the generation of structured responses containing relevance categories and short explanations. Its context window allows the model to process all prompt components within a single request. No additional fine-tuning was applied in this study.

Using the user query, the paper metadata, and relevance criteria provided in the prompt, the model evaluates each paper and determines whether its initial relevance category should be retained or revised. It then assigns one of the four categories: highly relevant, moderately relevant, somewhat relevant, or low relevance.

The model also generates a short explanation on each assigned category. This explanation describes the relationship between the paper and the user query based only on the available metadata provided in the prompt. An example of the generated relevance assessment is shown in Figure 9.

Figure 9. Example of relevance assessment result by LLM.

Finally, the assigned categories and generated explanations are stored in the database. The papers are sorted from highly relevant to low relevance and presented to the user. Thus, this stage replaces score-only relevance estimation with criteria-based assessment and provides a short reason for each result.

5. Evaluation and Analysis

In this section, we present the experimental evaluation of the enhanced system. The evaluation includes the experimental preparation, comparison with our previous system, ablation study, comparison with a standard neural cross-encoder model, and further analysis of the system performance and limitations.

5.1. Testing Preparations

This subsection describes the preparation procedures for the experimental evaluation, including query selection, human expert annotation, experimental environment, and evaluation metrics.

5.1.1. Query Selection

The experiments used 20 queries from five academic fields: Computer Science and Engineering (CSE), Medical and Health Sciences (MHS), Mathematics and Natural Sciences (MNS), Economics and Management (EM), and Law, Social Science, and Humanities (LSH). Four queries were prepared for each field with support from human experts before the retrieval experiments. These fields were selected to test the system across different vocabularies and domain-specific terms. The complete query list is provided in Appendix B.

5.1.2. Human Expert Profiles and Relevance Annotation Procedure

This study involved 20 human experts, including 6 lecturers, 2 researchers, and 12 doctoral students. The experts were selected based on their research background, experience in scientific research, and experience in analyzing the academic literature within the evaluated fields. The expert group mainly consisted of researchers with intermediate research experience (5–10 years), while two experts were categorized as early-stage researchers based on their shorter research experience of less than five years. The research experience refers to the period since the beginning of their formal research activities, including thesis research, research projects, and scientific publications. Their publication records and academic review experience were confirmed through publicly available academic profiles and brief interviews.

The expert profiles across the five academic fields are summarized in Table 7.

Table 7. Profiles of human experts across five academic fields.

Academic Field	Academic Position	Research Experience	Publication Experience	Review Experience
CSE	2 Doctoral Students in Information and Communication Systems	<5 years	10–15 papers	1–3 years
	2 Doctoral Students in Information Science and Engineering	5–10 years	20–40 papers	3–5 years
MHS	2 Doctoral Students in Interdisciplinary Science and Engineering in Health Systems	>10 years	10–20 papers	3–5 years
	2 Doctoral Students in Dental (Oral) Morphology	>10 years	10–20 papers	3–5 years
MNS	2 Researchers in the Mathematical Biology and Environmental Dynamics	5–10 years	20–40 papers	3–5 years
	2 Lecturers in Statistics	5–10 years	20–40 papers	3–5 years
EM	2 Lecturers in Management	>10 years	20–30 papers	3–5 years
	2 Doctoral Students in Accounting	5–10 years	10–20 papers	3–5 years
LSH	2 Lecturers in Law	5–10 years	20–30 papers	3–5 years
	2 Doctoral Students in Teaching English to Speakers of Other Languages (TESOL)	>10 years	30–40 papers	>5 years

For each academic field, four human experts were assigned as annotators. Since each query was evaluated by three experts to enable majority voting, the annotators were rotated across queries to distribute the annotation workload equally. In each rotation, three experts independently evaluated the retrieved papers, while the remaining expert was not assigned. Therefore, each expert evaluated the same number of queries within their respective field.

Each assigned expert reviewed the top-10 retrieved papers for each query together with the paper title, abstract, and keywords before assigning one of four relevance categories. Each paper was evaluated individually with respect to its corresponding query, and the expert assigned a relevance category based on the annotation criteria described below. The expert selected the corresponding relevance category for each paper through the annotation interface shown in Figure 9, and the selected category was recorded in the system for subsequent inter-expert agreement analysis and consensus annotation. With four queries in each academic field, each expert evaluated approximately 30 paper-query pairs within their respective field. The assigned experts within each academic field remained unchanged throughout the annotation process and followed the same relevance assessment criteria.

To support consistent annotations, the experts used the Systematic Mapping Study (SMS) method [49]. Each paper was examined according to its research domain, subject field, research topic, research focus, and research method before a relevance category was assigned. The research domain and subject field were classified according to the Scopus Subject Area and Subject Area Classification based on the All Science Journal Classification (ASJC) scheme [50], while the research topic, research focus, and research method were identified manually by the human experts based on the content of each paper. The relevance category was determined by comparing these characteristics with the corresponding user query based on the following rules:

• Highly relevant: The paper belongs to the same research domain and subject field, investigates the same research topic, addresses the same research focus/objective, and applies similar research methods.
• Moderately relevant: The paper belongs to the same research domain and subject field and investigates the same research topic, but addresses a narrower research focus/objective than the query. Similar or partially similar research methods provide additional support for this category.
• Somewhat relevant: The paper belongs to the same or related research domain and subject field, investigates a related research topic, and addresses a broader or more general research focus/objective than the query. The paper is related to the query but does not directly address the main research focus.
• Low relevance: The paper belongs to a different research domain or subject field or investigates an unrelated research topic. Similar research methods alone are not sufficient to classify a paper as relevant.

An example of this annotation structure for the CSE field is presented in Table 8.

Table 8. Example of systematic mapping with human expert annotation in the CSE field.

Paper Title	Domain/Subject Field	Research Topic	Research Focus	Research Method	Relevance Category
An Independent Learning System for Flutter Cross-Platform Mobile Programming with Code Modification Problems	Physical Sciences/Computer Science	Flutter Learning Assistant System	Self-learning system for Flutter/Dart using Code Modification Problems (CMP)	Exercise-based learning system with structured programming tasks	Highly Relevant
An Image-Based UI Testing Method for Flutter Programming Learning Assistant System	Physical Sciences/Computer Science	Automated Assessment in Flutter Learning System	Evaluation of student-generated Flutter User Interface outputs	Image-based comparison using ORB/SIFT feature matching	Moderately Relevant
Mobile Academy: A Ubiquitous Mobile Learning (mLearning) Platform	Physical Sciences/Computer Science	General Mobile Learning Platform	Development of a mobile learning platform to support learning via mobile devices	Requirements analysis, system development, and evaluation using UCD in HCI	Somewhat Relevant
Flutter Speed Prediction by Using Deep Learning	Physical Sciences/Mechanical Engineering	Aeroelasticity	Prediction of aeroelastic flutter speed	Deep learning using DNN and LSTM	Low Relevance

Before the consensus process, the agreement among the three assigned human experts for each query was measured using Gwet’s AC2 to evaluate the consistency of independent annotations. Since each academic field contained four queries, the reported values represent the mean and standard deviation of Gwet’s AC2 calculated across the four queries within each field, as summarized in Table 9.

Table 9. Inter-expert agreement before consensus annotation.

Academic Field	Gwet’s AC2 (Mean ± SD)
CSE	0.932 ± 0.047
MHS	0.909 ± 0.102
MNS	0.863 ± 0.112
EM	0.892 ± 0.025
LSH	0.912 ± 0.068
Average	0.902 ± 0.071

The final category was determined by majority voting among the assigned experts. When no majority was reached, the experts discussed the differences and assigned a consensus category. These consensus labels were used as reference labels for evaluating the relevance categories generated by the enhanced system.

5.1.3. Experimental Setup

The experiments were conducted using a CUDA-enabled GPU for sentence embedding and data processing. The system was developed in Python 3, deployed with Docker, and connected to a MySQL database for storing paper metadata, relevance categories, and generated explanations. The hardware and software specifications are listed in Table 10.

Table 10. Hardware and software specifications in experiment.

Specification	Details
GPU Model	NVIDIA GeForce RTX 2080
CUDA Version	12.6
Driver Version	560.35.03
GPU Memory	8 GB
Docker Version	27.3.1, build ce12230
Python Version	3.12.3
MySQL Version	8.4.3
BM25 Library	rank-bm25
Sentence Embedding Model	all-mpnet-base-v2

Note: Italics indicate the names of the software library and sentence embedding model used in the experiment.

For the relevance assessment and explanation evaluation stages, the LLM configurations are summarized in Table 11. The Mistral-Small-24B-Instruct-2501 model was used to generate relevance categories and explanations, while GPT-4o-mini was used as the evaluator model for the RAGAS Faithfulness and AspectCritic-based explanation relevance metrics. A fixed prompt template with few-shot examples was applied across all evaluations.

Table 11. LLM configuration for relevance assessment and explanation evaluation.

Component	Configuration
Relevance Assessment Model	Mistral-Small-24B-Instruct-2501 via OpenRouter API
Inference Provider	DeepInfra
Temperature	0
Top-p	1.0
Maximum Output Tokens	256
Random Seed	42
Prompt Strategy	Fixed template with few-shot examples
Explanation Evaluation Framework	RAGAS
Faithfulness and AspectCritic Model	GPT-4o-mini via OpenRouter API

Note: Italics indicate the names of language models, inference providers, and evaluation frameworks used in the experiment.

The inference temperature was set to 0 to reduce output variation in the generated relevance categories and explanations during the experiment. The top-p value was set to 1.0 to avoid additional restrictions on token selection, while the maximum output length was limited to 256 tokens to accommodate the generated relevance category and short explanation. A fixed prompt template with few-shot examples was used to maintain consistent instructions and provide examples of the expected output format. A fixed random seed of 42 was used to control randomness in processes that support seed configuration. The enhanced system was evaluated through one complete run using 20 queries distributed across five academic fields.

The API token cost was calculated using usage logs exported from OpenRouter. The logs were filtered by model name and the dates of the experimental runs, and the average cost and token usage were calculated across all relevant records. For each of the 20 queries, the top-50 papers ranked using the MPNet sentence embedding model were assessed using the Mistral-Small-24B-Instruct-2501 model, resulting in approximately 1000 paper relevance assessment requests. This model had an average cost of $0.000277 per paper relevance assessment request, with average input and output token usage of 5353.574 and 120.298 tokens, respectively.

The top-50 papers for each query were sorted in descending order based on the ordinal relevance categories assigned by Mistral. After sorting the papers, the RAGAS evaluation with GPT-4o-mini assessed the relevance explanations for the top-10 papers with the highest relevance categories, resulting in approximately 200 evaluated explanations across the 20 queries. The average cost was $0.00025 per evaluated explanation, with average input and output token usage of 855.810 and 222.396 tokens, respectively.

This environment supported the retrieval, candidate selection, relevance assessment, and explanation generation processes used in the experiments.

5.1.4. Evaluation Metrics

The evaluation measures relevance agreement, ranking quality, and generated explanation quality. Table 12 summarizes the metrics used in this study.

Table 12. Summary of evaluation metrics.

Category	Purpose	Metrics
Relevance Assessment Agreement	Measure the agreement between system-generated relevance categories and human expert annotations	Gwet’s AC2
Paper Ranking Quality	Measure the quality of ranked retrieved papers using human expert relevance judgments as the ground truth for ideal ranking	NDCG
Generated Explanation Quality	Measure whether generated explanations are supported by retrieved paper metadata and appropriately describe the query–paper relationship	RAGAS Faithfulness, custom AspectCritic-based explanation relevance evaluation

The details of each evaluation metric and its calculation are described below:

• Gwet’s AC2: This measures the agreement between the relevance categories assigned by the enhanced system and human experts. It is suitable for ordinal categories because it considers the degree of disagreement between categories and is less affected by imbalanced category distributions [51], and has also been applied in recent AI-based assessment studies [52]. In this study, the four relevance categories were converted into ordinal values: low relevance (0), somewhat relevant (1), moderately relevant (2), and highly relevant (3). A linear weighting scheme was applied because the relevance categories represent ordered levels of paper relevance, where larger differences between categories receive larger penalties. This weighting approach considers the distance between categories, where smaller differences receive higher agreement values and larger differences receive lower agreement values.

The weighted agreement coefficient is calculated using Equation (9):

$$Gwe{t}^{\prime }sAC2={\displaystyle \frac{P_o-P_e}{1-P_e}}$$

(9)

where $P_o$ represents the weighted observed agreement and $P_e$ represents the weighted expected agreement under chance agreement. The linear weight assigned to the agreement between categories i and j is defined as Equation (10):

$$w_{ij}=1-{\displaystyle \frac{|i-j|}{k-1}}$$

(10)

where i and j represent the ordinal category indices and k represents the total number of relevance categories.

The calculation of Gwet’s AC2 was performed using the Python irrCAC package with the CAC class and the gwet() function, which supports chance-corrected agreement coefficients, including Gwet’s AC1/AC2, and weighted agreement analysis for ordinal categories [53].

• NDCG: This measures ranking quality by considering both the relevance level and position of retrieved papers based on the discounted cumulative gain framework [54], with recent studies applying NDCG for retrieval performance evaluation [55]. In this study, the human expert relevance labels used in the Gwet’s AC2 evaluation were also used as reference labels for NDCG. The relevance categories assigned by the LLM were converted into ordinal scores to rank retrieved papers in descending order, while papers with the same category retained their original order in the system output. The relevance scores for positions 1–10 were obtained from the corresponding human expert labels, where DCG was calculated based on the system ranking and IDCG was obtained by sorting the human expert labels in descending order.

The DCG and NDCG values were calculated using Equations (11) and (12):

$$DCG@K={\displaystyle \sum _{i=1}^K}{\displaystyle \frac{2^{re{l}_i}-1}{{\log }_2(i+1)}}$$

(11)

$$NDCG@K={\displaystyle \frac{DCG@K}{IDCG@K}}$$

(12)

where $re{l}_i$ represents the relevance score assigned by human experts at position i, and K represents the number of evaluated retrieved papers, where $K=10$ is used.

• Faithfulness: This evaluates whether the claims in the generated explanation are supported by the available paper metadata provided to the LLM [17]. This metric examines the consistency between the generated explanation and the information available during the relevance assessment process. The score is calculated based on the proportion of the supported claims, where a higher value indicates that more claims in the explanation are supported by the available metadata, as shown in Equation (13):

  $$Faith fulness={\displaystyle \frac{\left|V\right|}{\left|C\right|}}$$

  (13)

  where C represents the total number of claims extracted from the generated explanation, and V represents the subset of claims supported by the available paper metadata.
• AspectCritic: This is an LLM-based evaluation method provided by the RAGAS framework that assesses generated outputs according to predefined aspects [56]. In this study, a custom aspect was defined to evaluate the explanation relevance, specifically whether the generated explanation clearly describes the relationship between the user query and the retrieved paper metadata and provides the sufficient justification. The evaluator assigns a score between 1 and 3, where 3 indicates a clear explanation with supporting evidence, 2 indicates an explanation that identifies the relationship but lacks sufficient detail or justification, and 1 indicates an explanation that does not clearly justify the relationship or contains unsupported claims.

The final AspectCritic-based explanation relevance score was obtained by averaging the scores assigned to all evaluated explanations, as shown in Equation (14):

$$AspectCritic={\displaystyle \frac{1}{K}}{\displaystyle \sum _{i=1}^K}{s}_i$$

(14)

where $s_i$ represents the score assigned by the LLM-based evaluator for explanation i, and K represents the total number of evaluated explanations.

All evaluations in this study use the top-10 papers for each query because users mainly examine the highest-ranked results. For evaluating relevance assessment and ranking performance, Gwet’s AC2 and NDCG use the same human expert annotations as the reference standard for comparison with system outputs, but evaluate different aspects of system performance. Gwet’s AC2 measures the agreement between the relevance categories assigned by the LLM and human expert judgments, whereas NDCG evaluates the ranking quality based on the ordering of papers according to human expert relevance labels.

5.2. Performance Comparison with Our Previous System

The enhanced system and the previous system were evaluated using the same 20 queries from five academic fields. The comparison covers relevance assessment agreement, paper ranking quality, and generated explanation quality. The statistical significance of the observed improvements was evaluated based on the scores obtained from each query. Since both systems were evaluated using the same query set, the paired Wilcoxon signed-rank test was applied to compare the metric scores. An ablation study was also conducted to analyze the contribution of different system components in the enhanced system.

5.2.1. Retrieved Paper Collection

Table 13 shows the average number of papers retrieved for each academic field. The averages range from 2764 papers in MHS to 3423 papers in LSH.

Table 13. Average number of retrieved papers across five academic fields.

Academic Field	Average Retrieved Papers
CSE	3319
MHS	2764
MNS	3013
EM	3315
LSH	3423

The retrieved paper collections contained thousands of candidate papers across the five academic fields. These collections were used as input for evaluating the enhanced system and our previous system in assessing paper relevance.

For evaluation, the top-10 retrieved papers from each query were selected, resulting in a maximum of 200 query–paper pairs from 20 queries. The relevance of each paper was assessed based on its relationship with the corresponding query rather than considering unique papers only. When the same paper appeared in multiple queries, it was evaluated separately according to each query context. For comparisons between the enhanced system and our previous system, overlapping papers were retained and evaluated based on their corresponding queries to measure the relevance assessment and ranking performance under the same query conditions.

5.2.2. Relevance Assessment Agreement

To evaluate H1, the agreement between the system-generated relevance categories and human expert annotations was measured using Gwet’s AC2. Figure 10 compares the average scores of the enhanced system and our previous system across the five academic fields, where the displayed values are rounded for presentation and the error bars represent the standard deviation of the scores within each academic field. The reported average Gwet’s AC2 score was calculated by first averaging the scores across the top-10 papers for each query, then averaging the query scores within each academic field, and finally averaging the results across the five academic fields using values before rounding.

Figure 10. Comparison of Gwet’s AC2 scores with standard deviation error bars.

The average Gwet’s AC2 score increased from 0.617 in the previous system to 0.813 in the enhanced system, corresponding to a relative improvement of 31.77%.

The statistical significance of the performance difference was evaluated using the paired Wilcoxon signed-rank test based on the scores obtained from the same 20 queries. The result showed a significant difference between the two systems, with higher Gwet’s AC2 scores achieved by the enhanced system ($W=4.000$, $p=0.000386<0.001$). These results support H1, indicating that the enhanced system achieved better agreement with human expert relevance judgments than the previous system.

5.2.3. Paper Ranking Quality

To evaluate H2, paper ranking quality was measured using NDCG based on the ideal ranking derived from human expert annotations. Figure 11 compares the average scores of the enhanced system and our previous system across the five academic fields, where the displayed values are rounded for presentation and the error bars represent the standard deviation of the scores within each academic field. The reported average NDCG score was calculated by first calculating the NDCG score for each query, then averaging the query scores within each academic field, and finally averaging the results across the five academic fields using values before rounding.

Figure 11. Comparison of NDCG scores with standard deviation error bars.

The average NDCG score increased from 0.852 in the previous system to 0.948 in the enhanced system, corresponding to a relative improvement of 11.27%.

The statistical significance of the performance difference was evaluated using the paired Wilcoxon signed-rank test based on the scores obtained from the same 20 queries. The result showed a significant difference between the two systems, with higher NDCG scores achieved by the enhanced system ($W=11.000$, $p=0.000724<0.001$). These results support H2, demonstrating that the enhanced system improved the ranking quality of retrieved papers compared with the previous system.

5.2.4. Generated Explanation Quality

To evaluate H3, the generated explanations were evaluated using RAGAS Faithfulness and a custom AspectCritic-based explanation relevance evaluation. The retrieved paper metadata provided to the LLM included the paper title, abstract, keywords, discipline/subject fields, and the relevance category rather than the full paper content. The evaluation was conducted on the generated explanations from the top-10 retrieved papers for each query, resulting in 200 evaluated explanations from 20 queries. Table 14 presents the scores across the five academic fields. The ± values represent the standard deviation calculated from the evaluated explanations within each academic field. The average values were calculated from all evaluated explanations across the five academic fields.

Table 14. Evaluation of LLM-generated relevance explanations.

Academic Field	Faithfulness	AspectCritic-Based Explanation Relevance
CSE	0.691 ± 0.067	2.70 ± 0.48
MHS	0.727 ± 0.066	2.93 ± 0.15
MNS	0.663 ± 0.117	2.63 ± 0.38
EM	0.727 ± 0.117	2.80 ± 0.23
LSH	0.696 ± 0.090	2.60 ± 0.18
Average	0.701 ± 0.091	2.73 ± 0.28

The average Faithfulness score was 0.701 on a 0–1 scale, while the average AspectCritic-based explanation relevance score was 2.73 on a 1–3 scale. These results support H3, indicating that the generated explanations were generally consistent with the retrieved paper metadata provided to the LLM and appropriately described the relationship between the user query and retrieved papers with sufficient justification.

5.2.5. Ablation Study

To analyze the contribution of key components in the enhanced system, an ablation study was conducted using three configurations: (1) MPNet-based reranking, (2) BM25 combined with MPNet reranking, and (3) BM25+MPNet with LLM-based relevance assessment.

The ablation results in Table 15 show the contribution of key components in the enhanced system. Adding BM25 retrieval to MPNet improves the average Gwet’s AC2 score from 0.617 to 0.729 and the average NDCG score from 0.852 to 0.913. Furthermore, adding the LLM-based relevance assessment stage further increases these scores to 0.813 and 0.948, respectively. These results indicate that the improvement is obtained from the combined contribution of retrieval, reranking, and LLM-based assessment components rather than from the LLM component alone.

Table 15. Ablation study based on Gwet’s AC2 and NDCG scores.

Academic Field	MPNet		BM25 + MPNet		BM25 + MPNet + LLM
	Gwet’s AC2	NDCG	Gwet’s AC2	NDCG	Gwet’s AC2	NDCG
CSE	0.606 ± 0.087	0.776 ± 0.139	0.714 ± 0.153	0.849 ± 0.155	0.791 ± 0.186	0.959 ± 0.029
MHS	0.702 ± 0.304	0.944 ± 0.047	0.799 ± 0.171	0.960 ± 0.052	0.904 ± 0.065	0.971 ± 0.040
MNS	0.531 ± 0.346	0.865 ± 0.023	0.472 ± 0.401	0.921 ± 0.072	0.681 ± 0.239	0.978 ± 0.017
EM	0.716 ± 0.143	0.862 ± 0.092	0.871 ± 0.074	0.900 ± 0.058	0.878 ± 0.114	0.922 ± 0.084
LSH	0.529 ± 0.271	0.814 ± 0.075	0.791 ± 0.171	0.932 ± 0.054	0.811 ± 0.111	0.908 ± 0.065
Average	0.617 ± 0.230	0.852 ± 0.075	0.729 ± 0.194	0.912 ± 0.078	0.813 ± 0.143	0.948 ± 0.047

5.3. Performance Comparison with Standard Neural Cross-Encoder Models

The enhanced system was further compared with the cross-encoder/ms-marco-MiniLM-L6-v2 model, a standard neural cross-encoder model for query-document relevance scoring [57]. This comparison evaluates the ranking performance of the proposed GenAI-based relevance assessment approach against a neural reranking baseline that directly estimates the relevance relationship between queries and paper metadata.

Since the cross-encoder produces continuous relevance scores in the form of raw logits rather than the predefined relevance categories used in the enhanced system, the comparison was conducted based on paper ranking quality using NDCG for the top-10 retrieved papers of each query across the five academic fields.

The comparison results are presented in Table 16. The enhanced system achieved a higher average NDCG score than the cross-encoder/ms-marco-MiniLM-L6-v2 model, with scores of 0.948 and 0.908, respectively. The enhanced system obtained higher scores in four out of five academic fields, while the cross-encoder model achieved a slightly higher score in the LSH field.

Table 16. Comparison of NDCG with the standard neural cross-encoder model.

Academic Field	Enhanced System	cross-encoder/ms-marco-MiniLM-L6-v2
CSE	0.959 ± 0.029	0.880 ± 0.110
MHS	0.971 ± 0.040	0.870 ± 0.060
MNS	0.978 ± 0.017	0.970 ± 0.030
EM	0.922 ± 0.084	0.890 ± 0.100
LSH	0.908 ± 0.065	0.930 ± 0.120
Average	0.948 ± 0.047	0.908 ± 0.084

These results show that the proposed GenAI-based relevance assessment approach provides the competitive ranking performance compared with the standard neural cross-encoder model. The improvement may be related to the use of predefined relevance criteria and category assignment rules during the LLM-based assessment process, which allows the system to consider the relationship between the query requirements and the retrieved paper metadata. In contrast, the cross-encoder model estimates relevance based on the query-document interaction learned from its training data without using task-specific relevance criteria.

5.4. Discussion

In this subsection, we discuss the findings of the proposed enhanced system, including its implications, remaining issues, error analysis, limitations, and code availability.

5.4.1. Interpretation of Relevance Assessment Agreement

The enhanced system achieved higher Gwet’s AC2 scores than our previous system across all five academic fields. This improvement indicates that the semantic similarity alone is insufficient for the paper relevance assessment. Sentence embedding methods can identify papers with the similar semantic information, but they do not directly evaluate whether a paper satisfies the relevance criteria related to the user’s research domain, subject field, research topic, research focus, and research method. In this study, the two-stage candidate paper selection using BM25 and MPNet improves the candidate paper selection, while the LLM performs the relevance assessment using the predefined criteria derived from human expert evaluations. Therefore, this combination results in relevance categories that are closer to human expert judgments.

5.4.2. Interpretation of Paper Ranking Quality

The enhanced system also achieved higher NDCG scores. The improvement indicates that ranking based only on semantic similarity is insufficient because papers with similar terminology may have different research objectives and relevance levels. The BM25 and MPNet stages first reduce the candidate collection by retaining papers with strong lexical and semantic matches. Then, the LLM reviews the selected papers using the relevance criteria and assigns them to four graded categories. By combining these approaches, the system can consider both the textual similarity and contextual relevance, allowing more relevant papers to be ranked at higher positions.

5.4.3. Interpretation of Generated Explanation Quality

Faithfulness and AspectCritic-based relevance evaluation scores indicate that most claims in the explanations generated by the LLM are supported by the available paper metadata and provide sufficient justification for the assigned relevance categories. This quality may be attributed to the use of structured paper metadata and predefined relevance criteria during the LLM-based assessment process. The generated explanations reflect the retrieved paper metadata provided to the LLM rather than the full paper content. Unlike the previous system, which only provided similarity scores and relevance categories, the enhanced system also provides explanations that help users understand the basis of each relevance decision before reading the full paper.

5.4.4. Remaining Issues

Although the enhanced system improved relevance assessment and ranking performance, some differences remained across academic fields. These differences indicate that relevance assessment is still affected by variations in terminology, research objectives, and the level of information available in paper metadata. In addition, the quality of candidate papers selected by the retrieval stage may influence the final relevance assessment.

5.4.5. Error Analysis

To analyze the remaining limitations of the enhanced system, several incorrect classification cases were examined by comparing the LLM-based assessment with human judgments.

The analysis was conducted on a query related to an open language model for mathematics, with keywords including Artificial Intelligence, Mathematics, Natural Language Processing, Large Language Model, and Linear Algebra. Table 17 presents examples where the system assigned higher relevance categories due to overlapping concepts between the query and paper content, while human experts provided lower relevance judgments based on differences in research focus.

Table 17. Examples of incorrect classification cases.

Paper Title	LLM Label	Human Label	Possible Cause
Mathlearner: A Large Language Model Agent Framework for Learning to Solve Mathematical Problems	Highly relevant	Moderately relevant	The paper focuses on improving mathematical reasoning using an LLM agent, but it does not fully represent the intended research focus.
Synthetic Data Enhances Mathematical Reasoning of Language Models Based on Artificial Intelligence	Highly relevant	Moderately relevant	The paper discusses mathematical reasoning and LLM development, but its main focus is synthetic data generation and model fine-tuning.
Atlas: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning	Highly relevant	Somewhat relevant	The paper evaluates scientific reasoning capabilities of LLMs, but it mainly focuses on benchmark development.
Exploring The MIT Mathematics and EECS Curriculum using Large Language Models	Moderately relevant	Somewhat relevant	The paper is related to mathematics education using LLMs, but its main objective is curriculum evaluation.

Note: Italics indicate the abbreviation for large language model(s) (LLM/LLMs).

These cases show that the LLM can identify relationships between query terms and paper content, but differences in research objectives may still lead to disagreement with human expert judgments.

5.4.6. Limitations of the Enhanced System

Several limitations of the proposed system should be considered. These limitations are related to the relevance assessment process, study quality assessment, evaluation settings, and the use of LLMs. The following limitations also provide directions for future research.

Performance varies across academic fields, particularly in the MNS field, where specialized terminology, narrow topics, and abstract concepts may be difficult for the LLM to assess consistently. The assessment also depends on predefined relevance criteria, which may not represent every search objective. In addition, the system uses paper metadata rather than full paper content, so some details required for relevance assessment may be missing. Future studies should examine whether using the full paper content and field-specific relevance criteria can improve the assessment consistency, particularly for specialized research topics.

The current system combines semantic similarity with publication reputation indicators for paper ranking and uses predefined relevance criteria and category assignment rules in the LLM prompt to assess paper relevance. However, these approaches do not provide a detailed evaluation of study quality. Such an evaluation requires examining the full paper content, including research motivation, hypotheses, research design, datasets, experimental procedures, data analysis, evaluation metrics, and validity of findings. Future research should investigate how study quality assessment can be added to the system alongside relevance assessment.

Preprint repositories may contain studies that have not undergone formal peer review, so their findings may be changed after further evaluation or publication. Since multiple versions of the same paper may exist with differences in content and metadata, the enhanced system selects the latest available preprint version and provides publication status information. However, the current system cannot identify links between preprints and their corresponding published journal articles. As the system focuses on paper relevance rather than scientific quality, a highly relevant preprint may still require further review before being used as a research reference. Future studies should investigate methods for linking preprints to their published versions and updating publication status information to help users identify changes in paper versions.

The relevance criteria and category assignment rules were formulated based on human expert evaluations from the previous system. Since the same set of research queries was used for the final evaluation, the prompt construction and system evaluation were not completely separated. This may influence the reported performance because the assessment framework was developed from information obtained during the previous evaluation process. Future studies should use separate query sets for prompt development and final testing to provide a more independent evaluation of system performance.

The evaluation was conducted using 20 queries from five academic fields, which may limit the generalization of the conclusions to broader research domains. Although statistical significance testing was performed using the paired Wilcoxon signed-rank test, evaluations with larger and more diverse query collections are required to further examine the robustness of the enhanced system. Future studies should include more queries, academic fields, and research topics to examine whether the observed performance differences remain consistent across different evaluation settings.

The candidate selection stages use fixed cutoff values, where the top-100 papers are retained after BM25-based filtering and the top-50 papers are forwarded to the LLM-based relevance assessment. These values were selected as practical settings to balance candidate reduction and processing cost. However, different cutoff values and their effects on retrieval performance were not specifically evaluated. Future studies should investigate these settings using retrieval-oriented metrics, such as Recall@50 and Recall@100, based on human relevance annotations. This would help determine whether different cutoff values provide a better balance between candidate coverage and processing cost.

Although the enhanced system uses the open-weight Mistral-Small-24B-Instruct-2501 model accessed through the OpenRouter API, possible overlap between publicly available scientific papers and the model’s pre-training data cannot be completely excluded. Future evaluations using temporally separated datasets or newly published papers are required to further examine this issue and assess whether potential overlap affects the reported results.

5.4.7. Code Availability

The source code of the proposed enhanced reference paper collection system is available at https://github.com/tresnamf/arpacs-genai (accessed on 21 September 2026).

6. Conclusions

In this paper, we proposed an enhanced reference paper collection system that integrates Generative Artificial Intelligence (GenAI) into paper relevance assessment. The system retrieves papers from multiple academic sources, reduces the candidate pool using BM25 and MPNet, and applies an LLM with predefined relevance criteria derived from human expert evaluations to assign four relevance categories and generate short explanations. The evaluation involved 20 queries from five academic fields with human expert annotations as the reference standard. Compared with our previous system, the enhanced system improved relevance assessment agreement by 31.77% and paper ranking quality by 11.27%. In addition, the generated explanations achieved a score of 0.701 in Faithfulness and 2.73 in the AspectCritic-based explanation relevance evaluation, indicating that the generated explanations were generally supported by the provided paper metadata and described the relationship between the user query and retrieved papers. In future work, we will investigate domain-specific fine-tuning to improve relevance assessment for fields with specialized terminology and narrow research topics. We will also consider using full paper content to provide more complete relevance assessments and support study quality assessment. Further studies will examine different paper selection strategies, broader evaluation settings, and different LLM models to examine their effects on the relevance assessment process and system performance.