A Comparative Analysis of Sentence Transformer Models for Automated Journal Recommendation Using PubMed Metadata

Abstract

We present an automated journal recommendation pipeline designed to evaluate the performance of five Sentence Transformer models—all-mpnet-base-v2 (Mpnet), all-MiniLM-L6-v2 (Minilm-l6), all-MiniLM-L12-v2 (Minilm-l12), multi-qa-distilbert-cos-v1 (Multi-qa-distilbert), and all-distilroberta-v1 (roberta)—for recommending journals aligned with a manuscript’s thematic scope. The pipeline extracts domain-relevant keywords from a manuscript via KeyBERT, retrieves potentially related articles from PubMed, and encodes both the test manuscript and retrieved articles into high-dimensional embeddings. By computing cosine similarity, it ranks relevant journals based on thematic overlap. Evaluations on 50 test articles highlight mpnet’s strong performance (mean similarity score 0.71 ± 0.04), albeit with higher computational demands. Minilm-l12 and minilm-l6 offer comparable precision at lower cost, while multi-qa-distilbert and roberta yield broader recommendations better suited to interdisciplinary research. These findings underscore key trade-offs among embedding models and demonstrate how they can provide interpretable, data-driven insights to guide journal selection across varied research contexts.

1. Introduction

Selecting the right journal for a scientific manuscript remains a critical step for researchers, regardless of their experience level [1]. A misaligned journal choice can lead to delayed publication, immediate rejection, and reduced visibility within the most relevant research communities [2]. Even seasoned authors sometimes grapple with overlapping scopes and emerging journal niches, while early-career researchers or those working on cross-disciplinary topics often lack clear criteria for identifying the most suitable outlets. The continuous creation of new journals and the evolution of established ones make it even more difficult for authors to keep pace with this shifting array of publication venues [3]. These issues make the case for data-driven systems that can automate and streamline the journal selection process, thereby reducing guesswork, saving time, and potentially improving the likelihood of a favorable peer-review outcome [4].

Historically, automated journal recommendation systems have relied on traditional information retrieval and machine learning techniques [5]. Early approaches, such as bag-of-words models and term frequency–inverse document frequency (TF–IDF [6]), primarily matched manuscripts to journal scopes based on term co-occurrences and document similarity. For instance, Wang et al. [7] applied softmax regression with TF–IDF-based feature selection to recommend computer science journals, demonstrating modest accuracy. Other studies employed n-gram classification for textual analysis [8] or combined stylometric features with collaborative filtering [9] to identify suitable journals. However, these methods often struggled to capture the semantic richness of scientific texts, especially in fields with rapidly evolving jargon or interdisciplinary overlap.

Various commercial platforms, including those offered by well-known publishing companies such as Springer and Elsevier, aim to assist authors in selecting journals. These tools typically rely on keyword-based matching or limited datasets, which can restrict their applicability across diverse research areas. Beel et al. [10] highlighted key limitations of these platforms, such as their reliance on proprietary data, lack of transparency in recommendation algorithms, and inability to effectively handle nuanced linguistic variations or semantic relationships.

The advent of Transformer-based language models, such as BERT and its variants [11,12], has revolutionized natural language processing by capturing more nuanced, context-sensitive relationships among words. Sentence Transformer models, specifically designed for semantic similarity tasks, have emerged as powerful tools for comparing scientific texts [13]. These models generate dense semantic embeddings that preserve meaningful relationships in a continuous vector space, enabling robust similarity matching even when different terms describe the same concept [14]. Applications of such embeddings range from semantic search and clustering to question answering and recommender systems [13].

In journal recommendation tasks, Transformer embeddings offer significant advantages over traditional methods. Recent studies, including Michail et al. [15], have shown that models such as Sentence Transformers can outperform earlier techniques by encoding article abstracts into high-dimensional vectors and comparing them via cosine similarity. These approaches have proven effective across a variety of domains, from computer science to biomedicine, where capturing the thematic nuances of a manuscript is essential for identifying appropriate journals [14].

Despite this potential and all the pragmatic hurdles described above, a key question remains: Which Sentence Transformer model best captures domain-specific semantic nuances for journal recommendation [16]? Existing models differ in training objectives, corpora, and parameter settings [17]. Domain-general models—such as all-mpnet-base-v2—often excel at broad semantic similarity, although they are not specifically tuned to scientific literature. Alternatively, compact models like all-MiniLM-L6-v2 promise faster runtime but may sacrifice some representational depth. Given these trade-offs, a comparative study of multiple models is needed to determine which embedding approach yields the most accurate, context-sensitive journal recommendations in a biomedical setting.

To address this issue, we propose an automated journal recommendation pipeline for the life sciences and systematically evaluate five Sentence Transformer models: all-mpnet-base-v2 [18], all-MiniLM-L6-v2 [19], all-MiniLM-L12-v2 [20], multi-qa-distilbert-cos-v1 [21], and all-distilroberta-v1 [22]. We started with test articles, which we used to extract keywords through KeyBERT [23]. We then queried PubMed [24] with these KeyBERT-derived keywords to retrieve custom corpora of potentially related articles. We then encoded the titles and abstracts of these articles and the test manuscripts into embeddings, calculated their cosine similarity scores, and generated a data-driven list of recommended journals. In addition to comparing the models’ semantic alignment, we assessed their computational efficiency, domain coverage, and alignment with expert-driven expectations.

2. Materials and Methods

2.1. Data Collection

We used NCBI E-utilities to query PubMed for articles potentially relevant to each test manuscript within the 2020–2024 timeframe. Specifically, KeyBERT identified three salient keywords from the manuscript’s title and abstract (see Section 2.2), which we combined with the logical operator AND to perform domain-relevant searches. PubMed often returned up to 20,000 PMIDs for a single query; to maintain computational feasibility, we randomly sampled 5000 articles when more than 5000 PMIDs were retrieved. For each retrieved PMID, we used Biopython’s Entrez.efetch to download metadata (Title, Abstract, Journal) and stored these fields in a pandas DataFrame [25]. The test article’s own PMID was excluded to prevent data leakage.

2.2. Preprocessing and Keyword Extraction

For each test article, we extracted its title and abstract. A keyword extraction step was performed using KeyBERT to identify three salient keywords representing the article’s core content. KeyBERT internally uses pre-trained Transformer embeddings to generate vector representations of both the entire document (title + abstract) and candidate N-grams extracted from the text. It then computes cosine similarity between the document embedding and these candidate N-gram embeddings, selecting the top-scoring ones as the most representative keywords. We chose KeyBERT for its simplicity and transparency, as it readily reveals which terms best capture a manuscript’s thematic essence. These keywords guided PubMed queries to build a dynamically tailored pool corpus for each test article. The number of keywords (n = 3) was empirically chosen to capture the core focus of the manuscript while avoiding overly narrow or excessively broad queries.

2.3. Test Article Selection and Evaluation Set

We selected 50 test articles to provide a varied yet tractable dataset for evaluating model performance. These articles span multiple biomedical and life sciences topics, reflecting both clinical and laboratory research published between 2020 and 2024. While this choice focuses on recent scientific discourse, we acknowledge that it may exclude older publications, potentially introducing a bias if journal scopes or key terms have evolved. Additionally, random sampling from PubMed could overlook niche or less frequently indexed research.

2.4. Embedding Models

We employed five Sentence Transformer models to encode textual data into high-dimensional vector embeddings: all-mpnet-base-v2 (mpnet), all-MiniLM-L6-v2 (minilm-l6), all-MiniLM-L12-v2 (minilm-l12), multi-qa-distilbert-cos-v1 (multi-qa-distilbert), and all-distilroberta-v1 (roberta). For each article, we concatenated its title and abstract into a single string and used the model.encode() method to generate embeddings. All embeddings were computed on Google Colab GPUs.

2.5. Similarity Computation

Cosine similarity was calculated between the test article embedding and all embeddings in the pool corpus. Cosine similarity is defined as

$$\mathrm{cosine}\_\mathrm{similarity}(\mathrm{u}, \mathrm{v})= {\displaystyle \frac{\mathrm{u}•\mathrm{v}}{\parallel \mathrm{u}\parallel \parallel \mathrm{v}\parallel }}$$

where u and v are high-dimensional embedding vectors. Cosine similarity is particularly well-suited for comparing sentence embeddings because it is scale-invariant, focusing on the angle between vectors rather than their magnitudes. This property makes it robust when embeddings differ in length or magnitude but still capture similar semantic content. Moreover, cosine similarity is widely recognized and interpreted in the NLP community, with values ranging from −1 to +1. By contrast, distance metrics such as Euclidean or Manhattan distance can be more sensitive to absolute vector magnitudes and do not necessarily provide an intuitive measure of semantic relatedness in high-dimensional spaces. Using PyTorch’s 2.6 torch.topk [26], we thus identified the top 10 most similar articles for each model, recording their associated PMIDs and journals.

Alternatively, articles were grouped by journal, and a mean embedding was calculated for each journal by averaging the embeddings of all articles belonging to that journal. The cosine similarity between the test article embedding and each journal’s mean embedding was then computed, yielding a ranked list of journals by average thematic alignment.

2.6. Evaluation Metrics

We further quantified the diversity and concentration of recommendation scores using two complementary metrics: Shannon entropy and the Gini coefficient.

2.6.1. Shannon Entropy

Shannon entropy is widely used in information theory to measure the diversity or “spread” of a distribution [27]. We applied Shannon entropy to the cosine similarity scores of the top recommended articles for each test case, aiming to capture how evenly the model distributed its top similarity scores among different articles or journals. Formally, for a given model M and a test article t, let {p_i} be the normalized similarity scores (so that ∑_ip_i = 1) of the top N recommended items, where p_i = ${\displaystyle \frac{Si}{\sum jSj}}$ and s_i is the raw similarity for item i. Shannon entropy was calculated using the following formula [28]:

$$\mathrm{H}(\mathrm{X})=-{\sum }_{i=1}^n \mathrm{p}\left({\mathrm{x}}_{\mathrm{i}}\right)\mathrm{l}\mathrm{o}{\mathrm{g}}_2 \mathrm{p}\left({\mathrm{x}}_{\mathrm{i}}\right)$$

A higher entropy indicates a more even distribution of similarity scores—i.e., no single article or journal dominates—while a lower entropy suggests one or a few items have much higher similarity scores than the rest.

2.6.2. Gini Coefficient

The Gini coefficient is commonly used in economics to assess inequality within a distribution. We employed it here to measure how concentrated the similarity scores are among top-ranked articles or journals. Let {x₁,x₂,…,x_n} represent the raw similarity scores of the top n recommendations, sorted in non-decreasing order. Following the procedure in Dorfman [29], we compute the Gini coefficient G as

$$\mathrm{G}={\displaystyle \frac{{\sum }_{i=1}^n{\sum }_{j=1}^n|x_i-x_j|}{2n{\sum }_{i=1}^n{x}_i}}$$

A G value close to 0 indicates that the model assigns relatively comparable similarity scores across the top recommendations, suggesting a more uniform distribution. By contrast, a G value nearer to 1 implies that a few recommendations dominate, resulting in greater concentration. In other words, a lower Gini coefficient denotes more balanced scoring, while a higher value signals that the model focuses its similarity on a small subset of items. We use the Gini coefficient alongside Shannon entropy to gain deeper insight into how each model distributes its top similarity scores and whether it concentrates recommendations in just a few items or spreads them more evenly.

2.7. Software and Environment

All code was implemented in Python (version 3.10) within a Google Colab environment [30]. The following primary libraries were used:

• Biopython (v1.80) for PubMed queries [31];
• pandas (v1.5+) for DataFrame manipulation [25];
• numpy (v1.23+) and torch (v1.13+) for numerical and tensor operations [32,33];
• sentence-transformers (v2.2+) for encoding text with pre-trained Transformer models;
• seaborn (v0.12+) and matplotlib (v3.6+) for data visualization [34,35].

All code was executed under Google Colab T4 GPU runtime configurations, respecting NCBI E-utilities usage policies (including Entrez.email specification and rate-limiting queries).

2.8. Study Focus and Limitations

Although our pipeline can suggest thematically relevant journals, it does not claim to offer a definitive or exhaustive publication venue recommendation. We acknowledge that real-world journal selection often involves considerations beyond thematic alignment—such as impact factor, review process, and editorial scope. Our results thus solely reflect each model’s ability to identify plausible journal options based on content similarity alone, serving as a starting point for further exploration.

3. Results

3.1. Case Study

In this study, we implemented a pipeline designed to recommend journals for a single test article by dynamically generating a corpus of potentially relevant PubMed articles. Our approach comprised four major steps: (1) extracting keywords from the test article’s title and abstract using KeyBERT, (2) issuing an ad hoc PubMed query to gather a custom pool of papers, (3) encoding the resulting pool with Sentence Transformers to create high-dimensional embeddings, and (4) computing similarity between the test article embedding and the pool embeddings.

To this purpose, we set out to evaluate and compare the recommendations produced by five Sentence Transformer models: all-mpnet-base-v2, all-MiniLM-L6-v2, all-MiniLM-L12-v2, multi-qa-distilbert-cos-v1, and all-distilroberta-v1 (

Table 1. This table provides a summary of the main characteristics of the five different Sentence Transformer models used for the present study.

Model	Embedding Dimension	Approx. Parameter Count	Pretraining/Fine-Tuning Corpus	Pooling Strategy
mpnet	68	~110 M	Trained on a >1 B sentence dataset and large-scale general text corpora using the MPNet architecture.	Mean pooling
minilm-l6	384	~22 M	Distilled from a teacher model on a >1 B paraphrase dataset, emphasizing efficiency with fewer layers (six layers).	Mean pooling
minilm-l12	384	~33 M	Distilled similarly to minilm-l6 but retains more layers (12 layers), trained on the same broad paraphrase/entailment dataset.	Mean pooling
multi-qa-distilbert	768	~66 M	Fine-tuned on 215 M QA datasets to produce universal question-answer embeddings.	Mean pooling
roberta	768	~82 M	Based on DistilRoBERTa-base, further tuned on a large paraphrase corpus (>1 B sentence pairs) for enhanced sentence embedding.	Mean pooling

).

To get a general overview of the performance of these models, we randomly chose a test article from a corpus of 10,000 scientific articles randomly retrieved from PubMed: “Problems of Clostridium difficile infection (CDI) in Polish healthcare units”, published in Annals of Agricultural and Environmental Medicine (AAEM) [36].

We began by applying KeyBERT, an unsupervised keyword extraction tool, to the article’s title and abstract. We thus identified the three most representative keywords describing the article’s core content.

Based on these three keywords—clostridium, infection, and difficile—we conducted a PubMed query to locate potentially relevant articles by joining the keywords with the AND operator. The number of keywords (n = 3) was empirically determined to retrieve a sufficiently large corpus without excessively restricting the query. PubMed returned 15,466 candidate articles matching the query “clostridium AND infection AND difficile”. For computational feasibility, we randomly selected 5000 articles from among those 15,466. We then ensured that our test article’s own PMID was not included in this pool, so as not to artificially inflate similarity scores.

Once we generated the 5000-article corpus, we extracted the title, abstract, and journal fields from each article using Biopython’s Entrez utilities. The goal was to measure two distinct but complementary aspects of recommendations:

• Article-Level Similarity: For each model, we sought the top ten articles (out of those 5000) most similar to the test article and identified the journals where they were published.
• Journal-Level Similarity: We computed the average similarity of each journal’s articles to the test article, and then ranked the journals accordingly.

To ensure consistency, each model was applied systematically. First, the model encoded all 5000 articles, concatenating their titles and abstracts into a single text input per article, and transformed the text into embeddings. Next, the same model encoded the test article’s title and abstract. By comparing each article’s embedding with the test article’s embedding using cosine similarity, we obtained a comprehensive list of similarity scores. The article with the highest similarity score was treated as the closest article; its journal can be found in

Table 2. This table provides a summary of journal recommendations generated by five different Sentence Transformer models when applied to the test article. “False” indicates that the actual journal in which the test article was published is absent from the model’s top 10 recommended journals.

Model	Top Article	Top Avg Journal	Avg Journal Similarity	Journal in Top 10	Encoding Time
minilm-l12	Journal of Preventive Medicine and Hygiene	International journal of environmental research and public health	0.852531	False	9”
minilm-l6	Journal of Hospital Infection	Risk management and healthcare policy	0.848661	False	9”
mpnet	International Journal of Environmental Research and Public Health	International journal of environmental research and public health	0.926116	False	98”
multi-qa-distilbert	European Review for Medical and Pharmacological Sciences	Risk management and healthcare policy	0.858523	False	54”
roberta	Journal of Hospital Infection	International journal of environmental research and public health	0.886216	False	52”

(in the Top Article column). Simultaneously, we grouped articles by their respective journals and computed a mean embedding per journal by averaging the embeddings of all articles belonging to that journal. A second round of similarity checks identified the journal with the highest average similarity to the test article (column Top Avg Journal in Table 2).

A preliminary comparison of computation time shows the great disparity between mpnet, which required a minute and a half to encode all the 5000 titles and abstracts of the dataset, and the minilm-l6 and minilm-l12 models, which only took 9 s to encode all 5000 titles and abstracts.

3.1.1. Article-Level Findings

Each model produced a list of ten articles deemed most semantically similar to the test article. A list of the journals where these articles appeared can be found in Supplementary Table S1. In general, all models identified articles published in journals that focused on infectious diseases, hospital-acquired infections, or microbial pathogens, confirming that the basic textual content was indeed guiding the retrieval. As shown in Table S1, mpnet consistently produced the highest similarity scores across its top ten recommendations, with articles often exceeding 0.85, and multiple entries from journals such as the International Journal of Environmental Research and Public Health and the Journal of Preventive Medicine and Hygiene. “Hospital management of Clostridium difficile infection: a review of the literature” [37] was mpnet’s suggestion as the closest article, with similarity = 90.62. The overall data suggest mpnet’s robustness in identifying journals strongly aligned with the test article’s thematic content, particularly those related to public health and infection control. Minilm-l6 and minilm-l12 exhibited overlapping trends in their recommendations, with journals such as the Journal of Hospital Infection and the International Journal of Environmental Research and Public Health frequently appearing in their top lists. However, minilm-l12 (top choice: “Clostridium difficile infection perceptions and practices: a multicenter qualitative study in South Africa” [38]) often returned higher maximum similarity scores than minilm-l6 (top choice: “The Burden of Clostridium Difficile (CDI) Infection in Hospitals, in Denmark, Finland, Norway And Sweden” [39]) for the same journal, suggesting that the additional model complexity of minilm-l12 might provide a marginally better thematic alignment.

The multi-qa-distilbert model mostly aligned with the results of the other models (top choice: “Ten-year review of Clostridium difficile infection in acute care hospitals in the USA, 2005–2014” [40]), although it also diverged from them by identifying broader healthcare-related journals, such as the European Review for Medical and Pharmacological Sciences and the Bulletin of Mathematical Biology. Despite this, multi-qa-distilbert consistently included journals like the Journal of Hospital Infection, indicating its capacity to recognize core infectious disease topics within its broader thematic spectrum. The roberta model demonstrated a strong performance, particularly with journals addressing clinical microbiology and infection control, such as the Journal of Hospital Infection (87.07 similarity) and the International Journal of Environmental Research and Public Health. Noticeably, roberta identified the same top choice article as mpnet: “Hospital management of Clostridium difficile infection: a review of the literature” [37].

3.1.2. Journal-Level Suggestions

To get a broader view of the dataset, we calculated the average similarity of each journal’s articles to the test article. For every journal in the 5000-article pool, we averaged all embeddings belonging to that journal, then measured how close that average embedding was to the test article’s embedding. The results can be found in Supplementary Table S2. Mpnet consistently identified journals strongly aligned with infectious diseases, hospital-acquired infections, and antimicrobial resistance. Its top-ranked journals, such as the International Journal of Environmental Research and Public Health and the Journal of Preventive Medicine and Hygiene, demonstrated high average similarity scores, with values exceeding 92% for the leading journal, and over 86% similarity for Infection, Disease & Health, which ranks 10th and last for this model. This pattern confirms mpnet’s capability to prioritize thematically congruent journals with precise alignment.

Similarly, minilm-l12 showed a strong (but lower than mpnet’s and always lower than 86%) affinity for journals relevant to infection control and epidemiology, with its highest-ranked journal also being the International Journal of Environmental Research and Public Health, with 85.2% similarity. Notable overlap was observed between mpnet and minilm-l12, particularly in their top journal selections, which included titles such as the International Journal of Environmental Research and Public Health, the Journal of Preventive Medicine and Hygiene, the American Journal of Infection Control and Journal of Hospital Infection. This confirms that these models share a nuanced ability to identify core journals within the test article’s domain.

Minilm-l6, while maintaining a comparable thematic focus, included a broader range of journals, such as Emerging Infectious Diseases and Frontiers in Public Health. Its slightly lower average similarity scores compared to mpnet and minilm-l12 (always lower than 85%) highlight a broader distribution of relevance, potentially indicating a greater inclusion of interdisciplinary journals. Similarly, multi-qa-distilbert displayed a tendency toward multidisciplinary recommendations. Journals such as Risk Management and Healthcare Policy and the Journal of Global Health appeared prominently in its rankings. Finally, roberta showcased a balanced profile, with its top-ranked journal being the International Journal of Environmental Research and Public Health. While there was some overlap with the other models, such as its inclusion of Przeglad Epidemiologiczny and the Journal of Hospital Infection, roberta also included journals like the Journal of Global Health and Minerva Medica.

Figure 1 summarizes these findings, highlighting how the models identify two journals as their most probable candidates, the International Journal of Environmental Research and Public Health for mpnet, minilm-l12, and roberta, and Risk Management and Healthcare Policy for minilm-l6 and multi-qa-distilbert. Interestingly, none of the models correctly identified the actual journal where the test article was published.

3.1.3. Overlap Among Model Recommendations

To quantify the overlap in recommended journals, we calculated the set intersection and union of the top ten journals from each pair of models, as calculated according to the average similarity, and computed the ratio of the set intersection to the set union.

A heatmap (Figure 2) displayed the percentage overlap, where the color scale ranged from minimal overlap (~30%) to substantial overlap (~60–70%) for certain model pairs; mpnet and minilm-l12 exhibited notably higher overlap, with 66.7% of their recommended journals in common. Multi-qa-distilbert frequently diverged, sharing less than 35% with minilm-12 and mpnet, suggesting it captures text semantics or research topics differently. Minilm-l6 yielded intermediate high overlap percentages with roberta (66.7%) and also had a relatively high alignment with mpnet or minilm-l12 (53.8%).

3.2. Quantitative Comparison

We then extended our initial analysis of a single test article to encompass a broader evaluation involving 50 distinct test articles (Table S3), each randomly selected from the initial test dataset. This expansion aimed to provide a more generalized understanding of how various Sentence Transformer models performed in identifying thematically similar articles and recommending appropriate journals for publication. For each test article, we employed KeyBERT to extract the top three keywords, which were subsequently used to query PubMed, retrieving up to 20,000 PMIDs per query. To maintain computational efficiency and ensure manageable data processing, we randomly sampled 5000 PMIDs (if available) from each pool, carefully excluding the test article’s own PMID, if present. The extracted articles were then consolidated into a DataFrame, ensuring the removal of duplicate entries based on PMID, resulting in a final pool size of maximum 5000 unique articles per test case.

Each of the five Sentence Transformer models was systematically applied to encode both the pooled articles and the respective test articles. For every model, we identified the top 10 most similar articles based on these scores. Additionally, we aggregated the similarity scores at the journal level by computing the average similarity of articles within each journal to the test article, thereby ranking journals according to their thematic alignment with the test case.

The analysis yielded a comprehensive set of metrics for each model across all 50 test articles. Shannon entropy was calculated to assess the diversity of similarity scores among the top-ranked articles, with higher entropy values indicating a more even distribution of similarities and lower values suggesting dominance by one or a few highly similar articles. The maximum similarity score within the top 10 was recorded to measure the peak alignment each model achieved with the most similar article. Furthermore, we evaluated whether the actual journal of the test article was included within the top 10 recommended journals based on average similarity, providing a binary measure of each model’s effectiveness in correctly identifying the appropriate publication venue.

Aggregating the results across all test articles (

Table 3. This table summarizes the performance of five Sentence Transformer models when applied to 50 test articles. The columns include Top Article, the maximum similarity to an article; Max Similarity, the highest similarity score achieved by the model with the average journal similarity; Mean Sim, the mean similarity identified by the model across the 50 test articles; Shannon Entropy and Gini score, a measure of diversity in similarity scores among the top-ranked articles; Journal in Top 10 (%), i.e., the percentage of test articles for which the actual journal of publication appeared in the top 10 recommended journals.

Model	Top Article	MaxSimilarity	MeanSim	Shannon Entropy	Gini	Journal in Top 10
minilm-l12	0.799	0.765548	0.67 ± 0.04	3.24	0.036077	12.5
minilm-l6	0.775	0.746310	0.66 ± 0.04	3.24	0.034712	14.5
mpnet	0.831	0.794203	0.71 ± 0.04	3.24	0.029997	14.5
multi-qa-distilbert	0.755	0.717609	0.63 ± 0.04	3.24	0.034632	12.5
roberta	0.803	0.773727	0.68 ± 0.04	3.24	0.033623	12.5

), we observed that mpnet consistently demonstrated the highest mean similarity scores, averaging 0.71 ± 0.04, indicating a strong ability to identify journals closely aligned with the test articles’ themes. This was followed by roberta, with an average mean similarity of 0.68 ± 0.04, further showing its proficiency in aligning with relevant journals. The minilm-l12 and minilm-l6 models displayed comparable performance, achieving mean similarity scores of 0.67 ± 0.04 and 0.66 ± 0.04, respectively, reflecting a solid alignment with test articles but slightly lower than that of mpnet and roberta. Conversely, multi-qa-distilbert exhibited the lowest mean similarity score of 0.63 ± 0.04, indicating a more moderate alignment with the test articles.

Additionally, mpnet achieved the highest Top Article similarity score of 0.83, emphasizing its capacity to match individual articles very closely to the test articles’ themes. Roberta followed with a similarity of 0.80, while minilm-l12, minilm-l6, and multi-qa-distilbert achieved Top Article similarity scores of 0.79, 0.77, and 0.76, respectively.

The “JournalinTop10” metric revealed that mpnet and minilm-l6 were the most proficient in correctly identifying the actual journal of the test article within their top 10 recommendations, achieving inclusion rates of 14.5%. This was closely followed by minilm-l12, roberta, and multi-qa-distilbert, all of which had inclusion rates of 12.5%.

All models exhibited identical Shannon entropy values of approximately 3.24, suggesting that the similarity scores were evenly distributed across their recommendations. This low variability reflects a tendency to concentrate recommendations among a few top-ranked journals rather than distributing them broadly. The Gini coefficients were uniformly low across all models, with minilm-l12 showing the highest value of 0.0361, reflecting slightly higher concentration of similarity scores among its top recommendations. Mpnet and roberta followed closely, with Gini values of 0.03 and 0.0336, respectively, indicating a relatively even distribution of similarity scores across their top journals.

4. Discussion

This study employed a straightforward pipeline for automated journal recommendation, serving primarily as a means to evaluate the performance of five freely available Sentence Transformer models. Specifically, we tested all-mpnet-base-v2, all-MiniLM-L6-v2, all-MiniLM-L12-v2, multi-qa-distilbert-cos-v1, and all-distilroberta-v1. We examined each model’s ability to match manuscripts with thematically appropriate journals, using both a single randomly chosen test article and a larger set of 50 randomly selected articles. Our findings align with the notion that Transformer-based text embeddings are very effective in capturing semantic similarities between texts [13,14].

However, our pipeline operates under the assumption that an author’s primary criterion for journal selection is thematic alignment. The final decision on where to submit a manuscript often involves additional considerations. Factors such as journal impact factor, acceptance rate, publication costs, and the expected turnaround time for reviews frequently influence authors’ choices [41]. Moreover, authors might initially aim for more prestigious journals and use automated recommendations only after facing rejection from their preferred venues [42]. In our work, we purposefully isolated the scenario in which a researcher needs to identify journals that have recently published similar articles, thus highlighting each model’s strengths and weaknesses in semantic matching without conflating these results with more pragmatic concerns.

Our pipeline relied on Transformer-based embeddings, which are numerical representation of sentence semantics. Embeddings allow for easy comparison between the meaning of two or more sentences [43,44,45,46]. This pipeline evaluated both article-level and journal-level alignment, illustrating how focusing on individual articles can help authors gauge thematic closeness to previously published work within a given venue, whereas examining journals in aggregate reveals which publication outlets frequently address the subject matter in question.

Consistently with our previous findings [16], mpnet excelled in identifying closely aligned journals, offering the highest average similarity score (MeanSim: 0.71 ± 0.04) when challenged with 50 test articles [36,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95] and the highest maximum similarity (MaxSimilarity: 0.83) among our 50-test-article dataset. It also occasionally included the actual journal of publication in its top 10 suggestions (14.5% inclusion rate). Yet mpnet’s strong performance came with longer encoding times, which could pose challenges in settings requiring swift feedback. Minilm-l12 performed almost as well, reaching a MeanSim of 0.67 ± 0.04 and a MaxSimilarity of 0.79, but required substantially less computational time. Similarly, minilm-l6 struck a balance between speed and accuracy, attaining a MeanSim of 0.66 ± 0.04 and matching mpnet’s inclusion rate of 14.5% for the real publication venue. Its compact architecture (and that of minilm-l12) generates embeddings efficiently, suggesting potential utility in large-scale or real-time applications.

While mpnet yielded the highest domain-specific alignment, roberta also scored highly (MeanSim: 0.68), slightly above minilm-l12 (0.67). However, roberta included more interdisciplinary journals in its top recommendations, suggesting it might strike a balance between precision and broader thematic coverage. By contrast, multi-qa-distilbert displayed the lowest average similarity (0.63), reflecting its generalist nature and possibly broader, policy-related scope. This diversity might be advantageous for interdisciplinary manuscripts or exploratory research, where thematic breadth is prioritized over narrow precision; future research should address this issue to optimize the use of this model in the biomedical field.

We used Shannon entropy and the Gini coefficient to examine whether similarity scores were concentrated among a few top suggestions or evenly distributed. Minilm-l12 exhibited slightly higher Gini coefficients (0.036), implying more pronounced concentration at the top, whereas mpnet’s lower Gini values (0.029) signaled a relatively equitable spread of scores. These metrics reinforce that each model presents different trade-offs between pinpointing the most thematically aligned options and revealing a broader swath of related venues.

One notable outcome in our single-article case study was that none of the models identified the actual journal where that article was published. This finding highlights a limitation of our evaluation approach, particularly our use of the “JournalinTop10” metric to gauge success. Authors may choose journals for a variety of reasons that are not captured purely by thematic alignment: prestige, speed, editorial focus, and acceptance rates all have significant impacts. Accordingly, we caution that a model’s failure to recover the actual publication venue does not necessarily signal poor performance. The real journal choice might have hinged on non-semantic factors or happened to be absent from the sampled dataset. Going forward, these observations point to the need for more comprehensive recommendation strategies that integrate additional context, such as editorial policies, historical acceptance rates, or calls for special issues.

From a methodological standpoint, our use of KeyBERT for keyword extraction and random sampling of up to 5000 articles per test case demonstrates how a scalable NLP pipeline can be easily assembled with open-source tools. However, future iterations might integrate advanced query expansion or domain-specific ontologies like MeSH terms to capture a more exhaustive range of content. Adopting a more targeted or domain-focused search could mitigate any bias introduced by broader sampling, particularly in fields with rapidly changing terminologies.

In sum, this study demonstrates the strengths and limitations of Transformer-based models for automated journal recommendation. Mpnet stands out for high thematic precision but at a higher computational cost; multi-qa-distilbert and roberta offer more diverse, interdisciplinary venues; and the MiniLM variants provide balanced performance with minimal overhead.

5. Conclusions

This study presents an automated pipeline designed principally to assess the performance of Sentence Transformer models in recommending journals aligned with a manuscript’s thematic content. By comparing five freely available models—mpnet, minilm-l6, minilm-l12, multi-qa-distilbert, and roberta—we demonstrate how their underlying architectures yield differing trade-offs in precision, computational efficiency, and breadth of recommendations. Mpnet stands out for its strong semantic alignment, whereas minilm-l6 and minilm-l12 offer more rapid processing with marginal compromises in similarity scores. Meanwhile, multi-qa-distilbert and roberta display a more interdisciplinary reach, potentially suiting authors whose research spans multiple domains.

Crucially, our pipeline does not attempt to identify a definitive or best venue for publication. Instead, it supplies a systematic, reproducible method for gauging each model’s capabilities under consistent conditions. Real-world journal selection relies on far more than thematic fit, often involving publication fees, prestige, editorial policies, and personal author preferences. Nevertheless, our approach underlines how sentence embeddings can robustly capture scientific content, furnishing a starting point for authors who wish to explore plausible candidate journals grounded in semantic similarity.

Future extensions could fold in practical aspects such as acceptance rates, review timelines, and journal impact metrics. By combining these considerations with refined domain-specific embeddings and advanced query mechanisms, researchers can further enhance how they evaluate and compare Sentence Transformer models for biomedical or interdisciplinary journal recommendation.