Skip to Content
MDPIMDPI

Advanced

Applied SciencesApplied Sciences
PDF
  • Article
  • Open Access

30 March 2026

On the Applicability of LLMs and SLMs for Privacy-Preserving Named Entity Recognition in Financial Applications

and
1
Industrial System Institute, Athena Research Center, 26504 Patras, Greece
2
Archimedes, Athena Research Center, 15125 Athens, Greece
3
Department of Informatics, University of Piraeus, Karaoli & Dimitriou 80, 18534 Piraeus, Greece
*
Author to whom correspondence should be addressed.
Appl. Sci.2026, 16(7), 3332;https://doi.org/10.3390/app16073332
This article belongs to the Special Issue Artificial Intelligence on the Edge for Industry 4.0
Version Notes
Order Reprints

Abstract

This work explores how deep learning models, with different numbers of parameters, can be effectively applied to detect personal data within unstructured text using Named Entity Recognition (NER) techniques. We evaluate the performance of various architectures by leveraging a plethora of language models (LMs) consisting of Distilbert-base-uncased, Distilbert-base-cased, Bert-base-uncased, Bert-base-cased, Bert-large-uncased, Bert-large-cased, ModernBERT-base, ModernBERT-large, nomic-BERT-2048, RoBERTa-base, DistilRoBERTa-base, RoBERTa-large, Deberta-v3-xsmall, Deberta-v3-small, and Deberta-v3-base, which are evaluated using the performance indices of accuracy, precision, recall, and F1-score. Our experiments show that some Small Language Models (SLMs) compete equally with some corresponding LLMs (Large Language Models), based on the specific PII (Personally Identifiable Information) dataset, thus enhancing personal data detection, which is of paramount importance in financial applications. Moreover, we proposed a novel architecture based on an optimized transformer fine-tuning strategy to improve PII recognition across diverse contexts and conducted an extensive comparative analysis to evaluate the performance of our proposed architecture in relation to all relevant existing approaches reported in the literature. This evaluation, performed on the AI4Privacy PII 43 K dataset, encompasses every publicly available work we identified and provides a thorough benchmarking of our methods within the current research field. The results highlight both the strengths and limitations of existing solutions and demonstrate the effectiveness of SLMs in addressing the challenges of privacy-preserving information extraction.

1. Introduction

The rapid progress in Large Language Models (LLMs) has created opportunities in many domains, including cybersecurity, where the threat landscape continues to evolve, and organizations increasingly seek advanced defensive technologies [1]. Cybersecurity remains a pressing concern, as the frequency and sophistication of cyberattacks continue to grow, posing substantial risks to individuals, industry, and governmental infrastructures [2]. Recent advances in artificial intelligence, and LLMs in particular, have shown considerable promise in reshaping how cybersecurity challenges are addressed [3].
As demonstrated by Yigit et al. [3], modern LLMs can support a wide spectrum of security-related tasks such as threat intelligence analysis, vulnerability assessment, malware classification, anomaly detection, process automation, error correction, and even the generation of secure or insecure code depending on the prompts used. Research has also explored using LLMs to better understand adversarial behaviors. For example, Fayyazi et al. [4] examined how LLMs can interpret and summarize cyberattack Tactics, Techniques, and Procedures (TTPs) from the MITRE ATT&CK framework. Their findings show that Retrieval Augmented Generation (RAG) substantially improves the clarity and completeness of TTP explanations by injecting relevant contextual knowledge into the model’s responses, underscoring LLMs’ value for enhancing threat intelligence workflows.
Other studies leverage external knowledge bases through RAG to support vulnerability analysis. Du et al. [5] introduced Vul-RAG, a method that integrates knowledge-level retrieval with LLM reasoning to assist in vulnerability detection. Beyond improving detection accuracy, the vulnerability descriptions produced by Vul-RAG offer high-quality interpretability that can aid human analysts during manual assessments. LLMs have also been used as autonomous agents for complex defensive tasks. Rigaki et al. [6] demonstrated that LLM-driven agents can operate effectively within cybersecurity environments, achieving performance that rivals or surpasses that of heavily trained agents on sequential decision-making tasks, even without additional fine-tuning. Their work also introduced NetSecGame, a flexible and modular environment designed to support sophisticated multi-agent cybersecurity experimentation.
Belcak et al. [7] argue that Small Language Models (SLMs) offer a compelling alternative to large-scale models for many agentic AI scenarios. Their work suggests that SLMs provide sufficient capability for the language processing tasks typical in agent-based applications while also being operationally better suited to such settings due to their reduced computational footprint. The authors further emphasize that, because of their compact size, SLMs deliver these advantages at substantially lower cost, making them a practical and economical choice for the majority of agentic system use cases when compared to general-purpose LLMs. Similarly, Thamm et al. [8] note a rising preference for SLMs within application domains where latency, efficiency, and customizability are critical. They highlight that SLMs not only support faster inference and reduced resource consumption but also simplify fine-tuning and domain adaptation. These characteristics make them particularly appealing for environments with limited computational capacity or strict privacy requirements. The authors conclude that SLMs are well positioned for tasks involving localized data processing, ultra-low-latency decision-making, and targeted knowledge acquisition, areas where large models frequently encounter significant operational limitations.
Within Natural Language Processing (NLP), Named Entity Recognition (NER) has long been recognized as a core technique for transforming raw, unstructured text into meaningful structured information. By identifying and categorizing entities, such as people, organizations, locations, temporal expressions, and other domain-specific terms, NER enables more efficient information retrieval, downstream analysis, and knowledge extraction [9]. Its importance stems from its ability to impose structure on free-form text, making large corpora more interpretable and actionable.
In the financial domain, NER plays a particularly critical role due to the high volume and complexity of unstructured textual data generated by sources such as financial reports, earnings calls, regulatory filings, and news articles. Accurate identification of entities, including companies, financial instruments, monetary values, events, and temporal references, supports key tasks such as risk assessment, market analysis, compliance monitoring, and automated decision-making. By structuring financial text, NER enables more precise extraction of actionable insights and facilitates the integration of textual information into quantitative financial models. According to Zang [10], financial institutions process millions of documents annually, ranging from loan applications to regulatory filings, each containing critical structured information embedded within unstructured text. The automated extraction of entities such as organization names, monetary values, and regulatory identifiers remains computationally intensive and accuracy-critical. Peeters et al. [11] introduced a NER system built on a Bidirectional Encoder Representations from Transformers (BERT) architecture, trained and evaluated on a dataset of 39,531 samples. Their results demonstrate strong performance, underscoring the effectiveness of transformer-based models for entity extraction tasks. In a related direction, Huang et al. [12] proposed FinBERT, a domain-adapted language model tailored specifically for financial text. Their study illustrates that FinBERT captures domain-specific linguistic patterns more effectively than general-purpose models. It excels in summarizing contextual nuances found in financial documents, particularly in settings where the available training data is limited or includes specialized terminology rarely observed in general texts.
This study investigates how deep learning models with widely varying parameter sizes can be effectively applied to identify personal data in unstructured text through NER. To assess their capabilities, we conducted a systematic evaluation of a broad spectrum of language model architectures, including DistilBERT-base-uncased, DistilBERT-base-cased, BERT-base-uncased, BERT-base-cased, BERT-large-uncased, BERT-large-cased, ModernBERT-base, ModernBERT-large, nomic-BERT-2048, RoBERTa-base, DistilRoBERTa-base, RoBERTa-large, DeBERTa-v3-xsmall, DeBERTa-v3-small, and DeBERTa-v3-base. These models were compared using accuracy, precision, recall, and F1-score to examine how, and to what extent, certain SLMs can perform comparably to corresponding LLMs on a Personally Identifiable Information (PII)-specific dataset. Our findings reveal both the strengths and limitations of current approaches and demonstrate that SLMs can offer competitive performance, underscoring their viability for privacy-preserving information extraction tasks.

2. Materials and Methods

2.1. Dataset and Preprocessing

In this study, we employed the AI4Privacy PII 43 K dataset, a notable contribution to the intersection of privacy and artificial intelligence. Available through the ai4privacy/pii-masking-43 K repository [13], the dataset addresses the increasing need for robust personal data protection in AI-driven systems. It contains roughly 43,000 English-language samples generated through proprietary algorithms, enabling the creation of synthetic data that mitigates the risk of privacy breaches. Designed specifically for training AI models to detect and mask PII, the dataset spans 54 PII categories across 229 diverse use cases, including domains such as business and law. By providing a comprehensive foundation for data anonymization, this resource supports organizations that manage sensitive information, such as financial or medical institutions, in applying PII masking models to safely anonymize data prior to analysis, research, or data sharing. Each record in the dataset consists of six fields with corresponding text categories, as analytically described in Appendix A. Additionally, in Table A1 of Appendix A, we describe the records from the AI4Privacy PII 43 K dataset analytically.

2.2. Related Work and Comparative Analysis

Additionally, we performed a comparative analysis by identifying all existing works that referenced the NER-based AI4Privacy PII dataset used in our study. We executed two queries in Google Scholar. The first query searched for the exact dataset link on the Kaggle website (“https://www.kaggle.com/datasets/verracodeguacas/ai4privacy-pii”) (accessed on Friday 10 October 2025), which returned only one result [14]. This paper was relevant because it reported evaluation metrics based on the AI4Privacy PII dataset, specifically the 43 K version, which is the same version used in our work, but their work [14] used only 42 out of the 56 PII classes by performing data cleaning. The second query searched for the dataset name (“AI4Privacy-PII”), which returned 33 results. Our Google Scholar search was conducted in October 2025.
From the 33 results, five were excluded for the following reasons:
  • One result (the 16th) was a duplicate of the same work as in the first query [14].
  • Three results (the 12th, 17th, and 26th) were citations within other publications and did not correspond to standalone works.
  • Two results (the 25th and 29th) [15] represented the same work from different sources, so they were treated as a single publication.
From the remaining 28 works, none used the entire AI4Privacy PII 43 K dataset without data cleaning or data augmentation. These 28 works are categorized as follows:
  • Five works in total used the 43 K version of the dataset: (i) three works used the 43 K version without demonstrating the same evaluation metrics as in our study (the 1st, 8th, and 21st results from the 33 results) [16,17,18], (ii) one work used only a fragment of the dataset (the 22nd result) (with the evaluation results of precision and recall) [19], and (iii) one work used the dataset while increasing contextual diversity by employing a synthetic PII extension (65 K version) without demonstrating the same evaluation metrics as in our study, which appeared twice in the search results (the 25th and 29th results), representing the same publication from different sources [15].
  • Five works (the 2nd, 6th, 9th, 28th, and 33rd results) [20,21,22,23,24] mentioned the dataset only in their related work sections and did not use it in their experiments.
  • Eight works (the 3rd, 5th, 7th, 14th, 18th, 19th, 23rd, and 32nd results) [25,26,27,28,29,30,31,32] used the 200 K version of the dataset.
  • Seven works (the 4th, 11th, 13th, 15th, 20th, 27th, and 30th results) [33,34,35,36,37,38,39] used the 300 K version of the dataset.
  • Two works (the 24th and 31st results) [40,41] used the 400 K version of the dataset.
  • One work (the 10th result) [42] used both the 300 K and 500 K versions of the dataset.
From the two queries combined, only two works use the AI4Privacy PII 43 K dataset and provide evaluation metrics, but they apply data cleaning [14,19], unlike our work, which uses the entire dataset. The inclusion and exclusion process is illustrated in Figure 1. As described analytically above, among these two works, (i) one used 42 of the 56 PII classes [14] and (ii) one used only a fragment of the dataset [19].
Figure 1. Comparative analysis based on the literature.

2.3. Language Models Under Study and Experimental Setup

This work explores how deep learning models, with completely different numbers of parameters, can be effectively applied to detect personal data within unstructured text using NER techniques. Using the indices of accuracy, precision, recall, and F1-score, we evaluate the performance of various architectures by leveraging the following language models:
  • BERT-large-uncased (2018): This larger configuration of BERT increases depth and parameter count, allowing it to model more subtle contextual relationships. Released with the original BERT paper, the uncased model remains a powerful baseline for many high-performance NLP tasks.
  • BERT-large-cased (2018): Similarly introduced in 2018, this variant combines BERT-large’s capacity with case-sensitive tokenization. It is particularly effective for entity-rich tasks, albeit with substantially higher computational cost.
  • BERT-base-uncased (2018): BERT-base, introduced in 2018, set a new standard for contextualized language representations. The uncased version is suited to tasks where capitalization does not carry semantic importance but deep bidirectional context does.
  • BERT-base-cased (2018): Published in the same foundational work, the cased version preserves case distinctions and, therefore, tends to perform better in tasks involving names, formal terminology, and other case-sensitive expressions.
  • DistilBERT-base-uncased (2019): DistilBERT was introduced in 2019 as a compact alternative to BERT, designed to retain most of its capability while being faster and lighter. The uncased version ignores capitalization and performs well on broad text corpora where case is not critical.
  • DistilBERT-base-cased (2019): Released alongside the uncased variant, the cased version keeps capitalization information, which can help identify names, acronyms, and structured entities. It offers a strong balance between efficiency and the ability to capture fine-grained textual cues.
  • RoBERTa-base (2019): RoBERTa was introduced in 2019 as an improved, more robustly trained version of BERT. Its base model eliminates next sentence prediction and uses dynamic masking, leading to stronger performance across many downstream tasks.
  • RoBERTa-large (2019): The large version, introduced alongside the base model in 2019, deepens the architecture to capture richer semantics. It provides excellent performance in detailed entity recognition tasks, though it requires considerably more compute.
  • DistilRoBERTa-base (2019): Released soon after RoBERTa, this distilled model delivers much of RoBERTa’s accuracy with a significantly smaller footprint. Its efficiency makes it ideal for real-time or resource-constrained applications.
  • DeBERTa-v3-xsmall (2021): DeBERTa-v3 was released in 2021, incorporating disentangled attention and an improved masked language modeling objective. The xsmall version brings these benefits to a very compact model that still performs well on NER tasks.
  • DeBERTa-v3-small (2021): Also introduced in 2021, this variant expands capacity while maintaining efficiency. It is well-suited for tasks requiring stronger contextual understanding without incurring heavy computational costs.
  • DeBERTa-v3-base (2021): The base version of DeBERTa-v3 further leverages the improved architecture released in 2021, consistently outperforming similarly sized models in structured extraction tasks, including PII recognition.
  • nomic-BERT-2048 (2023): Introduced in 2023, nomic-BERT was developed to handle significantly longer inputs, up to 2048 tokens. This capability makes it well-suited for processing long documents where important information may be spread far apart.
  • ModernBERT-base (2023): ModernBERT appeared in 2023 with architectural refinements designed to speed up training and improve efficiency. The base version provides a balanced option for users who want a streamlined transformer without compromising too much on performance.
  • ModernBERT-large (2023): Released in the same year, the large version increases the parameter count and depth, enabling more expressive modeling. While more demanding computationally, it typically offers stronger results in complex NER tasks.
Across these models, the underlying architectures share the same transformer foundation but differ in depth, parameter count, and design refinements that shape their performance profiles. The BERT and RoBERTa families rely on multi-layer bidirectional transformers that encode context from both directions, with the larger variants offering deeper stacks of attention layers and richer representational capacity. DistilBERT and DistilRoBERTa compress these architectures through knowledge distillation, reducing computation while attempting to retain the core contextual reasoning abilities. ModernBERT introduces architectural optimizations that streamline attention computations and accelerate processing, making it more efficient for large-scale inference. DeBERTa-v3 further departs from the traditional transformer structure by disentangling the attention mechanism and enhancing the masked language modeling objective, enabling the model to better separate content from positional information. Nomic-BERT adds another dimension by extending its context window to 2048 tokens, making it capable of handling longer sequences than conventional BERT-based models. While all these models operate within the transformer paradigm, their architectural nuances, ranging from depth and embedding size to optimized attention mechanisms, collectively shape how effectively they capture semantic relationships and identify entities in unstructured text.
We studied the fine-tuning process of all these models. Table 1 contains the list of models with the year of publication, model category, number of parameters, and runtime.
Table 1. Year of publication, category, runtime, and number of parameters of the studied models.
All the experiments were conducted with a computer with an Intel i7 14,700 CPU and 64 GB of RAM, equipped with a GeForce RTX 5060 16 GB GDDR 7 GPU. Among all the training parameters, we chose epochs = 5, learning rate = 3 × 10−5, batch_size = 32, and weight_decay = 0.01. The dataset, after sufficient normalization, was divided into training and test datasets in a ratio of 0.8/0.2, respectively. It is worth noting that during fine-tuning, the GPU memory consumption was at the upper limit (100%).
Although the field of LMs does not follow a strict or universally agreed-upon taxonomy for categorizing transformer models by parameter size, several influential works provide commonly adopted conventions. Foundational encoder models, such as BERT (~110 M parameters), are widely referred to as base models, with their larger counterparts (~340 M parameters) labeled large [43]. More recent studies and large model releases (e.g., GPT-3, PaLM) generally reserve the term LLMs for architectures that exceed billions of parameters [44,45]. Similarly, comprehensive surveys on pretrained model compression and scaling behavior indicate that models under ~50 M parameters are frequently treated as extra-small, those in the 50–100 M range as small, around 100–150 M as base, and 300–400 M as large [46,47,48].
In this study, we follow these widely used parameter ranges, as they align well with the BERT family architectures included in our experiments. Our evaluated models span from approximately 22 M to 395 M parameters, making it practical to group them into three meaningful categories: (i) lightweight (<150 M), (ii) medium (150–300 M), and (iii) large encoder variants (>300 M). This classification is consistent with community conventions and reflects the fact that models in the hundreds of millions of parameters are typically viewed as compact encoder/decoder architectures, not true LLMs [44,45]. The categories of all the evaluated models are shown in Table 1. By clearly defining the parameter categories used in our comparison, we aim to maintain transparency and ensure consistent terminology for readers and reviewers.

3. Results

In this section, we present the comparison of our work with other similar studies in terms of the performance metrics of accuracy, precision, f1-score, and loss, and then we discuss the results. Additionally, we compare our work with other studies conducted on the same or similar datasets, as demonstrated in Table 2.
Table 2. Performance indices of our fine-tuned models and comparison with other works.
In Table 3, Table 4 and Table 5, the change in performance metrics per epoch for the best models of RoBERTa-large, nomic-BERT-2048, and RoBERTa-base are presented, respectively.
Table 3. Changes in performance metrics per epoch are for the model RoBERTa-large.
Table 4. Changes in performance metrics per epoch are for the model nomic-BERT-2048.
Table 5. Changes in performance metrics per epoch are for the model RoBERTa-base.
The comparison among all models we studied on the basis of F1-score is illustrated in Figure 2.
Figure 2. Comparison among all models on the basis of F1-score.
The comparison among all models we studied on the basis of the accuracy index is illustrated in Figure 3.
Figure 3. Comparison among all models on the basis of the accuracy index.
Additionally, Table 6 demonstrates each model’s evaluation samples and steps per second.
Table 6. Each model’s evaluation samples and steps per second.
Our experiments show that some SLMs compete equally with some corresponding LLMs, based on the specific PII dataset, thus enhancing personal data detection, which is of paramount importance in financial applications. The evaluation of fifteen transformer-based architectures across accuracy, precision, recall, F1-score, training loss, and computational cost reveals several notable patterns about model performance on our dataset. Overall, all models achieve high accuracy (≥96%), indicating that the task is well-aligned with transformer-based language understanding. However, the subtler distinctions across precision, recall, and F1-score highlight how architectural choices and model size influence reliability and generalization. Across the full set of models, the highest F1-scores are obtained by RoBERTa-large (93.82%), nomic-BERT-2048 (93.61%), BERT-LARGE-uncased (93.41%), and DeBERTa-v3-base (93.36%). These systems consistently balance strong precision with high recall, resulting in the most robust overall performance. As expected, these models also come with substantially higher parameter counts, ranging from 137 M to 355 M, and significantly longer training times. For example, ModernBERT-large required over 106 h of training, while BERT-LARGE-cased exceeded 62 h.
To contextualize the performance of our models, we compare our results against those reported by Areskoug and Mörk (2025) [14], who evaluated a range of architectures, including BiLSTM variants, BERT-based models, DistilBERT, DeBERTa, and ModernBERT on a similar personal data detection task. Across our experiments, the highest F1-scores are obtained by RoBERTa-large (93.82%), nomic-BERT-2048 (93.61%), DeBERTa-v3-base (93.36%), and BERT-LARGE-uncased (93.41%). A closer inspection of the ranking positions further clarifies the comparative behavior of large and small models. While the single best-performing model in both accuracy and F1-score is a large architecture (RoBERTa-large), the remaining large models do not consistently dominate the top positions. Specifically, the four large models (RoBERTa-large, BERT-LARGE-uncased, ModernBERT-large, and BERT-LARGE-cased) are ranked first, sixth, seventh, and eighth in terms of accuracy and first, third, fifth, and 13th in terms of F1-score, respectively. In contrast, the positions immediately following the top-ranked model are largely occupied by SLMs. For accuracy, SLMs hold positions 2 through 5, while for F1-score, they occupy positions 2 and 4. This ranking-based comparison shows that, aside from the top result, smaller and mid-sized models consistently achieve performance comparable to, or better than, most large architectures, providing direct evidence that SLMs are strong competitors to LLMs on this PII-specific NER task. These results place our best-performing models very close to, though slightly below, the top values reported by Areskoug and Mörk [14]. For example, their strongest models, DeBERTa and DeBERTa-CRF, reach F1-scores of 96.7% and 96.8%, respectively. Similarly, their BERT, BERT-CRF, and DistilBERT-CRF models achieve F1-scores around 95.8–96.1%. Although their highest scores exceed ours by a small margin, it is important to consider the broader context of model evaluation, reporting completeness, and computational transparency. Additionally, it is worth noting that their work [14] used only 42 of the 56 PII classes, whereas we used all 56.
Zhou et al. [19] further highlight the importance of evaluation conditions when comparing reported performance. Their study relies on substantial data cleaning and, notably, on using only a fragment of the original dataset, which significantly simplifies the detection task. While their GPT-4o-based Rescriber achieves a precision of 94%, its recall drops to 88%, indicating that a considerable portion of sensitive entities remains undetected. This limitation is even more pronounced for the Rescriber-Llama3-8B model, where recall falls sharply to 63%, despite similar precision. Such an imbalance suggests that these systems may be overly conservative, prioritizing precision at the expense of comprehensive coverage. In contrast, our models consistently maintain both high precision and high recall across all 56 PII classes, resulting in F1-scores above 93% for several architectures. Importantly, this performance is achieved without data cleaning or class reduction, making our results more representative of real-world financial scenarios.
Overall, this places our work at a clear competitive advantage. By evaluating a broad range of models under identical and transparent conditions, and using the full dataset with all 56 PII classes, we provide a more realistic and demanding benchmark than prior studies. Our results show that strong and well-balanced PII detection does not require extremely large or costly models, nor heavy data filtering, but can be achieved with carefully selected transformer architectures that generalize well across diverse entity types.

4. Discussion

While the large architectures generally lead the rankings, what stands out is how closely several smaller models trail behind. For instance, DeBERTa-v3-small (44 M parameters) achieves an F1-score of 92.46%, almost identical to BERT-base-cased (92.33%) despite having less than half the parameters and a run time of only 14 h compared to 12 h for BERT-base, remarkably efficient given its size. DistilBERT-base-uncased and DistilBERT-base-cased (82 M parameters) maintain F1-scores of around 91.6–91.8%, only about two percentage points behind the best-performing large models, while the run time is just over six hours. Distil-RoBERTa-base also performs strongly, with an F1-score of 91.84% and a comparatively short training time of 10 h. These findings reinforce that, although large models deliver marginally higher performance, the gap is much smaller than their computational cost suggests.
Furthermore, we have the following architecture-based observations: RoBERTa variants consistently perform near the top, with the large variant achieving the best overall F1-score (93.82%), confirming RoBERTa’s well-known strengths in masked language model pretraining. BERT-based models show predictable scaling behavior: performance improves from DistilBERT to BERT-base, and from BERT-base to BERT-large, though the gains shrink as the model capacity increases. DeBERTa-v3 models, especially the base version, show strong behavior across all metrics, benefiting from their enhanced disentangled attention and improved vocabulary embedding structure. ModernBERT, while extremely fast to train, underperforms slightly in precision and recall relative to the other architectures. Nevertheless, its speed, particularly the base version, which finishes in just over an hour, makes it promising for scenarios requiring rapid experimentation.
Additionally, we notice that run time varies enormously between models, from 1 h and 10 min for ModernBERT-base to nearly 58–62 h for BERT-LARGE variants. When combining performance and time cost, several models achieve an excellent balance. DistilBERT models achieve competitive results at a fraction of the run time. Nomic-BERT-2048 is especially efficient, offering the third-highest F1-score while the run time lasts for only about 80 min, making it an exceptional speed–performance trade-off. DeBERTa-v3-small also delivers near state-of-the-art results at moderate computational cost. These patterns support the broader conclusion that small and mid-size models can rival large ones on specialized tasks, especially when the dataset is well-structured and domain-specific.

5. Conclusions

Our study demonstrates that certain SLMs can perform as well as their larger counterparts (LLMs) when evaluated on a specific PII dataset, highlighting their potential for enhancing personal data detection, an especially critical task in financial applications. Additionally, we introduced a novel architecture that leverages an optimized transformer fine-tuning strategy to improve PII recognition across varied contexts. Through an extensive comparative analysis using the AI4Privacy PII 43 K dataset, we benchmarked our approach against all relevant publicly available methods, providing a comprehensive assessment of current solutions. Overall, the results clearly support our conclusion that SLMs are strong competitors to LLMs. While the highest accuracy and F1-score are achieved by a large model, SLMs consistently follow in the top rankings, holding positions 2–5 for accuracy and positions 2 and 4 for F1-score. This close performance gap highlights that smaller models can deliver results comparable to much larger architectures, reinforcing their practical value for privacy-preserving NER in real-world financial settings. The results not only underscore the strengths and weaknesses of existing approaches but also confirm the effectiveness of SLMs in addressing the challenges of privacy-preserving information extraction.
For future work, we plan to conduct a per-class confusion matrix for all the models we examined, as well as to expand our study to include additional SLMs and LLMs, further enriching the comparative evaluation and deepening our understanding of their performance dynamics.

Author Contributions

Conceptualization, E.P. and K.S.; methodology, E.P. and K.S.; software, E.P. and K.S.; validation, E.P. and K.S.; formal analysis, E.P. and K.S.; investigation, E.P. and K.S.; resources, E.P. and K.S.; data curation E.P. and K.S.; writing—original draft preparation, E.P. and K.S.; writing—review and editing, E.P. and K.S.; visualization, E.P. and K.S.; supervision, K.S.; project administration, K.S.; funding acquisition, K.S. All authors have read and agreed to the published version of the manuscript.

Funding

This work has been partially supported by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

AI4Privacy PII 43 K dataset: https://www.kaggle.com/datasets/verracodeguacas/ai4privacy-pii (accessed on 1 December 2025).

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

The following abbreviations are used in this manuscript:
NERNamed Entity Recognition
LMsLanguage Models
SLMsSmall Language Models
LLMsLarge Language Models
PIIPersonally Identifiable Information
TTPsTactics, Techniques, and Procedures
RAGRetrieval Augmented Generation
NLPNatural Language Processing
BERTBidirectional Encoder Representations from Transformers

Appendix A

Each record in the AI4Privacy PII 43 K dataset consists of six fields with corresponding text categories:
  • Masked text: contains the original text where PII words were masked;
  • Unmasked text: which is the original text;
  • Privacy mask: the PII classes that belong to each sentence;
  • Span labels: the spanning range of each PII class;
  • BIO labels: where we have the labels of the PII classes;
  • Tokenized text: each sentence is split into its respective tokens.
Let us assume we have the class “phone_IMEI_number” (where IMEI stands for International Mobile Equipment Identity), where:
  • B is the beginning of the PII text: B-phone_IMEI_number;
  • I is within the PII text: I-phone_IMEI_number;
  • O is the nOn PII class of the text: O.
The dataset consists of PII classes (e.g., ACCOUNTNUMBER, CREDITCARDISSUER, BITCOINADDRESS, CREDITCARDCVV, CURRENCYCODE, PASSWORD, PHONEIMEI, SNN, USERNAME, EMAIL).
Examples from the record set that we used [13] are shown below in Table A1.
Table A1. Records from the AI4Privacy PII 43 K dataset.

References

  1. Zhang, J.; Bu, H.; Wen, H.; Liu, Y.; Fei, H.; Xi, R.; Li, L.; Yang, Y.; Zhu, H.; Meng, D. When Llms Meet Cybersecurity: A Systematic Literature Review. Cybersecurity 2025, 8, 55. [Google Scholar] [CrossRef] [Scilit]
  2. Scala, N.M.; Reilly, A.C.; Goethals, P.L.; Cukier, M. Risk and the Five Hard Problems of Cybersecurity. Risk Anal. 2019, 39, 2119–2126. [Google Scholar] [CrossRef] [Scilit]
  3. Yigit, Y.; Buchanan, W.J.; Tehrani, M.G.; Maglaras, L. Review of Generative AI Methods in Cybersecurity. arXiv 2024, arXiv:2403.08701. [Google Scholar] [CrossRef] [Scilit]
  4. Fayyazi, R.; Taghdimi, R.; Yang, S.J. Advancing TTP Analysis: Harnessing the power of large language models with retrieval augmented generation. In Proceedings of the 2024 Annual Computer Security Applications Conference Workshops (ACSAC Workshops); IEEE: New York, NY, USA, 2024; pp. 255–261. [Google Scholar] [CrossRef] [Scilit]
  5. Du, X.; Zheng, G.; Wang, K.; Zou, Y.; Wang, Y.; Deng, W.; Feng, J.; Liu, M.; Chen, B.; Peng, X.; et al. Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG. arXiv 2025, arXiv:2406.11147. [Google Scholar] [CrossRef] [Scilit]
  6. Rigaki, M.; Lukáš, O.; Catania, C.; Garcia, S. Out of the cage: How stochastic parrots win in Cyber Security Environments. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence; SCITEPRESS—Science and Technology Publications, Lda.: Setúbal, Portugal, 2024; pp. 774–781. [Google Scholar] [CrossRef] [Scilit]
  7. Belcak, P.; Heinrich, G.; Diao, S.; Fu, Y.; Dong, X.; Muralidharan, S.; Lin, Y.C.; Molchanov, P. Small Language Models are the Future of Agentic AI. arXiv 2025, arXiv:2506.02153. [Google Scholar] [CrossRef] [Scilit]
  8. Thamm, B.G. Trustworthy and secure ai: How small language models strengthen data security. Service Contractor Magazine, 8 October 2024.
  9. Pakhale, K. Comprehensive overview of named entity recognition: Models, domain-specific applications and challenges. arXiv 2023, arXiv:2309.14084. [Google Scholar] [CrossRef] [Scilit]
  10. Wang, Z. Enhancing Financial Named Entity Recognition through Adaptive Few-Shot Learning: A Comparative Study of Pre-Trained Language Models. J. Adv. Comput. Syst. 2024, 4, 13–25. [Google Scholar] [CrossRef] [Scilit]
  11. Peeters, C.; Vijverberg, K.; Pouwer, M.; Westerman, B.; Boot, M.; Verberne, S. Evaluation of surus: A named entity recognition NLP system to extract knowledge from Interventional Study Records. BMC Med. Res. Methodol. 2025, 25, 184. [Google Scholar] [CrossRef] [Scilit]
  12. Huang, A.H.; Wang, H.; Yang, Y. FinBERT: A large language model for extracting information from financial text. Contemp. Account. Res. 2023, 40, 806–841. [Google Scholar] [CrossRef] [Scilit]
  13. Henao, W.E. Ai4privacy-PII. Available online: https://www.kaggle.com/datasets/verracodeguacas/ai4privacy-pii (accessed on 1 December 2025).
  14. Areskoug, T.; Mörk, V. Detection of Personal Data in Unstructured Text Using Deep Learning Models. arXiv 2025, arXiv:2506.22305. [Google Scholar]
  15. Song, M.; Kim, H.; Kim, J.; Shin, S.; Son, S. Refusal Is Not an Option: Unlearning Safety Alignment of Large Language Models. In Proceedings of the 34th USENIX Security Symposium (USENIX Security 25), Seattle, WA, USA, 13–15 August 2025; pp. 319–338. [Google Scholar]
  16. Viswanath, V.; Thenmozhi, M.; Naganandh, S. Artificial Intelligence and Privacy Concerns: Balancing Innovation with Security. Cureus 2025, 2, 2–11. [Google Scholar] [CrossRef] [Scilit]
  17. Zhan, J.; Zhang, W.; Zhang, Z.; Xue, H.; Zhang, Y.; Wu, Y. Portcullis: A Scalable and Verifiable Privacy Gateway for Third-Party LLM Inference. In Proceedings of the AAAI Conference on Artificial Intelligence; AAAI Press: Washington, DC, USA, 2025; Volume 39, pp. 1022–1030. [Google Scholar]
  18. Shen, Z.; Xi, Z.; He, Y.; Tong, W.; Hua, J.; Zhong, S. The Fire Thief Is Also the Keeper: Balancing Usability and Privacy in Prompts. arXiv 2024, arXiv:2406.14318. [Google Scholar] [CrossRef] [Scilit]
  19. Zhou, J.; Xu, E.; Wu, Y.; Li, T. Rescriber: Smaller-LLM-Powered User-Led Data Minimization for LLM-Based Chatbots. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems; Association for Computing Machinery: New York, NY, YSA, 2025; pp. 1–28. [Google Scholar]
  20. Savkin, M.; Ionov, T.; Konovalov, V. SPY: Enhancing Privacy with Synthetic PII Detection Dataset. In Proceedings of the 2025 Conference of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Student Research Workshop); Association for Computational Linguistics: Albuquerque, NM, USA, 2025; pp. 236–246. [Google Scholar]
  21. Meena, B.; Skubisz, J.; Rajgarhia, H.; Dave, N.; Ganesh, K.; Dalmia, S.; Mukherji, A.; Sundarababu, V. Scalable Multilingual PII Annotation for Responsible AI in LLMs. arXiv 2025, arXiv:2510.06250. [Google Scholar]
  22. Rajgarhia, H.; Gupta, S.; Shaik, A.; Kumar, G.P.; Santhoshraj, Y.; Nishitha, S.N.T.; Mukherji, A. An Evaluation Study of Hybrid Methods for Multilingual PII Detection. arXiv 2025, arXiv:2510.07551. [Google Scholar] [CrossRef] [Scilit]
  23. Hasegawa, M. LLM-Assisted Detecting and Redacting Confidential Information for Government Information Disclosure. Ph.D. Thesis, Virginia Tech, Blacksburg, VA, USA, 2025. [Google Scholar]
  24. Li, H.; Ye, J.; Wu, J. Privacy Dilemmas and Opportunities in Large Language Models: A Brief Review. Front. Comput. Sci. 2025, 19, 1910356. [Google Scholar] [CrossRef] [Scilit]
  25. Mainetti, L.; Elia, A. Detecting Personally Identifiable Information Through Natural Language Processing: A Step Forward. Appl. Syst. Innov. 2025, 8, 55. [Google Scholar] [CrossRef] [Scilit]
  26. Karavdić, N. Handling Confidential Data in LLM Prompts. Master’s Thesis, Mälardalens University, Västerås, Sweden, June 2025. [Google Scholar]
  27. Wen, Y.; Marchyok, L.; Hong, S.; Geiping, J.; Goldstein, T.; Carlini, N. Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-Trained Models. Adv. Neural Inf. Process. Syst. 2024, 37, 83374–83396. [Google Scholar]
  28. Rashid, M.R.U.; Liu, J.; Koike-Akino, T.; Wang, Y.; Mehnaz, S. Forget to Flourish: Leveraging Machine Unlearning on Pretrained Language Models for Privacy Leakage. In Proceedings of the AAAI Conference on Artificial Intelligence; AAAI Press: Washington, DC, USA, 2025; Volume 39, pp. 20139–20147. [Google Scholar]
  29. Chu, K.; Lin, Z.; Xiang, D.; Shen, Z.; Su, J.; Chu, C.; Yang, Y.; Zhang, W.; Wu, W.; Zhang, W. Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference. arXiv 2025, arXiv:2508.08438. [Google Scholar]
  30. Chowdhury, A.R.; Glukhov, D.; Anshumaan, D.; Chalasani, P.; Papernot, N.; Jha, S.; Bellare, M. Prεεmpt: Sanitizing Sensitive Prompts for LLMs. arXiv 2025, arXiv:2504.05147. [Google Scholar]
  31. Zhu, Y.; Gao, N.; Liang, X.; Zhang, H. Exploiting Privacy-Preserving Prompt Techniques for Online Large Language Model Usage. In Proceedings of the GLOBECOM 2024—IEEE Global Communications Conference; IEEE: New York, NY, USA, 2024; pp. 4304–4309. [Google Scholar]
  32. Assoudi, H. Natural Language Processing on Oracle Cloud Infrastructure: Building Transformer-Based NLP Solutions Using Oracle AI and Hugging Face; Springer Nature: Berlin, Germany, 2024. [Google Scholar]
  33. Chen, X.; Tang, S.; Zhu, R.; Yan, S.; Jin, L.; Wang, Z.; Su, L.; Zhang, Z.; Wang, X.; Tang, H. The Janus Interface: How Fine-Tuning in Large Language Models Amplifies Privacy Risks. In Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security; Association for Computing Machinery: New York, NY, USA, 2024; pp. 1285–1299. [Google Scholar]
  34. Cheng, S.; Li, Z.; Meng, S.; Ren, M.; Xu, H.; Hao, S.; Yue, C.; Zhang, F. Understanding PII leakage in large language models: A systematic survey. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, Montreal, QC, Canada, 16–22 August 2025; pp. 10409–10417. [Google Scholar]
  35. Sachenko, A.; Vizhevskyi, P.; Savenko, O.; Ostroverkhov, V.; Maslyyak, B. Modern Strategies for Data Leak Detection and Prevention in Corporate Networks. In Proceedings of the Modern Data Science Technologies Doctoral Consortium, Lviv, Ukraine, 15–16 June 2025. [Google Scholar]
  36. Tao, J.; Shokri, R. (Token-Level) InfoRMIA: Stronger Membership Inference and Memorization Assessment for LLMs. arXiv 2025, arXiv:2510.05582. [Google Scholar]
  37. Lu, S.; Wang, L.; Wen, S.; Wang, Z.; Zhang, H. FedDTRE: Federated Dialogue Generation Models Powered by Trustworthiness Evaluation. arXiv 2025, arXiv:2510.08058. [Google Scholar] [CrossRef] [Scilit]
  38. Tsai, Y.; Hsiao, H.; Lin, S.D. Detecting and Perturbing Privacy-Sensitive Neurons to Defend Embedding Inversion Attacks. 2025. Available online: https://openreview.net/pdf?id=DF5TVzpTW0 (accessed on 19 March 2026).
  39. Härle, R.; Friedrich, F.; Brack, M.; Wäldchen, S.; Deiseroth, B.; Schramowski, P.; Kersting, K. Measuring and Guiding Monosemanticity. arXiv 2025, arXiv:2506.19382. [Google Scholar] [CrossRef] [Scilit]
  40. Loiseau, G.; Sileo, D.; Riquet, D.; Meyer, M.; Tommasi, M. Tau-Eval: A Unified Evaluation Framework for Useful and Private Text Anonymization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations; Association for Computational Linguistics: Albuquerque, NM, USA, 2025; pp. 216–227. [Google Scholar]
  41. Skapars, A.; Manino, E.; Sun, Y.; Cordeiro, L.C. GPT, But Backwards: Exactly Inverting Language Model Outputs. arXiv 2025, arXiv:2507.01693. [Google Scholar] [CrossRef] [Scilit]
  42. Garza, L.; Kotal, A.; Piplai, A.; Elluri, L.; Das, P.; Chadha, A. PRvL: Quantifying the Capabilities and Risks of Large Language Models for PII Redaction. arXiv 2025, arXiv:2508.05545. [Google Scholar] [CrossRef] [Scilit]
  43. Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv 2018, arXiv:1810.04805. [Google Scholar]
  44. Brown, T.B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. Language Models Are Few-Shot Learners. arXiv 2020, arXiv:2005.14165. [Google Scholar] [CrossRef] [Scilit]
  45. Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H.W.; Sutton, C.; Gehrmann, S.; et al. PaLM: Scaling Language Modeling with Pathways. arXiv 2022, arXiv:2204.02311. [Google Scholar] [CrossRef] [Scilit]
  46. Xia, M.; Zhang, X.; Zhang, R.; Chen, H.; Li, X. A Survey of Model Compression and Acceleration for Pretrained Language Models. arXiv 2022, arXiv:2202.07105. [Google Scholar] [CrossRef] [Scilit]
  47. Tay, Y.; Dehghani, M.; Abnar, S.; Chung, H.; Fedus, W.; Rao, J.; Narang, S.; Tran, V.; Yogatama, D.; Metzler, D. Scaling Laws vs. Model Architectures: How Does Inductive Bias Influence Scaling? arXiv 2022, arXiv:2207.10551. [Google Scholar] [CrossRef] [Scilit]
  48. Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; Liu, P.J. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 2020, 21, 1–67. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Journal Statistics
Multiple requests from the same IP address are counted as one view.
VarDiff: A Conceptual Model for Representing Variable Differences Between Clinical Decision Support Systems
Previous
A Quantum Public-Key Cryptosystem with Reusable Keys Using Entangled States
Next