Search Results (380)

This study proposes an application framework based on Large Language Models (LLMs) to analyze multimodal heterogeneous data in the power sector and introduces the CLB-BER model for classifying user electricity consumption behavior. We first employ the Euclidean–Cosine Dynamic Windowing (ECDW) method to optimize the adjustment phase of the CLUBS clustering algorithm, improving the classification accuracy of electricity consumption patterns and establishing a mapping between unlabeled behavioral features and user types. To overcome the limitations of traditional clustering algorithms in recognizing emerging consumption patterns, we fine-tune a pre-trained DistilBERT model and integrate it with a Softmax layer to enhance classification performance. The experimental results on real-world power grid data demonstrate that the CLB-BER model significantly outperforms conventional algorithms in terms of classification efficiency and accuracy, achieving 94.21% accuracy and an F1 score of 94.34%, compared to 92.13% accuracy for Transformer and lower accuracy for baselines like KNN (81.45%) and SVM (86.73%); additionally, the Improved-C clustering achieves a silhouette index of 0.63, surpassing CLUBS (0.62) and K-means (0.55), underscoring its potential for power grid analysis and user behavior understanding. Our framework inherently preserves temporal symmetry in consumption patterns through dynamic sequence alignment, enhancing its robustness for real-world applications. Full article

The rising frequency of network intrusions has significantly impacted critical infrastructures, leading to an increased focus on the detection of malicious network traffic in recent years. However, traditional port-based and classical machine learning-based malicious network traffic detection methods suffer from a dependence on expert experience and limited generalizability. In this paper, we propose a malicious traffic detection method based on an efficient federated learning framework of Bidirectional Encoder Representations from Transformers (BERT), called MT-FBERT. It offers two major advantages over most existing approaches. First, MT-FBERT pretrains BERT using two pre-training tasks along with an overall pre-training loss on large-scale unlabeled network traffic, allowing the model to automatically learn generalized traffic representations, which do not require human experience to extract the behavior features or label the malicious samples. Second, MT-FBERT finetunes BERT for malicious network traffic detection through an efficient federated learning framework, which both protects the data privacy of critical infrastructures and reduces resource consumption by dynamically identifying and updating only the most significant neurons in the global model. Evaluation experiments on public datasets demonstrated that MT-FBERT outperforms state-of-the-art baselines in malicious network traffic detection. Full article

Poultry feces, a critical biomarker for health assessment, requires timely and accurate pathological identification for food safety. Conventional visual-only methods face limitations due to environmental sensitivity and high visual similarity among feces from different diseases. To address this, we propose MMCD (Multimodal Chicken-feces Diagnosis), a ResNet50-based multimodal fusion model leveraging semantic complementarity between images and descriptive text to enhance diagnostic precision. Key innovations include the following: (1) Integrating MASA(Manhattan self-attention)and DSconv (Depthwise Separable convolution) into the backbone network to mitigate feature confusion. (2) Utilizing a pre-trained BERT to extract textual semantic features, reducing annotation dependency and cost. (3) Designing a lightweight Gated Cross-Attention (GCA) module for dynamic multimodal fusion, achieving a 41% parameter reduction versus cross-modal transformers. Experiments demonstrate that MMCD significantly outperforms single-modal baselines in Accuracy (+8.69%), Recall (+8.72%), Precision (+8.67%), and F1 score (+8.72%). It surpasses simple feature concatenation by 2.51–2.82% and reduces parameters by 7.5M and computations by 1.62 GFLOPs versus the base ResNet50. This work validates multimodal fusion’s efficacy in pathological fecal detection, providing a theoretical and technical foundation for agricultural health monitoring systems. Full article

This study investigates the ability of six pre-trained sentence transformers to organize medical knowledge by performing unsupervised clustering on 70 high-level Medical Subject Headings (MeSH) terms across seven medical specialties. We evaluated models from different pre-training paradigms: general-purpose, domain-adapted, and from-scratch domain-specific. The results reveal a clear performance hierarchy. A top tier of models, including the general-purpose MPNet and the domain-adapted BioBERT and RoBERTa, produced highly coherent, specialty-aligned clusters (Adjusted Rand Index > 0.80). Conversely, models pre-trained from scratch on specialized corpora, such as PubMedBERT and BioClinicalBERT, performed poorly (Adjusted Rand Index < 0.51), with BioClinicalBERT yielding a disorganized clustering. These findings challenge the assumption that domain-specific pre-training guarantees superior performance for all semantic tasks. We conclude that model architecture, alignment between the pre-training objective and the downstream task, and the nature of the training data are more critical determinants of success for creating semantically coherent embedding spaces for medical concepts. Full article

Driven by blockchain technology, numerous industries are increasingly adopting smart contracts to enhance efficiency, reduce costs, and improve transparency. As a result, ensuring the security of smart contracts has become critical. Traditional detection methods often suffer from low efficiency, are prone to missing complex vulnerabilities, and have limited accuracy. Although deep learning approaches address some of these challenges, issues with both accuracy and efficiency remain in current solutions. To overcome these limitations, this paper proposes a symmetry-inspired solution that harmonizes bidirectional and generative semantic patterns. First, we generate distinct feature extraction segments for different vulnerabilities. We then use the Bidirectional Encoder Representations from Transformers (BERT) module to extract original semantic features from these segments and the Generative Pre-trained Transformer (GPT) module to extract generative semantic features. Finally, the two sets of semantic features are fused using a multi-attention mechanism and input into a classifier for result prediction. Our method was tested on three datasets, achieving F1 scores of 93.33%, 93.65%, and 92.31%, respectively. The results demonstrate that our approach outperforms most existing methods in smart contract detection. Full article

Handwritten Keyword Spotting (KWS) remains a challenging task, particularly in segmentation-free scenarios where word images must be retrieved and ranked based on their similarity to a query without relying on prior page-level segmentation. Traditional KWS methods primarily focus on visual similarity, often overlooking the underlying semantic relationships between words. In this work, we propose a novel NLP-driven re-ranking approach that refines the initial ranked lists produced by state-of-the-art KWS models. By leveraging semantic embeddings from pre-trained BERT-like Large Language Models (LLMs, e.g., RoBERTa, MPNet, and MiniLM), we introduce a relevance feedback mechanism that improves both verbatim and semantic keyword spotting. Our framework operates in two stages: (1) projecting retrieved word image transcriptions into a semantic space via LLMs and (2) re-ranking the retrieval list using a weighted combination of semantic and exact relevance scores based on pairwise similarities with the query. We evaluate our approach on the widely used George Washington (GW) and IAM collections using two cutting-edge segmentation-free KWS models, which are further integrated into our proposed pipeline. Our results show consistent gains in Mean Average Precision (mAP), with improvements of up to $2.3\%$ (from $94.3\%$ to $96.6\%$) on GW and $3\%$ (from $79.15\%$ to $82.12\%$) on IAM. Even when mAP gains are smaller, qualitative improvements emerge: semantically relevant but inexact matches are retrieved more frequently without compromising exact match recall. We further examine the effect of fine-tuning transformer-based OCR (TrOCR) models on historical GW data to align textual and visual features more effectively. Overall, our findings suggest that semantic feedback can enhance retrieval effectiveness in KWS pipelines, paving the way for lightweight hybrid vision-language approaches in handwritten document analysis. Full article

Software-defined networking (SDN) represents a transformative shift in network architecture by decoupling the control plane from the data plane, enabling centralized and flexible management of network resources. However, this architectural shift introduces significant security challenges, as SDN’s centralized control becomes an attractive target for various types of attacks. While the body of current research on attack detection in SDN has yielded important results, several critical gaps remain that require further exploration. Addressing challenges in feature selection, broadening the scope beyond Distributed Denial of Service (DDoS) attacks, strengthening attack decisions based on multi-flow analysis, and building models capable of detecting unseen attacks that they have not been explicitly trained on are essential steps toward advancing security measures in SDN environments. In this paper, we introduce a novel approach that leverages Natural Language Processing (NLP) and the pre-trained Bidirectional Encoder Representations from Transformers (BERT)-base-uncased model to enhance the detection of attacks in SDN environments. Our approach transforms network flow data into a format interpretable by language models, allowing BERT-base-uncased to capture intricate patterns and relationships within network traffic. By utilizing Random Forest for feature selection, we optimize model performance and reduce computational overhead, ensuring efficient and accurate detection. Attack decisions are made based on several flows, providing stronger and more reliable detection of malicious traffic. Furthermore, our proposed method is specifically designed to detect previously unseen attacks, offering a solution for identifying threats that the model was not explicitly trained on. To rigorously evaluate our approach, we conducted experiments in two scenarios: one focused on detecting known attacks, achieving an accuracy, precision, recall, and F1-score of 99.96%, and another on detecting previously unseen attacks, where our model achieved 99.96% in all metrics, demonstrating the robustness and precision of our framework in detecting evolving threats, and reinforcing its potential to improve the security and resilience of SDN networks. Full article

Financial forecasting is a research and practical challenge, providing meaningful economic and strategic insights. While Machine Learning (ML) models are employed in various studies to examine the impact of technical and sentiment factors on financial markets forecasting, in this work, macroeconomic indicators are also combined to forecast the Standard & Poor’s (S&P) 500 index. Initially, contextual data are scored using TextBlob and pre-trained DistilBERT-base-uncased models, and then a combined dataset is formed. Followed by preprocessing, feature engineering and selection techniques, three corresponding datasets are generated and their impact on future prices is examined, by employing ML models, such as Linear Regression (LR), Random Forest (RF), Gradient Boosting (GB), XGBoost, and Multi-Layer Perceptron (MLP). LR and MLP show robust results with high R² scores, close to 0.998, and low error MSE and MAE rates, averaging at 350 and 13 points, respectively, across both training and test datasets, with technical indicators contributing the most to the prediction. While other models also perform very well under different dataset combinations, overfitting challenges are evident in the results, even after additional hyperparameter tuning. Potential limitations are highlighted, motivating further exploration and adaptation techniques in financial modeling that enhance predictive capabilities. Full article

Diagnosing Parkinson’s disease (PD) through speech analysis is a promising area of research, as speech impairments are often one of the early signs of the disease. This study investigates the efficacy of fine-tuning pre-trained Automatic Speech Recognition (ASR) models, specifically Wav2Vec 2.0 and HuBERT, for PD detection using transfer learning. These models, pre-trained on large unlabeled datasets, can be capable of learning rich speech representations that capture acoustic markers of PD. The study also proposes the integration of a supervised contrastive (SupCon) learning approach to enhance the models’ ability to distinguish PD-specific features. Additionally, the proposed ASR-based features were compared against two common acoustic feature sets: mel-frequency cepstral coefficients (MFCCs) and the extended Geneva minimalistic acoustic parameter set (eGeMAPS) as a baseline. We also employed a gradient-based method, Grad-CAM, to visualize important speech regions contributing to the models’ predictions. The experiments, conducted using the NeuroVoz dataset, demonstrated that features extracted from the pre-trained ASR models exhibited superior performance compared to the baseline features. The results also reveal that the method integrating SupCon consistently outperforms traditional cross-entropy (CE)-based models. Wav2Vec 2.0 and HuBERT with SupCon achieved the highest F1 scores of 90.0% and 88.99%, respectively. Additionally, their AUC scores in the ROC analysis surpassed those of the CE models, which had comparatively lower AUCs, ranging from 0.84 to 0.89. These results highlight the potential of ASR-based models as scalable, non-invasive tools for diagnosing and monitoring PD, offering a promising avenue for the early detection and management of this debilitating condition. Full article

Large pre-trained models (PLMs) have provided tremendous opportunities and potentialities for multimodal fake news detection. However, existing multimodal fake news detection methods never manipulate the token-wise hierarchical semantics of news yielded from PLMs and extremely rely on contrastive learning but ignore the symmetry between text and image in terms of the abstract level. This paper proposes a novel multimodal fake news detection method that helps to balance the understanding between text and image via (1) designing a global-token across-attention mechanism to capture the correlations between global text and tokenwise image representations (or tokenwise text and global image representations) obtained from BERT and ViT; (2) proposing a QK-sharing strategy within cross-attention to enforce model symmetry that reduces information redundancy and accelerates fusion without sacrificing representational power; (3) deploying a semantic augmentation module that systematically extracts token-wise multilayered text semantics from stacked BERT blocks via CNN and Bi-LSTM layers, thereby rebalancing abstract-level disparities by symmetrically enriching shallow and deep textual signals. We also prove the effectiveness of our approach by comparing it with four state-of-the-art baselines. All the comparisons were conducted using three widely adopted multimodal fake news datasets. The results show that our approach outperforms the benchmarks by 0.8% in accuracy and 2.2% in F1-score on average across the three datasets, which demonstrates a symmetric, token-centric fusion of fine-grained semantic fusion, thereby driving more robust fake news detection. Full article

Semantic text chunking refers to segmenting text into coherently semantic chunks, i.e., into sets of statements that are semantically related. Semantic chunking is an essential pre-processing step in various NLP tasks e.g., document summarization, sentiment analysis and question answering. In this paper, we propose a hybrid chunking; two-steps semantic text chunking method that combines the effectiveness of unsupervised semantic text chunking based on the similarities between sentences embeddings and the pre-trained language models (PLMs) especially BERT by fine-tuning the BERT on semantic textual similarity task (STS) to provide a flexible and effective semantic text chunking. We evaluated the proposed method in English and Arabic. To the best of our knowledge, there is an absence of an Arabic dataset created to assess semantic text chunking at this level. Therefore, we created an AraWiki50k to evaluate our proposed text chunking method inspired by an existing English dataset. Our experiments showed that exploiting the fine-tuned pre-trained BERT on STS enhances results over unsupervised semantic chunking by an average of 7.4 in the PK metric and by an average of 11.19 in the WindowDiff metric on four English evaluation datasets, and 0.12 in the PK and 2.29 in the WindowDiff for the Arabic dataset. Full article

Despite remarkable advances in neural language models, a substantial gap remains in precisely interpreting the complex semantics of Electronic Medical Records (EMR). We propose Contrastive Representations Pre-Training (CRPT) to address this gap, replacing the conventional Next Sentence Prediction task’s cross-entropy loss with contrastive loss and incorporating whole-word masking to capture multi-token domain-specific terms better. We also introduce a carefully designed negative sampling strategy that balances intra-document and cross-document sentences, enhancing the model’s discriminative power. Implemented atop a BERT-based architecture and evaluated on the Biomedical Language Understanding Evaluation (BLUE) benchmark, our Discharge Summary CRPT model achieves significant performance gains, including a natural language inference precision of 0.825 and a sentence similarity score of 0.775. We further extend our approach through Bio+Discharge Summary CRPT, combining biomedical and clinical corpora to boost downstream performance across tasks. Our framework demonstrates robust interpretive capacity in clinical texts by emphasizing sentence-level semantics and domain-aware masking. These findings underscore CRPT’s potential for advancing semantic accuracy in healthcare applications and open new avenues for integrating larger negative sample sets, domain-specific masking techniques, and multi-task learning paradigms. Full article

This study addresses the sentiment classification of short texts in teaching evaluations. To mitigate concerns regarding data security in cloud-based sentiment analysis and to overcome the limited feature extraction capacity of traditional deep-learning methods, we propose a distributed symmetric approximate homomorphic hybrid sentiment classification model, denoted BHE+ALBERT-Mixplus. To enable homomorphic encryption of non-polynomial functions within the ALBERT-Mixplus architecture—a mixing-and-enhancement variant of ALBERT—we introduce the BHE (BERT-based Homomorphic Encryption) algorithm. The BHE establishes a distributed symmetric approximation workflow, constructing a cloud–user symmetric encryption framework. Within this framework, simplified computations and mathematical approximations are applied to handle non-polynomial operations (e.g., GELU, Softmax, and LayerNorm) under the CKKS homomorphic-encryption scheme. Consequently, the ALBERT-Mixplus model can securely perform classification on encrypted data without compromising utility. To improve feature extraction and enhance prediction accuracy in sentiment classification, ALBERT-Mixplus incorporates two core components: 1. A meta-information extraction layer, employing a lightweight pre-trained ALBERT model to capture extensive general semantic knowledge and thereby bolster robustness to noise. 2. A hybrid feature-extraction layer, which fuses a bidirectional gated recurrent unit (BiGRU) with a multi-scale convolutional neural network (MCNN) to capture both global contextual dependencies and fine-grained local semantic features across multiple scales. Together, these layers enrich the model’s deep feature representations. Experimental results on the TAD-2023 and SST-2 datasets demonstrate that BHE+ALBERT-Mixplus achieves competitive improvements in key evaluation metrics compared to mainstream models, despite a slight increase in computational overhead. The proposed framework enables secure analysis of diverse student feedback while preserving data privacy. This allows marginalized student groups to benefit equally from AI-driven insights, thereby embodying the principles of educational equity and inclusive education. Moreover, through its innovative distributed encryption workflow, the model enhances computational efficiency while promoting environmental sustainability by reducing energy consumption and optimizing resource allocation. Full article

During crises, people use X to share real-time updates. These posts reveal public sentiment and evolving emergency situations. However, the changing sentiment in tweets coupled with anomalous patterns may indicate significant events, misinformation or emerging hazards that require timely detection. By using a neural network, and employing deep learning techniques for crisis observation, this study proposes a pipeline for sentiment analysis and anomaly detection in crisis-related tweets. The authors used pre-trained BERT to classify tweet sentiment. For sentiment anomaly detection, autoencoders and recurrent neural networks (RNNs) with an attention mechanism were applied to capture sequential relationships and identify irregular sentiment patterns that deviate from standard crisis talk. Experimental results show that neural networks are more accurate than traditional machine learning methods for both sentiment categorization and anomaly detection tasks, with higher precision and recall for identifying sentiment shifts in the public. This study indicates that neural networks can be used for crisis management and the early detection of significant sentiment anomalies. This could be beneficial to emergency responders and policymakers and support data-driven decisions. Full article

Wine production is an important sector of the food industry in Bulgaria, contributing to both economic development and cultural heritage. The present study aims to show how natural language processing (NLP) and machine learning methods can be applied to analyze expert-written Bulgarian wine descriptions and to extract patterns related to wine quality and style. Based on a bilingual dataset of reviews (in Bulgarian and English), semantic analysis, classification, regression and clustering models were used, which combine textual and structured data. The descriptions were transformed into numerical representations using a pre-trained language model (BERT), after which algorithms were used to predict style categories and ratings. Additional sentiment and segmentation analyses revealed differences between wine types, and clustering identified thematic structures in the expert language. The comparison between predefined styles and automatically derived clusters was evaluated using metrics such as Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI). The resulting analysis shows that text descriptions contain valuable information that allows for automated wine profiling. These findings can be applied by a wide range of stakeholders—researchers, producers, retailers, and marketing specialists. Full article