Search Results (2,529)

The rising frequency of network intrusions has significantly impacted critical infrastructures, leading to an increased focus on the detection of malicious network traffic in recent years. However, traditional port-based and classical machine learning-based malicious network traffic detection methods suffer from a dependence on expert experience and limited generalizability. In this paper, we propose a malicious traffic detection method based on an efficient federated learning framework of Bidirectional Encoder Representations from Transformers (BERT), called MT-FBERT. It offers two major advantages over most existing approaches. First, MT-FBERT pretrains BERT using two pre-training tasks along with an overall pre-training loss on large-scale unlabeled network traffic, allowing the model to automatically learn generalized traffic representations, which do not require human experience to extract the behavior features or label the malicious samples. Second, MT-FBERT finetunes BERT for malicious network traffic detection through an efficient federated learning framework, which both protects the data privacy of critical infrastructures and reduces resource consumption by dynamically identifying and updating only the most significant neurons in the global model. Evaluation experiments on public datasets demonstrated that MT-FBERT outperforms state-of-the-art baselines in malicious network traffic detection. Full article

Traditional methods for predicting feeding amounts rely on historical data and experience but fail to account for non-linear fish growth and the influence of water quality and meteorological factors. This study presents a novel approach for sea bass feeding prediction based on Spearman + RF feature optimization and multi-scale feature fusion using a transformer model. A logistic growth curve model is used to analyze sea bass growth and establish the relationship between biomass and feeding amount. Spearman correlation analysis and random forest optimize the feature set for improved prediction accuracy. A dual-encoder structure incorporates historical feeding data and biomass along with water quality and meteorological information. Multi-scale feature fusion addresses time-scale inconsistencies between input variables The results showed that the MSE and MAE of the improved transformer model for sea bass feeding prediction were 0.42 and 0.31, respectively, which decreased by 43% in MSE and 33% in MAE compared to the traditional transformer model. Full article

With the growing global demand for clean energy, the accuracy of wind-power forecasting plays a vital role in ensuring the stable operation of power systems. However, wind-power generation is significantly influenced by meteorological conditions and is characterized by high uncertainty and multiscale fluctuations. Traditional recurrent neural network (RNN) and long short-term memory (LSTM) models, although capable of handling sequential data, struggle with modeling long-term temporal dependencies due to the vanishing gradient problem; thus, they are now rarely used. Recently, Transformer models have made notable progress in sequence modeling compared to RNNs and LSTM models. Nevertheless, when dealing with long wind-power sequences, their quadratic computational complexity (O(L²)) leads to low efficiency, and their global attention mechanism often fails to capture local periodic features accurately, tending to overemphasize redundant information while overlooking key temporal patterns. To address these challenges, this paper proposes a wind-power forecasting method based on dimension-transformed collaborative multidimensional multiscale attention (DTCMMA). This method first employs fast Fourier transform (FFT) to automatically identify the main periodic components in wind-power data, reconstructing the one-dimensional time series as a two-dimensional spatiotemporal representation, thereby explicitly encoding periodic features. Based on this, a collaborative multidimensional multiscale attention (CMMA) mechanism is designed, which hierarchically integrates channel, spatial, and pixel attention to adaptively capture complex spatiotemporal dependencies. Considering the geometric characteristics of the reconstructed data, asymmetric convolution kernels are adopted to enhance feature extraction efficiency. Experiments on multiple wind-farm datasets and energy-related datasets demonstrate that DTCMMA outperforms mainstream methods such as Transformer, iTransformer, and TimeMixer in long-sequence forecasting tasks, achieving improvements in MSE performance by 34.22%, 2.57%, and 0.51%, respectively. The model’s training speed also surpasses that of the fastest baseline by 300%, significantly improving both prediction accuracy and computational efficiency. This provides an efficient and accurate solution for wind-power forecasting and contributes to the further development and application of wind energy in the global energy mix. Full article

This study proposes an interpretable hybrid deep learning framework for classifying attention deficit hyperactivity disorder (ADHD) using EEG signals recorded during cognitively demanding tasks. The core architecture integrates convolutional neural networks (CNNs), gated recurrent units (GRUs), and long short-term memory (LSTM) layers to jointly capture spatial and temporal dynamics. In addition to the final hybrid architecture, the CNN–GRU–LSTM model alone demonstrates excellent accuracy (99.63%) with minimal variance, making it a strong baseline for clinical applications. To evaluate the role of global attention mechanisms, transformer encoder models with two and three attention blocks, along with a spatiotemporal transformer employing 2D positional encoding, are benchmarked. A hybrid CNN–RNN–transformer model is introduced, combining convolutional, recurrent, and transformer-based modules into a unified architecture. To enhance interpretability, SHapley Additive exPlanations (SHAP) are employed to identify key EEG channels contributing to classification outcomes. Experimental evaluation using stratified five-fold cross-validation demonstrates that the proposed hybrid model achieves superior performance, with average accuracy exceeding 99.98%, F1-scores above 0.9999, and near-perfect AUC and Matthews correlation coefficients. In contrast, transformer-only models, despite high training accuracy, exhibit reduced generalization. SHAP-based analysis confirms the hybrid model’s clinical relevance. This work advances the development of transparent and reliable EEG-based tools for pediatric ADHD screening. Full article

This study investigates the ability of six pre-trained sentence transformers to organize medical knowledge by performing unsupervised clustering on 70 high-level Medical Subject Headings (MeSH) terms across seven medical specialties. We evaluated models from different pre-training paradigms: general-purpose, domain-adapted, and from-scratch domain-specific. The results reveal a clear performance hierarchy. A top tier of models, including the general-purpose MPNet and the domain-adapted BioBERT and RoBERTa, produced highly coherent, specialty-aligned clusters (Adjusted Rand Index > 0.80). Conversely, models pre-trained from scratch on specialized corpora, such as PubMedBERT and BioClinicalBERT, performed poorly (Adjusted Rand Index < 0.51), with BioClinicalBERT yielding a disorganized clustering. These findings challenge the assumption that domain-specific pre-training guarantees superior performance for all semantic tasks. We conclude that model architecture, alignment between the pre-training objective and the downstream task, and the nature of the training data are more critical determinants of success for creating semantically coherent embedding spaces for medical concepts. Full article

Detecting biased language in large-scale corpora, such as the Wiki Neutrality Corpus, is essential for promoting neutrality in digital content. This study systematically evaluates a range of machine learning (ML) and deep learning (DL) models for the detection of biased and pre-conditioned phrases. Conventional classifiers, including Extreme Gradient Boosting (XGBoost), Light Gradient-Boosting Machine (LightGBM), and Categorical Boosting (CatBoost), are compared with advanced neural architectures such as Bidirectional Encoder Representations from Transformers (BERT), Long Short-Term Memory (LSTM) networks, and Generative Adversarial Networks (GANs). A novel hybrid architecture is proposed, integrating DistilBERT, LSTM, and GANs within a unified framework. Extensive experimentation with intermediate variants DistilBERT + LSTM (without GAN) and DistilBERT + GAN (without LSTM) demonstrates that the fully integrated model consistently outperforms all alternatives. The proposed hybrid model achieves a cross-validation accuracy of 99.00%, significantly surpassing traditional baselines such as XGBoost (96.73%) and LightGBM (96.83%). It also exhibits superior stability, statistical significance (paired t-tests), and favorable trade-offs between performance and computational efficiency. The results underscore the potential of hybrid deep learning models for capturing subtle linguistic bias and advancing more objective and reliable automated content moderation systems. Full article

As a plant-specific peroxidase family, class III peroxidase (PRX) plays an important role in plant growth, development, and stress response. In this study, a preliminary functional analysis of CqPRX9L1 was conducted. Bioinformatics analysis revealed that CqPRX9L1 encodes a 349-amino acid protein belonging to the plant-peroxidase-like superfamily, featuring a transmembrane domain and cytoplasmic localization. The promoter region of CqPRX9L1 harbors various cis-acting elements associated with stress responses, hormone signaling, light regulation, and meristem-specific expression. The tissue-specific expression pattern of the CqPRX9L1 gene and its characteristics in response to different stresses were explored using subcellular localization, quantitative real-time PCR (qRT-PCR), and heterologous transformation into Arabidopsis thaliana. The results showed that CqPRX9L1, with a transmembrane structure, was localized in the cytoplasm, which encodes 349 amino acids and belongs to the plant-peroxisome-like superfamily. The promoter region contains stress-response elements, hormone-response elements, light-response elements, and meristem expression-related elements. The expression of CqPRX9L1 was relatively higher in ears and roots at the panicle stage than in stems and leaves. CqPRX9L1 showed a dynamic expression pattern of first decreasing and then increasing under abiotic stresses such as 15% PEG 6000, low temperature, and salt damage, with differences in response time and degree. CqPRX9L1 plays an important role in response to abiotic stress by affecting the activity of antioxidant enzymes such as superoxide dismutase (SOD) and peroxidase (POD), as well as the synthesis and decomposition of proline (Pro). CqPRX9L1 also affects plant bolting and flowering by regulating key flowering genes (such as FT and AP1) and gibberellin (GA)-related pathways. The results establish a foundation for revealing the functions and molecular mechanisms of the CqPRX9L1 gene. Full article

The transformer-based deep learning approach represents the current state-of-the-art in machine translation (MT) research. Large-scale pretrained transformer models produce state-of-the-art performance across a wide range of MT tasks for many languages. However, such deep neural network (NN) models are often data-, compute-, space-, power-, and energy-hungry, typically requiring powerful GPUs or large-scale clusters to train and deploy. As a result, they are often regarded as “non-green” and “unsustainable” technologies. Distilling knowledge from large deep NN models (teachers) to smaller NN models (students) is a widely adopted sustainable development approach in MT as well as in broader areas of natural language processing (NLP), including speech, and image processing. However, distilling large pretrained models presents several challenges. First, increased training time and cost that scales with the volume of data used for training a student model. This could pose a challenge for translation service providers (TSPs), as they may have limited budgets for training. Moreover, CO₂ emissions generated during model training are typically proportional to the amount of data used, contributing to environmental harm. Second, when querying teacher models, including encoder–decoder models such as NLLB, the translations they produce for low-resource languages may be noisy or of low quality. This can undermine sequence-level knowledge distillation (SKD), as student models may inherit and reinforce errors from inaccurate labels. In this study, the teacher model’s confidence estimation is employed to filter those instances from the distilled training data for which the teacher exhibits low confidence. We tested our methods on a low-resource Urdu-to-English translation task operating within a constrained training budget in an industrial translation setting. Our findings show that confidence estimation-based filtering can significantly reduce the cost and CO₂ emissions associated with training a student model without drop in translation quality, making it a practical and environmentally sustainable solution for the TSPs. Full article

Driven by blockchain technology, numerous industries are increasingly adopting smart contracts to enhance efficiency, reduce costs, and improve transparency. As a result, ensuring the security of smart contracts has become critical. Traditional detection methods often suffer from low efficiency, are prone to missing complex vulnerabilities, and have limited accuracy. Although deep learning approaches address some of these challenges, issues with both accuracy and efficiency remain in current solutions. To overcome these limitations, this paper proposes a symmetry-inspired solution that harmonizes bidirectional and generative semantic patterns. First, we generate distinct feature extraction segments for different vulnerabilities. We then use the Bidirectional Encoder Representations from Transformers (BERT) module to extract original semantic features from these segments and the Generative Pre-trained Transformer (GPT) module to extract generative semantic features. Finally, the two sets of semantic features are fused using a multi-attention mechanism and input into a classifier for result prediction. Our method was tested on three datasets, achieving F1 scores of 93.33%, 93.65%, and 92.31%, respectively. The results demonstrate that our approach outperforms most existing methods in smart contract detection. Full article

Accurate segmentation of key tumor subregions in adult gliomas from Magnetic Resonance Imaging (MRI) is of critical importance for brain tumor diagnosis, treatment planning, and prognosis. However, this task remains poorly investigated and highly challenging due to the considerable variability in shape and appearance of these areas across patients. This study proposes a novel Deep Learning architecture leveraging modality-specific encoding and attention-based refinement for the segmentation of glioma subregions, including peritumoral edema (ED), necrotic core (NCR), and enhancing tissue (ET). The model is trained and validated on the Brain Tumor Segmentation (BraTS) 2023 challenge dataset and benchmarked against a state-of-the-art transformer-based approach. Our architecture achieves promising results, with Dice scores of 0.78, 0.86, and 0.88 for NCR, ED, and ET, respectively, outperforming SegFormer3D while maintaining comparable model complexity. To ensure a comprehensive evaluation, performance was also assessed on standard composite tumor regions, i.e., tumor core (TC) and whole tumor (WT). The statistically significant improvements obtained on all regions highlight the effectiveness of integrating complementary modality-specific information and applying channel-wise feature recalibration in the proposed model. Full article

Currently, quantum steganography schemes utilizing the least significant bit (LSB) approach are primarily optimized for fixed-point data processing, yet they encounter precision limitations when handling extended floating-point data structures owing to quantization error accumulation. To overcome precision constraints in quantum data hiding, the EPlsb-MFQS and MVlsb-MFQS quantum steganography algorithms are constructed based on the LSB approach in this study. The multichannel floating-point quantum representation of digital signals (MFQS) model enhances information hiding by augmenting the number of available channels, thereby increasing the embedding capacity of the LSB approach. Firstly, we analyze the limitations of fixed-point signals steganography schemes and propose the conventional quantum steganography scheme based on the LSB approach for the MFQS model, achieving enhanced embedding capacity. Moreover, the enhanced embedding efficiency of the EPlsb-MFQS algorithm primarily stems from the superposition probability adjustment of the LSB approach. Then, to prevent an unauthorized person easily extracting secret messages, we utilize channel qubits and position qubits as novel carriers during quantum message encoding. The secret message is encoded into the signal’s qubits of the transmission using a particular modulo value rather than through sequential embedding, thereby enhancing the security and reducing the time complexity in the MVlsb-MFQS algorithm. However, this algorithm in the spatial domain has low robustness and security. Therefore, an improved method of transferring the steganographic process to the quantum Fourier transformed domain to further enhance security is also proposed. This scheme establishes the essential building blocks for quantum signal processing, paving the way for advanced quantum algorithms. Compared with available quantum steganography schemes, the proposed steganography schemes achieve significant improvements in embedding efficiency and security. Finally, we theoretically delineate, in detail, the quantum circuit design and operation process. Full article

The accurate prediction of soil organic matter (SOM) content is essential for promoting sustainable soil management and addressing global climate change. Due to multiple factors such as topography and climate, especially in mountainous areas, SOM spatial prediction faces significant challenges. The main novelty of this study lies in proposing a geographic positional encoding mechanism that embeds geographic location information into the feature representation of a Transformer model. The encoder structure is further modified to enhance spatial awareness, resulting in the development of the Geo-Positional Transformer (GPTransformer). Furthermore, this model is integrated with a 1D-CNN to form a dual-branch neural network called the Geo-Positional Transformer-CNN (GPTransCNN). This study collected 1490 topsoil samples (0–20 cm) from cultivated land in Longshan County to develop a predictive model for mapping the spatial distribution of SOM across the entire cultivated area. Different models were comprehensively evaluated through ten-fold cross-validation, ablation experiments, and uncertainty analysis. The results show that GPTransCNN has the best performance, with an R² improvement of approximately 43% over the Transformer, 19% over the GPTransformer, and 15% over the 1D-CNN. This study demonstrates that by incorporating geographic positional information, GPTransCNN effectively combines the global modeling capabilities of the GPTransformer with the local feature extraction strengths of the 1D-CNN, which can improve the accuracy of SOM mapping in mountainous areas. This approach provides data support for sustainable soil management and decision-making in response to global climate change. Full article

The address is a key element in the construction of smart cities. When receiving reports from citizens, public security officers need to quickly and accurately locate a crime scene based on the address provided by the reporter. The address from the reporter may be a standard address or it may be a point of interest, abbreviation, or common name. The difficulty in converting the address into a standard address can be solved through the analysis of address elements and address matching. We developed a bidirectional encoder representations from transformers (BERT)-based address feature resolution method and an address-matching algorithm. On this basis, a police force address-matching system for City A was designed and implemented. A Web application system was also developed based on the address database of City A. The developed address resolution and matching method with the database maintenance module successfully matched the reported address to the standard one. Full article

Semantic segmentation is a vast field with many contributions, which can be difficult to organize and comprehend due to the amount of research available. Advancements in technology and processing power over the past decade have led to a significant increase in the number of developed models and architectures. This paper provides a brief perspective on 2D segmentation by summarizing the mechanisms of various neural network models and the tools and datasets used for their training, testing, and evaluation. Additionally, this paper discusses methods for identifying new architectures, such as Neural Architecture Search, and explores the emerging research field of continuous learning, which aims to develop models capable of learning continuously from new data. Full article

The real-time acquisition of an accurate underwater sound velocity profile (SSP) is crucial for tracking the propagation trajectory of underwater acoustic signals, making it play a key role in ocean communication positioning. SSPs can be directly measured by instruments or inverted leveraging sound field data. Although measurement techniques provide a good accuracy, they are constrained by limited spatial coverage and require a substantial time investment. The inversion method based on the real-time measurement of acoustic field data improves operational efficiency but loses the accuracy of SSP estimation and suffers from limited spatial applicability due to its stringent requirements for ocean observation infrastructures. To achieve accurate long-term ocean SSP estimation independent of real-time underwater data measurements, we propose a semi-transformer neural network (STNet) specifically designed for simulating sound velocity distribution patterns from the perspective of time series prediction. The proposed network architecture incorporates an optimized self-attention mechanism to effectively capture long-range temporal dependencies within historical sound velocity time-series data, facilitating an accurate estimation of current SSPs or prediction of future SSPs. Through the architectural optimization of the transformer framework and integration of a time encoding mechanism, STNet could effectively improve computational efficiency. For long-term forecasting (using the Pacific Ocean as a case study), STNet achieved an annual average RMSE of 0.5811 m/s, outperforming the best baseline model, H-LSTM, by 26%. In short-term forecasting for the South China Sea, STNet further reduced the RMSE to 0.1385 m/s, demonstrating a 51% improvement over H-LSTM. Comparative experimental results revealed that STNet outperformed state-of-the-art models in predictive accuracy and maintained good computational efficiency, demonstrating its potential for enabling accurate long-term full-depth ocean SSP forecasting. Full article