Entropy

Research

20 pages, 1855 KB

Open AccessArticle

Automated Working Alliance Assessment in Psychological Counseling Using Gemini and XGBoost

by Yuexi Li, Ningtao Sun, Zhuoxi Mai, Dalin Li, Guifang Fu and Xueling Yang

Entropy 2026, 28(6), 699; https://doi.org/10.3390/e28060699 - 17 Jun 2026

Viewed by 336

Session dialogue assessment based on machine learning is gradually becoming an effective solution for therapeutic alliance measurement which is an important factor for successful psychotherapy. However, most existing models assume clean and pre-structured dialogue transcripts, whereas real-world counseling documentation often contains heterogeneous case [...] Read more.

Session dialogue assessment based on machine learning is gradually becoming an effective solution for therapeutic alliance measurement which is an important factor for successful psychotherapy. However, most existing models assume clean and pre-structured dialogue transcripts, whereas real-world counseling documentation often contains heterogeneous case reports. This gap limits the applicability of current automated assessment models in realistic documentation scenarios. In this work, we propose a framework for automated working alliance assessment from complex, multilingual reports. First, language-specific BERT models are fine-tuned to process case reports across different languages, enabling accurate speaker role delineation and dialogue structuring. Second, Gemini-2.5-Flash is leveraged to annotate the dialogues with working alliance ratings. Third, a hybrid feature representation strategy is then developed to jointly capture linguistic style and semantic content from the counseling dialogues. Furthermore, an entropy-based mutual information analysis is conducted to identify the most informative linguistic features. Finally, the extracted hybrid features serve as inputs to XGBoost for alliance assessment. In experiments, the proposed framework shows better performance in the comparison with SOTA methods and generalization ability. Full article

(This article belongs to the Special Issue Entropy in Machine Learning Applications, 2nd Edition)

► Show Figures

Figure 1

42 pages, 3547 KB

Open AccessArticle

Risk-Sensitive Machine Learning for Financial Decision Modeling Under Imbalanced Data: Evidence from Bank Telemarketing

by Bowen Dong, Xinyu Zhang, Yang Liu, Tianhui Zhang, Xianchen Liu, Lingmin Hou, Lingyi Meng, Zhen Guo and Aliya Mulati

Entropy 2026, 28(3), 354; https://doi.org/10.3390/e28030354 - 21 Mar 2026

Viewed by 1880

Abstract

Bank telemarketing campaigns often experience low subscription rates due to customer heterogeneity and severe class imbalance, which pose challenges for reliable predictive modeling. This study investigates a data-driven approach that integrates synthetic minority oversampling and cost-sensitive learning to improve the prediction of telemarketing [...] Read more.

Bank telemarketing campaigns often experience low subscription rates due to customer heterogeneity and severe class imbalance, which pose challenges for reliable predictive modeling. This study investigates a data-driven approach that integrates synthetic minority oversampling and cost-sensitive learning to improve the prediction of telemarketing outcomes. Experiments are conducted using the Portuguese Bank Marketing dataset, comprising 41,188 instances with a positive response rate of 11.3%. Eight machine learning models are evaluated under a unified preprocessing pipeline and five-fold stratified cross-validation, including Logistic Regression, Decision Tree, Random Forest, and Ensemble methods. The results show that Ensemble models, particularly CatBoost, XGBoost, and LightGBM, achieve improved performance compared with traditional baselines, with notable gains in minority-class recall and overall discrimination ability. The best-performing model attains an F1-score of 0.540, a recall of 0.812 for the positive class, and a ROC–AUC of 0.908. To enhance interpretability, SHAP-based analysis is applied to quantify feature contributions, identifying campaign duration, previous contact outcomes, and selected macroeconomic indicators as key predictors. These findings indicate that combining resampling strategies with cost-sensitive optimization provides a robust and transparent approach for learning from imbalanced telemarketing data, thereby supporting reproducible and data-driven financial decision-making by explicitly addressing difficulty in minority-class identification under imbalance and class imbalance under cross-entropy training in imbalanced banking data. Full article

(This article belongs to the Special Issue Entropy in Machine Learning Applications, 2nd Edition)

► Show Figures

Figure 1

26 pages, 3199 KB

Open AccessArticle

EDAER: Entropy-Driven Approach for Entity and Relation Extraction in Chinese Cyber Threat Intelligence

by Yong Li, Xiuping Li, Yangbai Zhang, Zhiqiang Liu, Xiaowei Li, Qi Xu and Xiaolin Chang

Entropy 2026, 28(3), 261; https://doi.org/10.3390/e28030261 - 27 Feb 2026

Viewed by 823

Abstract

Cyber threat intelligence (CTI) has been explored to strengthen system security via taking raw threat data from various data sources and transforming it into actionable insights that enable organizations to predict, detect, and respond to cyber threats. Named entity recognition (NER) and relation [...] Read more.

Cyber threat intelligence (CTI) has been explored to strengthen system security via taking raw threat data from various data sources and transforming it into actionable insights that enable organizations to predict, detect, and respond to cyber threats. Named entity recognition (NER) and relation extraction (RE) are the key tasks of CTI data mining. However, current CTI NER and/or RE research is mainly focused on English CTI, which is not directly transferable to Chinese CTI due to fundamental linguistic and terminological differences. Moreover, the existing limited studies on Chinese CTI do not effectively address uncertainty in predictions in low-resource scenarios where entities and relations are sparse. This work aims to improve the performance of NER and RE tasks in low-resource Chinese CTI scenarios, and we make two major contributions. The first is that we construct a Chinese CTI dataset, which includes 16 types of entities and 9 types of relations—more than those of the existing open-source dataset on Chinese CTI. The second is that we propose an entropy-driven approach for entity and relation (EDAER) extraction. EDAER is the first to combine the techniques of RoBERTa_wwm, Mamba, RDCNN and CRF to perform NER tasks. In addition, EDAER is the first to apply entropy to quantify the uncertainty of the model’s predictions in NER and RE tasks in Chinese CTI scenarios. Moreover, EDAER is the first to apply contrastive learning techniques in Chinese CTI scenarios to learn meaningful features by maximizing the similarity between positive samples and minimizing the similarity between negative samples. Extensive experimental results on public and our built datasets demonstrate that our proposed approach performs the best. These results show that (1) RoBERTa_wwwm significantly outperforms BERT on both NER and RE tasks; (2) Mamba outperforms BiLSTM on the NER task; (3) the entropy-based dynamic gating mechanism contributes to performance improvements in both NER and RE tasks; and (4) the uncertainty-guided contrastive learning mechanism is helpful for performance improvement in the NER task. Full article

(This article belongs to the Special Issue Entropy in Machine Learning Applications, 2nd Edition)

► Show Figures

Figure 1

25 pages, 18819 KB

Open AccessArticle

Application of the Two-Layer Regularized Gated Recurrent Unit (TLR-GRU) Model Enhanced by Sliding Window Features in Water Quality Parameter Prediction

by Xianhe Wang, Meiqi Liu, Ying Li, Adriano Tavares, Weidong Huang and Yanchun Liang

Entropy 2026, 28(2), 186; https://doi.org/10.3390/e28020186 - 6 Feb 2026

Cited by 1 | Viewed by 595

Abstract

Water quality monitoring is critical for public health, ecology, and economic sustainability, but traditional methods are limited by temporal-spatial coverage and cost, failing to meet real-time assessment needs. Deep learning for water quality prediction is often hindered by high complexity and noise in [...] Read more.

Water quality monitoring is critical for public health, ecology, and economic sustainability, but traditional methods are limited by temporal-spatial coverage and cost, failing to meet real-time assessment needs. Deep learning for water quality prediction is often hindered by high complexity and noise in raw time series. This study aims to address the high complexity and noise of hydrological time series by proposing a prediction framework integrating sliding window feature enhancement, principal component analysis (PCA), and a two-layer regularized gated recurrent unit (TLR-GRU). The core goal is to achieve high-precision real-time prediction of four key water quality parameters (dissolved oxygen (DO), ammonia nitrogen (

{NH}_{3} - N

), total phosphorus (TP), and total nitrogen (TN)) for aquaculture and irrigation. Sample entropy (SampEn,

m = 2

,

r = 0.2 \times std (X)

), a univariate complexity metric capturing intra-series pattern repetition, quantifies time series regularity, showing sliding windows reduce SampEn by filtering transient noise while retaining ecological patterns. This optimization synergizes with TLR-GRU’s regularization (L2, Dropout) to avoid overfitting. A total of 4970 water quality records (2020–2023, 4 h sampling interval) were collected from a monitoring station in a typical aquaculture-irrigated water body. After dimensionality reduction via PCA, experimental results demonstrate that the TLR-GRU model outperforms six state-of-the-art deep learning models (e.g., TLD-LSTM, WaveNet) on both the base dataset and the sliding window-enhanced dataset. On the latter, DO and TP test set

R^{2}

rise from 0.82 to 0.93 and 0.81 to 0.92, with RMSE decreasing by 49.4% and 55.6%, respectively. This framework supports water resource management, applicable to rivers and lakes beyond aquaculture. Future work will optimize the model and integrate multi-source data. Full article

(This article belongs to the Special Issue Entropy in Machine Learning Applications, 2nd Edition)

► Show Figures

Figure 1

18 pages, 1993 KB

Open AccessArticle

Prediction, Uncertainty Quantification, and ANN-Assisted Operation of Anaerobic Digestion Guided by Entropy Using Machine Learning

by Zhipeng Zhuang, Xiaoshan Liu, Jing Jin, Ziwen Li, Yanheng Liu, Adriano Tavares and Dalin Li

Entropy 2025, 27(12), 1233; https://doi.org/10.3390/e27121233 - 5 Dec 2025

Cited by 5 | Viewed by 1252

Abstract

Anaerobic digestion (AD) is a nonlinear and disturbance-sensitive process in which instability is often induced by feedstock variability and biological fluctuations. To address this challenge, this study develops an entropy-guided machine learning framework that integrates parameter prediction, uncertainty quantification, and entropy-based evaluation of [...] Read more.

Anaerobic digestion (AD) is a nonlinear and disturbance-sensitive process in which instability is often induced by feedstock variability and biological fluctuations. To address this challenge, this study develops an entropy-guided machine learning framework that integrates parameter prediction, uncertainty quantification, and entropy-based evaluation of AD operation. Using six months of industrial data (~10,000 samples), three models—support vector machine (SVM), random forest (RF), and artificial neural network (ANN)—were compared for predicting biogas yield, fermentation temperature, and volatile fatty acid (VFA) concentration. The ANN achieved the highest performance (accuracy = 96%, F1 = 0.95, root mean square error (RMSE) = 1.2 m³/t) and also exhibited the lowest prediction error entropy, indicating reduced uncertainty compared to RF and SVM. Feature entropy and permutation analysis consistently identified feed solids, organic matter, and feed rate as the most influential variables (>85% contribution), in agreement with the RF importance ranking. When applied as a real-time prediction and decision-support tool in the plant (“sensor → prediction → programmable logic controller (PLC)/operation → feedback”), the ANN model was associated with a reduction in gas-yield fluctuation from approximately ±18% to ±5%, a decrease in process entropy, and an improvement in operational stability of about 23%. Techno-economic and life-cycle assessments further indicated a 12–15 USD/t lower operating cost, 8–10% energy savings, and 5–7% CO₂ reduction compared with baseline operation. Overall, this study demonstrates that combining machine learning with entropy-based uncertainty analysis offers a reliable and interpretable pathway for more stable and low-carbon AD operation. Full article

(This article belongs to the Special Issue Entropy in Machine Learning Applications, 2nd Edition)

► Show Figures

Figure 1

29 pages, 565 KB

Open AccessArticle

Exploring Universal Domain Adaptation with CLIP Models: A Calibration Method

by Bin Deng

Entropy 2025, 27(12), 1213; https://doi.org/10.3390/e27121213 - 28 Nov 2025

Cited by 1 | Viewed by 1646

Abstract

CLIP models have shown their impressive learning and transfer capabilities in a wide range of visual tasks. It is, however, interesting that these foundation models have not been fully explored for Universal Domain Adaptation (UniDA). In this paper, we make comprehensive empirical studies [...] Read more.

CLIP models have shown their impressive learning and transfer capabilities in a wide range of visual tasks. It is, however, interesting that these foundation models have not been fully explored for Universal Domain Adaptation (UniDA). In this paper, we make comprehensive empirical studies of state-of-the-art UniDA methods using these foundation models. We first demonstrate that although the foundation models greatly improve the performance of the baseline method (which trains the models on the source data alone), existing UniDA methods struggle to improve over the baseline. This suggests that new research efforts are necessary for UniDA using these foundation models. Finally, we observe that calibration of CLIP models plays a key role in UniDA. To this end, we propose a very simple calibration method via automatic temperature scaling, which significantly enhances the baseline’s out-of-class detection capability. We show that a single learned temperature outperforms previous approaches in most benchmark tasks when adapting from CLIP models, excelling in evaluation metrics including H-score and a newly proposed Universal Classification Rate (UCR) metric. We hope that our investigation and the proposed simple framework can serve as a strong baseline to facilitate future studies in this field. Full article

(This article belongs to the Special Issue Entropy in Machine Learning Applications, 2nd Edition)

► Show Figures

Figure 1

30 pages, 2818 KB

Open AccessArticle

LAViTSPose: A Lightweight Cascaded Framework for Robust Sitting Posture Recognition via Detection– Segmentation–Classification

by Shu Wang, Adriano Tavares, Carlos Lima, Tiago Gomes, Yicong Zhang, Jiyu Zhao and Yanchun Liang

Entropy 2025, 27(12), 1196; https://doi.org/10.3390/e27121196 - 25 Nov 2025

Viewed by 1010

Abstract

Sitting posture recognition, defined as automatically localizing and categorizing seated human postures, has become essential for large-scale ergonomics assessment and longitudinal health-risk monitoring in classrooms and offices. However, in real-world multi-person scenes, pervasive occlusions and overlaps induce keypoint misalignment, causing global-attention backbones to [...] Read more.

Sitting posture recognition, defined as automatically localizing and categorizing seated human postures, has become essential for large-scale ergonomics assessment and longitudinal health-risk monitoring in classrooms and offices. However, in real-world multi-person scenes, pervasive occlusions and overlaps induce keypoint misalignment, causing global-attention backbones to fail to localize critical local structures. Moreover, annotation scarcity makes small-sample training commonplace, leaving models insufficiently robust to misalignment perturbations and thereby limiting cross-domain generalization. To address these challenges, we propose LAViTSPose, a lightweight cascaded framework for sitting posture recognition. Concretely, a YOLOR-based detector trained with a Range-aware IoU (RaIoU) loss yields tight person crops under partial visibility; ESBody suppresses cross-person leakage and estimates occlusion/head-orientation cues; a compact ViT head (MLiT) with Spatial Displacement Contact (SDC) and a learnable temperature (LT) mechanism performs skeleton-only classification with a local structural-consistency regularizer. From an information-theoretic perspective, our design enhances discriminative feature compactness and reduces structural entropy under occlusion and annotation scarcity. We conducted a systematic evaluation on the USSP dataset, and the results show that LAViTSPose outperforms existing methods on both sitting posture classification and face-orientation recognition while meeting real-time inference requirements. Full article

(This article belongs to the Special Issue Entropy in Machine Learning Applications, 2nd Edition)

► Show Figures

Figure 1

16 pages, 1005 KB

Open AccessArticle

A Two-Step Machine Learning Approach Integrating GNSS-Derived PWV for Improved Precipitation Forecasting

by Laura Profetto, Andrea Antonini, Luca Fibbi, Alberto Ortolani and Giovanna Maria Dimitri

Entropy 2025, 27(10), 1034; https://doi.org/10.3390/e27101034 - 2 Oct 2025

Cited by 6 | Viewed by 1864

Abstract

Global Navigation Satellite System (GNSS) meteorology has emerged as a valuable tool for atmospheric monitoring, providing high-resolution, near-real-time data that can significantly improve precipitation nowcasting. This study aims to enhance short-term precipitation forecasting by integrating GNSS-derived Precipitable Water Vapor (PWV)—a key indicator of [...] Read more.

Global Navigation Satellite System (GNSS) meteorology has emerged as a valuable tool for atmospheric monitoring, providing high-resolution, near-real-time data that can significantly improve precipitation nowcasting. This study aims to enhance short-term precipitation forecasting by integrating GNSS-derived Precipitable Water Vapor (PWV)—a key indicator of atmospheric moisture—with traditional meteorological observations. A novel two-step machine learning framework is proposed that combines a Random Forest (RF) model and a Long Short-Term Memory (LSTM) neural network. The RF model first estimates current precipitation based on PWV, surface weather parameters, and auxiliary atmospheric variables. Then, the LSTM network leverages temporal dependencies within the data to predict precipitation for the subsequent hour. This hybrid method capitalizes on the RF’s ability to model complex nonlinear relationships and the LSTM’s strength in handling time series data. The results demonstrate that the proposed approach improves forecasting accuracy, particularly during extreme weather events such as intense rainfall and thunderstorms, outperforming conventional models. By integrating GNSS meteorology with advanced machine learning techniques, this study offers a promising tool for meteorological services, early warning systems, and disaster risk management. The findings highlight the potential of GNSS-based nowcasting for real-time decision-making in weather-sensitive applications. Full article

(This article belongs to the Special Issue Entropy in Machine Learning Applications, 2nd Edition)

► Show Figures

Figure 1

21 pages, 753 KB

Open AccessArticle

Learnable Convolutional Attention Network for Unsupervised Knowledge Graph Entity Alignment

by Weishan Cai and Wenjun Ma

Entropy 2025, 27(9), 924; https://doi.org/10.3390/e27090924 - 3 Sep 2025

Cited by 3 | Viewed by 1672

Abstract

The success of current entity alignment (EA) tasks largely depends on the supervision information provided by labeled data. Considering the cost of labeled data, most supervised methods are challenging to apply in practical scenarios. Therefore, an increasing number of works based on contrastive [...] Read more.

The success of current entity alignment (EA) tasks largely depends on the supervision information provided by labeled data. Considering the cost of labeled data, most supervised methods are challenging to apply in practical scenarios. Therefore, an increasing number of works based on contrastive learning, active learning, or other deep learning techniques have been developed, to solve the performance bottleneck caused by the lack of labeled data. However, existing unsupervised EA methods still face certain limitations; either their modeling complexity is high or they fail to balance the effectiveness and practicality of alignment. To overcome these issues, we propose a learnable convolutional attention network for unsupervised entity alignment, named LCA-UEA. Specifically, LCA-UEA performs convolution operations before the attention mechanism, ensuring the acquisition of structural information and avoiding the superposition of redundant information. Then, to efficiently filter out invalid neighborhood information of aligned entities, LCA-UEA designs a relation structure reconstruction method based on potential matching relations, thereby enhancing the usability and scalability of the EA method. Notably, a similarity function based on consistency is proposed to better measure the similarity of candidate entity pairs. Finally, we conducted extensive experiments on three datasets of different sizes and types (cross-lingual and monolingual) to verify the superiority of LCA-UEA. Experimental results demonstrate that LCA-UEA significantly improved alignment accuracy, outperforming 25 supervised or unsupervised methods, and improving by 6.4% in Hits@1 over the best baseline in the best case. Full article

(This article belongs to the Special Issue Entropy in Machine Learning Applications, 2nd Edition)

► Show Figures

Figure 1

18 pages, 899 KB

Open AccessArticle

Hyperparameter Optimization EM Algorithm via Bayesian Optimization and Relative Entropy

by Dawei Zou, Chunhua Ma, Peng Wang and Yanqiu Geng

Entropy 2025, 27(7), 678; https://doi.org/10.3390/e27070678 - 25 Jun 2025

Cited by 2 | Viewed by 2125

Abstract

Hyperparameter optimization (HPO), which is also called hyperparameter tuning, is a vital component of developing machine learning models. These parameters, which regulate the behavior of the machine learning algorithm and cannot be directly learned from the given training data, can significantly affect the [...] Read more.

Hyperparameter optimization (HPO), which is also called hyperparameter tuning, is a vital component of developing machine learning models. These parameters, which regulate the behavior of the machine learning algorithm and cannot be directly learned from the given training data, can significantly affect the performance of the model. In the context of relevance vector machine hyperparameter optimization, we have used zero-mean Gaussian weight priors to derive iterative equations through evidence function maximization. For a general Gaussian weight prior and Bayesian linear regression, we similarly derive iterative reestimation equations for hyperparameters through evidence function maximization. Subsequently, after using relative entropy and Bayesian optimization, the aforementioned non-closed-form reestimation equations can be partitioned into E and M steps, providing a clear mathematical and statistical explanation for the iterative reestimation equations of hyperparameters. The experimental result shows the effectiveness of the EM algorithm of hyperparameter optimization, and the algorithm also has the merit of fast convergence, except that the covariance of the posterior distribution is a singular matrix, which affects the increase in the likelihood. Full article

(This article belongs to the Special Issue Entropy in Machine Learning Applications, 2nd Edition)

► Show Figures

Figure 1

21 pages, 7233 KB

Open AccessArticle

Advancing Traditional Dunhuang Regional Pattern Design with Diffusion Adapter Networks and Cross-Entropy

by Yihuan Tian, Tao Yu, Zuling Cheng and Sunjung Lee

Entropy 2025, 27(5), 546; https://doi.org/10.3390/e27050546 - 21 May 2025

Cited by 3 | Viewed by 2129

Abstract

To promote the inheritance of traditional culture, a variety of emerging methods rooted in machine learning and deep learning have been introduced. Dunhuang patterns, an important part of traditional Chinese culture, are difficult to collect in large numbers due to their limited availability. [...] Read more.

To promote the inheritance of traditional culture, a variety of emerging methods rooted in machine learning and deep learning have been introduced. Dunhuang patterns, an important part of traditional Chinese culture, are difficult to collect in large numbers due to their limited availability. However, existing text-to-image methods are computationally intensive and struggle to capture fine details and complex semantic relationships in text and images. To address these challenges, this paper proposes the Diffusion Adapter Network (DANet). It employs a lightweight adapter module to extract visual structural information, enabling the diffusion model to generate Dunhuang patterns with high accuracy, while eliminating the need for expensive fine-tuning of the original model. The attention adapter incorporates a multihead attention module (MHAM) to enhance image modality cues, allowing the model to focus more effectively on key information. A multiscale attention module (MSAM) is employed to capture features at different scales, thereby providing more precise generative guidance. In addition, an adaptive control mechanism (ACM) dynamically adjusts the guidance coefficients across feature layers to further enhance generation quality. In addition, incorporating a cross-entropy loss function enhances the model’s capability in semantic understanding and the classification of Dunhuang patterns. The DANet achieves state-of-the-art (SOTA) performance on the proposed Diversified Dunhuang Patterns Dataset (DDHP). Specifically, it attains a perceptual similarity score (LPIPS) of 0.498, a graph matching score (CLIP score) of 0.533, and a feature similarity score (CLIP-I) of 0.772. Full article

(This article belongs to the Special Issue Entropy in Machine Learning Applications, 2nd Edition)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Entropy in Machine Learning Applications, 2nd Edition

Share This Special Issue

Editor

Special Issue Information

Keywords

Benefits of Publishing in a Special Issue

Related Special Issue

Published Papers (11 papers)

Research

Further Information

Guidelines

MDPI Initiatives

Follow MDPI