Entropy

Research

21 pages, 887 KiB

Open AccessArticle

Detecting Signatures of Criticality Using Divergence Rate

by Tenzin Chan, De Wen Soh and Christopher Hillar

Entropy 2025, 27(5), 487; https://doi.org/10.3390/e27050487 (registering DOI) - 30 Apr 2025

Oftentimes in a complex system it is observed that as a control parameter is varied, there are certain intervals during which the system undergoes dramatic change. In biology especially, these signatures of criticality are thought to be connected with efficient computation and information [...] Read more.

Oftentimes in a complex system it is observed that as a control parameter is varied, there are certain intervals during which the system undergoes dramatic change. In biology especially, these signatures of criticality are thought to be connected with efficient computation and information processing. Guided by the classical theory of rate–distortion (RD) from information theory, we propose a measure for detecting and characterizing such phenomena from data. When applied to RD problems, the measure correctly identifies exact critical trade-off parameters emerging from the theory and allows for the discovery of new conjectures in the field. Other application domains include efficient sensory coding, machine learning generalization, and natural language. Our findings give support to the hypothesis that critical behavior is a signature of optimal processing. Full article

(This article belongs to the Special Issue Information-Theoretic Methods in Data Analytics)

► Show Figures

Figure 1

25 pages, 1002 KiB

Open AccessArticle

InfoMat: Leveraging Information Theory to Visualize and Understand Sequential Data

by Dor Tsur and Haim Permuter

Entropy 2025, 27(4), 357; https://doi.org/10.3390/e27040357 - 28 Mar 2025

Viewed by 224

Abstract

Despite the widespread use of information measures in analyzing probabilistic systems, effective visualization tools for understanding complex dependencies in sequential data are scarce. In this work, we introduce the information matrix (InfoMat), a novel and intuitive matrix representation of information transfer in sequential [...] Read more.

Despite the widespread use of information measures in analyzing probabilistic systems, effective visualization tools for understanding complex dependencies in sequential data are scarce. In this work, we introduce the information matrix (InfoMat), a novel and intuitive matrix representation of information transfer in sequential systems. InfoMat provides a structured visual perspective on mutual information decompositions, enabling the discovery of new relationships between sequential information measures and enhancing interpretability in time series data analytics. We demonstrate how InfoMat captures key sequential information measures, such as directed information and transfer entropy. To facilitate its application in real-world datasets, we propose both an efficient Gaussian mutual information estimator and a neural InfoMat estimator based on masked autoregressive flows to model more complex dependencies. These estimators make InfoMat a valuable tool for uncovering hidden patterns in data analytics applications, encompassing neuroscience, finance, communication systems, and machine learning. We further illustrate the utility of InfoMat in visualizing information flow in real-world sequential physiological data analysis and in visualizing information flow in communication channels under various coding schemes. By mapping visual patterns in InfoMat to various modes of dependence structures, we provide a data-driven framework for analyzing causal relationships and temporal interactions. InfoMat thus serves as both a theoretical and empirical tool for data-driven decision making, bridging the gap between information theory and applied data analytics. Full article

(This article belongs to the Special Issue Information-Theoretic Methods in Data Analytics)

► Show Figures

Figure 1

26 pages, 12992 KiB

Open AccessArticle

AlphaRouter: Bridging the Gap Between Reinforcement Learning and Optimization for Vehicle Routing with Monte Carlo Tree Searches

by Won-Jun Kim, Junho Jeong, Taeyeong Kim and Kichun Lee

Entropy 2025, 27(3), 251; https://doi.org/10.3390/e27030251 - 27 Feb 2025

Viewed by 678

Abstract

Deep reinforcement learning (DRL) as a routing problem solver has shown promising results in recent studies. However, an inherent gap exists between computationally driven DRL and optimization-based heuristics. While a DRL algorithm for a certain problem is able to solve several similar problem [...] Read more.

Deep reinforcement learning (DRL) as a routing problem solver has shown promising results in recent studies. However, an inherent gap exists between computationally driven DRL and optimization-based heuristics. While a DRL algorithm for a certain problem is able to solve several similar problem instances, traditional optimization algorithms focus on optimizing solutions to one specific problem instance. In this paper, we propose an approach, AlphaRouter, which solves routing problems while bridging the gap between reinforcement learning and optimization. Fitting to routing problems, our approach first proposes attention-enabled policy and value networks consisting of a policy network that produces a probability distribution over all possible nodes and a value network that produces the expected distance from any given state. We modify a Monte Carlo tree search (MCTS) for the routing problems, selectively combining it with the routing problems. Our experiments demonstrate that the combined approach is promising and yields better solutions compared to original reinforcement learning (RL) approaches without MCTS, with good performance comparable to classical heuristics. Full article

(This article belongs to the Special Issue Information-Theoretic Methods in Data Analytics)

► Show Figures

Figure 1

21 pages, 1531 KiB

Open AccessArticle

Dual-Regularized Feature Selection for Class-Specific and Global Feature Associations

by Chenchen Wang, Jun Wang, Yanfei Li, Chengkai Piao and Jinmao Wei

Entropy 2025, 27(2), 190; https://doi.org/10.3390/e27020190 - 13 Feb 2025

Viewed by 507

Abstract

Understanding feature associations is vital for selecting the most informative features. Existing methods primarily focus on global feature associations, which capture overall relationships across all samples. However, they often overlook class-specific feature interactions, which are essential for capturing locality features that may only [...] Read more.

Understanding feature associations is vital for selecting the most informative features. Existing methods primarily focus on global feature associations, which capture overall relationships across all samples. However, they often overlook class-specific feature interactions, which are essential for capturing locality features that may only be significant within certain classes. In this paper, we propose Dual-Regularized Feature Selection (DRFS), which incorporates two feature association regularizers to address both class-specific and global feature relationships. The class-specific regularizer captures the local geometric structure of features within each class. Meanwhile, the global regularizer utilizes a global feature similarity matrix to eliminate redundant features across classes. By combining these regularizers, DRFS selects features that preserve both local interactions within each class and global discriminative power, with each regularizer complementing the other to enhance feature selection. Experimental results on eight public real-world datasets demonstrate that DRFS outperforms existing methods in classification accuracy. Full article

(This article belongs to the Special Issue Information-Theoretic Methods in Data Analytics)

► Show Figures

Figure 1

16 pages, 293 KiB

Open AccessArticle

Adaptive CoCoLasso for High-Dimensional Measurement Error Models

by Qin Yu

Entropy 2025, 27(2), 97; https://doi.org/10.3390/e27020097 - 21 Jan 2025

Cited by 1 | Viewed by 631

Abstract

A significant portion of theoretical and empirical studies in high-dimensional regression have primarily concentrated on clean datasets. However, in numerous practical scenarios, data are often corrupted by missing values and measurement errors, which cannot be ignored. Despite the substantial progress in high-dimensional regression [...] Read more.

A significant portion of theoretical and empirical studies in high-dimensional regression have primarily concentrated on clean datasets. However, in numerous practical scenarios, data are often corrupted by missing values and measurement errors, which cannot be ignored. Despite the substantial progress in high-dimensional regression with contaminated covariates, methods that achieve an effective trade-off among prediction accuracy, feature selection, and computational efficiency remain significantly underexplored. We introduce adaptive convex conditioned Lasso (Adaptive CoCoLasso), offering a new approach that can handle high-dimensional linear models with error-prone measurements. This estimator combines a projection onto the nearest positive semi-definite matrix with an adaptively weighted

ℓ_{1}

penalty. Theoretical guarantees are provided by establishing error bounds for the estimators. The results from the synthetic data analysis indicate that the Adaptive CoCoLasso performs strongly in prediction accuracy and mean squared error, particularly in scenarios involving both additive and multiplicative noise in measurements. While the Adaptive CoCoLasso estimator performs comparably or is slightly outperformed by certain methods, such as Hard, in reducing the number of incorrectly identified covariates, its strength lies in offering a more favorable trade-off between prediction accuracy and sparse modeling. Full article

(This article belongs to the Special Issue Information-Theoretic Methods in Data Analytics)

17 pages, 4033 KiB

Open AccessArticle

Motor Fault Diagnosis Based on Convolutional Block Attention Module-Xception Lightweight Neural Network

by Fengyun Xie, Qiuyang Fan, Gang Li, Yang Wang, Enguang Sun and Shengtong Zhou

Entropy 2024, 26(9), 810; https://doi.org/10.3390/e26090810 - 23 Sep 2024

Cited by 1 | Viewed by 1309

Abstract

Electric motors play a crucial role in self-driving vehicles. Therefore, fault diagnosis in motors is important for ensuring the safety and reliability of vehicles. In order to improve fault detection performance, this paper proposes a motor fault diagnosis method based on vibration signals. [...] Read more.

Electric motors play a crucial role in self-driving vehicles. Therefore, fault diagnosis in motors is important for ensuring the safety and reliability of vehicles. In order to improve fault detection performance, this paper proposes a motor fault diagnosis method based on vibration signals. Firstly, the vibration signals of each operating state of the motor at different frequencies are measured with vibration sensors. Secondly, the characteristic of Gram image coding is used to realize the coding of time domain information, and the one-dimensional vibration signals are transformed into grayscale diagrams to highlight their features. Finally, the lightweight neural network Xception is chosen as the main tool, and the attention mechanism Convolutional Block Attention Module (CBAM) is introduced into the model to enforce the importance of the characteristic information of the motor faults and realize their accurate identification. Xception is a type of convolutional neural network; its lightweight design maintains excellent performance while significantly reducing the model’s order of magnitude. Without affecting the computational complexity and accuracy of the network, the CBAM attention mechanism is added, and Gram’s corner field is combined with the improved lightweight neural network. The experimental results show that this model achieves a better recognition effect and faster iteration speed compared with the traditional Convolutional Neural Network (CNN), ResNet, and Xception networks. Full article

(This article belongs to the Special Issue Information-Theoretic Methods in Data Analytics)

► Show Figures

Figure 1

19 pages, 7779 KiB

Open AccessArticle

A Study of Adjacent Intersection Correlation Based on Temporal Graph Attention Network

by Pengcheng Li, Baotian Dong and Sixian Li

Entropy 2024, 26(5), 390; https://doi.org/10.3390/e26050390 - 30 Apr 2024

Viewed by 1355

Abstract

Traffic state classification and relevance calculation at intersections are both difficult problems in traffic control. In this paper, we propose an intersection relevance model based on a temporal graph attention network, which can solve the above two problems at the same time. First, [...] Read more.

Traffic state classification and relevance calculation at intersections are both difficult problems in traffic control. In this paper, we propose an intersection relevance model based on a temporal graph attention network, which can solve the above two problems at the same time. First, the intersection features and interaction time of the intersections are regarded as input quantities together with the initial labels of the traffic data. Then, they are inputted into the temporal graph attention (TGAT) model to obtain the classification accuracy of the target intersections in four states—free, stable, slow moving, and congested—and the obtained neighbouring intersection weights are used as the correlation between the intersections. Finally, it is validated by VISSIM simulation experiments. In terms of classification accuracy, the TGAT model has a higher classification accuracy than the three traditional classification models and can cope well with the uneven distribution of the number of samples. The information gain algorithm from the information entropy theory was used to derive the average delay as the most influential factor on intersection status. The correlation from the TGAT model positively correlates with traffic flow, making it interpretable. Using this correlation to control the division of subareas improves the road network’s operational efficiency more than the traditional correlation model does. This demonstrates the effectiveness of the TGAT model’s correlation. Full article

(This article belongs to the Special Issue Information-Theoretic Methods in Data Analytics)

► Show Figures

Figure 1

16 pages, 510 KiB

Open AccessArticle

A Comprehensive Framework for Measuring the Immediate Impact of TV Advertisements: TV-Impact

by Afra Arslan, Koray Tecimer, Hacer Turgut, Ömür Bali, Arda Yücel, Gülfem Isiklar Alptekin and Günce Keziban Orman

Entropy 2024, 26(2), 109; https://doi.org/10.3390/e26020109 - 25 Jan 2024

Cited by 2 | Viewed by 3291

Abstract

Measuring the immediate impact of television advertisements (TV ads) on online traffic poses significant challenges in many aspects. Nonetheless, a comprehensive consideration is essential to fully grasp consumer reactions to TV ads. So far, the measurement of this effect has not been studied [...] Read more.

Measuring the immediate impact of television advertisements (TV ads) on online traffic poses significant challenges in many aspects. Nonetheless, a comprehensive consideration is essential to fully grasp consumer reactions to TV ads. So far, the measurement of this effect has not been studied to a large extent. Existing studies have either determined how a specific focus group, i.e., toddlers, people of a certain age group, etc., react to ads via simple statistical tests using a case study approach or have examined the effects of advertising with simple regression models. This study introduces a comprehensive framework called TV-Impact. The framework uses a Bayesian structural time-series model called CausalImpact. There are additional novel approaches developed within the framework. One of the novelties of TV-Impact lies in its dynamic algorithm for selecting control variables which are supporting data sources and presumed to be unaffected by TV ads. In addition, we proposed the concept of Group Ads to combine overlapping ads into a single ad structure. Then, Random Forest Regressor, which is a commonly preferred supervised learning method, is used to decompose the impact into single ads. The TV-Impact framework was applied to the data of iLab, a venture company in Turkey, and manages its companies’ advertising strategies. The findings reveal that the TV-Impact model positively influenced the companies’ strategies for allocating their TV advertisement budgets and increased the amount of traffic driven to company websites, serving as an effective decision support system. Full article

(This article belongs to the Special Issue Information-Theoretic Methods in Data Analytics)

► Show Figures

Figure 1

15 pages, 1153 KiB

Open AccessArticle

Information Difference of Transfer Entropies between Head Motion and Eye Movement Indicates a Proxy of Driving

by Runlin Zhang, Qing Xu, Shunbo Wang, Simon Parkinson and Klaus Schoeffmann

Entropy 2024, 26(1), 3; https://doi.org/10.3390/e26010003 - 19 Dec 2023

Cited by 2 | Viewed by 1659

Abstract

Visual scanning is achieved via head motion and gaze movement for visual information acquisition and cognitive processing, which plays a critical role in undertaking common sensorimotor tasks such as driving. The coordination of the head and eyes is an important human behavior to [...] Read more.

Visual scanning is achieved via head motion and gaze movement for visual information acquisition and cognitive processing, which plays a critical role in undertaking common sensorimotor tasks such as driving. The coordination of the head and eyes is an important human behavior to make a key contribution to goal-directed visual scanning and sensorimotor driving. In this paper, we basically investigate the two most common patterns in eye–head coordination: “head motion earlier than eye movement” and “eye movement earlier than head motion”. We utilize bidirectional transfer entropies between head motion and eye movements to determine the existence of these two eye–head coordination patterns. Furthermore, we propose a unidirectional information difference to assess which pattern predominates in head–eye coordination. Additionally, we have discovered a significant correlation between the normalized unidirectional information difference and driving performance. This result not only indicates the influence of eye–head coordination on driving behavior from a computational perspective but also validates the practical significance of our approach utilizing transfer entropy for quantifying eye–head coordination. Full article

(This article belongs to the Special Issue Information-Theoretic Methods in Data Analytics)

► Show Figures

Figure 1

15 pages, 1557 KiB

Open AccessArticle

Anomaly Detection Using an Ensemble of Multi-Point LSTMs

by Geonseok Lee, Youngju Yoon and Kichun Lee

Entropy 2023, 25(11), 1480; https://doi.org/10.3390/e25111480 - 26 Oct 2023

Cited by 4 | Viewed by 3507

Abstract

As technologies for storing time-series data such as smartwatches and smart factories become common, we are collectively accumulating a great deal of time-series data. With the accumulation of time-series data, the importance of time-series abnormality detection technology that detects abnormal patterns such as [...] Read more.

As technologies for storing time-series data such as smartwatches and smart factories become common, we are collectively accumulating a great deal of time-series data. With the accumulation of time-series data, the importance of time-series abnormality detection technology that detects abnormal patterns such as Cyber-Intrusion Detection, Fraud Detection, Social Networks Anomaly Detection, and Industrial Anomaly Detection is emerging. In the past, time-series anomaly detection algorithms have mainly focused on processing univariate data. However, with the development of technology, time-series data has become complicated, and corresponding deep learning-based time-series anomaly detection technology has been actively developed. Currently, most industries rely on deep learning algorithms to detect time-series anomalies. In this paper, we propose an anomaly detection algorithm with an ensemble of multi-point LSTMs that can be used in three cases of time-series domains. We propose our anomaly detection model that uses three steps. The first step is a model selection step, in which a model is learned within a user-specified range, and among them, models that are most suitable are automatically selected. In the next step, a collected output vector from M LSTMs is completed by stacking ensemble techniques of the previously selected models. In the final step, anomalies are finally detected using the output vector of the second step. We conducted experiments comparing the performance of the proposed model with other state-of-the-art time-series detection deep learning models using three real-world datasets. Our method shows excellent accuracy, efficient execution time, and a good F1 score for the three datasets, though training the LSTM ensemble naturally requires more time. Full article

(This article belongs to the Special Issue Information-Theoretic Methods in Data Analytics)

► Show Figures

Figure 1

18 pages, 29855 KiB

Open AccessArticle

Sentiment Analysis on Online Videos by Time-Sync Comments

by Jiangfeng Li, Ziyu Li, Xiaofeng Ma, Qinpei Zhao, Chenxi Zhang and Gang Yu

Entropy 2023, 25(7), 1016; https://doi.org/10.3390/e25071016 - 2 Jul 2023

Cited by 1 | Viewed by 2532

Abstract

Video highlights are welcomed by audiences, and are composed of interesting or meaningful shots, such as funny shots. However, video shots of highlights are currently edited manually by video editors, which is inconvenient and consumes an enormous amount of time. A way to [...] Read more.

Video highlights are welcomed by audiences, and are composed of interesting or meaningful shots, such as funny shots. However, video shots of highlights are currently edited manually by video editors, which is inconvenient and consumes an enormous amount of time. A way to help video editors locate video highlights more efficiently is essential. Since interesting or meaningful highlights in videos usually imply strong sentiments, a sentiment analysis model is proposed to automatically recognize sentiments of video highlights by time-sync comments. As the comments are synchronized with video playback time, the model detects sentiment information in time series of user comments. Moreover, in the model, a sentimental intensity calculation method is designed to compute sentiments of shots quantitatively. The experiments show that our approach improves the F1 score by 12.8% and overlapped number by 8.0% compared with the best existing method in extracting sentiments of highlights and obtaining sentimental intensities, which provides assistance for video editors in editing video highlights efficiently. Full article

(This article belongs to the Special Issue Information-Theoretic Methods in Data Analytics)

► Show Figures

Figure 1

18 pages, 1064 KiB

Open AccessArticle

Topic Discovery and Hotspot Analysis of Sentiment Analysis of Chinese Text Using Information-Theoretic Method

by Changlu Zhang, Haojie Fan, Jian Zhang, Qiong Yang and Liqian Tang

Entropy 2023, 25(6), 935; https://doi.org/10.3390/e25060935 - 13 Jun 2023

Cited by 2 | Viewed by 2253

Abstract

Currently, sentiment analysis is a research hotspot in many fields such as computer science and statistical science. Topic discovery of the literature in the field of text sentiment analysis aims to provide scholars with a quick and effective understanding of its research trends. [...] Read more.

Currently, sentiment analysis is a research hotspot in many fields such as computer science and statistical science. Topic discovery of the literature in the field of text sentiment analysis aims to provide scholars with a quick and effective understanding of its research trends. In this paper, we propose a new model for the topic discovery analysis of literature. Firstly, the FastText model is applied to calculate the word vector of literature keywords, based on which cosine similarity is applied to calculate keyword similarity, to carry out the merging of synonymous keywords. Secondly, the hierarchical clustering method based on the Jaccard coefficient is used to cluster the domain literature and count the literature volume of each topic. Thirdly, the information gain method is applied to extract the high information gain characteristic words of various topics, based on which the connotation of each topic is condensed. Finally, by conducting a time series analysis of the literature, a four-quadrant matrix of topic distribution is constructed to compare the research trends of each topic within different stages. The 1186 articles in the field of text sentiment analysis from 2012 to 2022 can be divided into 12 categories. By comparing and analyzing the topic distribution matrices of the two phases of 2012 to 2016 and 2017 to 2022, it is found that the various categories of topics have obvious research development changes in different phases. The results show that: ① Among the 12 categories, online opinion analysis of social media comments represented by microblogs is one of the current hot topics. ② The integration and application of methods such as sentiment lexicon, traditional machine learning and deep learning should be enhanced. ③ Semantic disambiguation of aspect-level sentiment analysis is one of the current difficult problems this field faces. ④ Research on multimodal sentiment analysis and cross-modal sentiment analysis should be promoted. Full article

(This article belongs to the Special Issue Information-Theoretic Methods in Data Analytics)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Information-Theoretic Methods in Data Analytics

Share This Special Issue

Special Issue Editor

Special Issue Information

Keywords

Benefits of Publishing in a Special Issue

Related Special Issue

Published Papers (12 papers)

Research

Further Information

Guidelines

MDPI Initiatives

Follow MDPI