Search Results (15)

In this paper, we study the beamforming design for multiple-input multiple-output (MIMO) ISAC systems, with the weighted mutual information (MI) comprising sensing and communication perspectives adopted as the performance metric. In particular, the weighted sum of the communication mutual information and the sensing mutual information is shown to asymptotically converge to a deterministic limit when the number of transmitting and receiving antennas grow to infinity. This deterministic limit is derived by utilizing the operator-valued free probability theory. Subsequently, an efficient projected gradient ascent (PGA) algorithm is proposed to optimize the transmit beamforming matrix with the aim of maximizing the weighted asymptotic MI. Numerical results validate that the derived closed-form expression matches well with the Monte Carlo simulation results and the proposed optimization algorithm is able to improve the weighted asymptotic MI significantly. We also illustrate the trade-off between asymptotic sensing and asymptotic communication MI. Full article

Feature selection methods are essential for accurate disease classification and identifying informative biomarkers. While information-theoretic methods have been widely used, they often exhibit limitations such as high computational costs. Our previously proposed method, ClearF, addresses these issues by using reconstruction error from low-dimensional embeddings as a proxy for the entropy term in the mutual information. However, ClearF still has limitations, including a nontransparent bottleneck layer selection process, which can result in unstable feature selection. To address these limitations, we propose ClearF++, which simplifies the bottleneck layer selection and incorporates feature-wise clustering to enhance biomarker detection. We compare its performance with other commonly used methods such as MultiSURF and IFS, as well as ClearF, across multiple benchmark datasets. Our results demonstrate that ClearF++ consistently outperforms these methods in terms of prediction accuracy and stability, even with limited samples. We also observe that employing the Deep Embedded Clustering (DEC) algorithm for feature-wise clustering improves performance, indicating its suitability for handling complex data structures with limited samples. ClearF++ offers an improved biomarker prioritization approach with enhanced prediction performance and faster execution. Its stability and effectiveness with limited samples make it particularly valuable for biomedical data analysis. Full article

Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently to gain insight into, among others, DNNs’ generalization ability. However, it is by no means obvious how to estimate the mutual information (MI) between each hidden layer and the input/desired output to construct the IP. For instance, hidden layers with many neurons require MI estimators with robustness toward the high dimensionality associated with such layers. MI estimators should also be able to handle convolutional layers while at the same time being computationally tractable to scale to large networks. Existing IP methods have not been able to study truly deep convolutional neural networks (CNNs). We propose an IP analysis using the new matrix-based Rényi’s entropy coupled with tensor kernels, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data. Our results shed new light on previous studies concerning small-scale DNNs using a completely new approach. We provide a comprehensive IP analysis of large-scale CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks. Full article

A new trend in deep learning, represented by Mutual Information Neural Estimation (MINE) and Information Noise Contrast Estimation (InfoNCE), is emerging. In this trend, similarity functions and Estimated Mutual Information (EMI) are used as learning and objective functions. Coincidentally, EMI is essentially the same as Semantic Mutual Information (SeMI) proposed by the author 30 years ago. This paper first reviews the evolutionary histories of semantic information measures and learning functions. Then, it briefly introduces the author’s semantic information G theory with the rate-fidelity function R(G) (G denotes SeMI, and R(G) extends R(D)) and its applications to multi-label learning, the maximum Mutual Information (MI) classification, and mixture models. Then it discusses how we should understand the relationship between SeMI and Shannon’s MI, two generalized entropies (fuzzy entropy and coverage entropy), Autoencoders, Gibbs distributions, and partition functions from the perspective of the R(G) function or the G theory. An important conclusion is that mixture models and Restricted Boltzmann Machines converge because SeMI is maximized, and Shannon’s MI is minimized, making information efficiency G/R close to 1. A potential opportunity is to simplify deep learning by using Gaussian channel mixture models for pre-training deep neural networks’ latent layers without considering gradients. It also discusses how the SeMI measure is used as the reward function (reflecting purposiveness) for reinforcement learning. The G theory helps interpret deep learning but is far from enough. Combining semantic information theory and deep learning will accelerate their development. Full article

Aiming at the difficulty of accurately identifying latent mechanical faults inside high-voltage shunt reactors (HVSRs), this paper proposes a new method for HVSR state feature extraction and intelligent diagnosis. The method integrates a modified complementary ensemble empirical mode decomposition (CEEMD)–permutation entropy–CEEMD (MCPCEEMD) method, mutual information theory (MI), multiscale fuzzy entropy (MFE), and an improved grasshopper optimization algorithm to optimize the probabilistic neural network (IGOA-PNN) model. First, we introduce MCPCEEMD for suppressing modal aliasing to decompose the HVSR raw vibration signals. Then, the correlation degree between the obtained intrinsic mode function (IMF) components and the HVSR original vibration signals is judged through MI, and the IMF with the highest correlation is selected for feature extraction. Furthermore, this study uses MFE to quantify the selected IMF. Finally, we employ piecewise inertial weights to improve GOA to select the best smoothing factor for PNN, and use the optimized IGOA-PNN model to identify feature subsets. The experimental results show that the proposed method can successfully diagnose different types and degrees of HVSR mechanical faults, and the identification accuracy rate reaches more than 98%. The high recognition accuracy of the proposed method is helpful for the state detection and field application of HVSRs. Full article

Real-time regional risk prediction can play a crucial role in preventing traffic accidents. Thus, this study established a lane-level real-time regional risk prediction model. Based on observed data, the least squares-support vector machines (LS-SVM) algorithm was used to identify each lane region of the mainline, and the initial traffic parameters and surrogate safety measures (SSMs) were extracted and aggregated. The negative samples that characterized normal traffic and the positive samples that characterized regional risk were identified. Mutual information (MI) was used to determine the information gain of various feature variables in the samples, and the key feature variables affecting the regional conditions were tested and screened by means of binary logit regression analysis. Upon screening the variables and corresponding labels, the construction and verification of a lane-level regional risk prediction model was completed using the catastrophe theory. The results showed that lane difference is an important parameter to reduce the uncertainty of regional risk, and its odds ratio (OR) was 16.30 at the 95% confidence level. The 10%-quantile modified time to collision (MTTC) inverse, the speed difference between lanes, and 10%-quantile headway (DHW) had an obvious influence on regional status. The model achieved an overall accuracy of 86.50%, predicting 84.78% of regional risks with a false positive rate of 13.37% and 86.63% of normal traffic with a false positive rate of 15.22%. The proposed model can provide a basis for formulating individualized active traffic control strategies for different lanes. Full article

The state-of-the-art provides data-driven and knowledge-driven diagnostic methods. Each category has its strengths and shortcomings. The knowledge-driven methods rely mainly on expert knowledge and resemble the diagnostic thinking of domain experts with a high capacity in the reasoning of uncertainties, diagnostics of different fault severities, and understandability. However, these methods involve higher and more time-consuming effort; they require a deep understanding of the causal relationships between faults and symptoms; and there is still a lack of automatic approaches to improving the efficiency. The data-driven methods rely on similarities and patterns, and they are very sensitive to changes of patterns and have more accuracy than the knowledge-driven methods, but they require massive data for training, cannot inform about the reason behind the result, and represent black boxes with low understandability. The research problem is thus the combination of knowledge-driven and data-driven diagnosis in DCV and heating systems, to benefit from both categories. The diagnostic method presented in this paper involves less effort for experts without requiring deep understanding of the causal relationships between faults and symptoms compared to existing knowledge-driven methods, while offering high understandability and high accuracy. The fault diagnosis uses a data-driven classifier in combination with knowledge-driven inference with both fuzzy logic and a Bayesian Belief Network (BBN). In offline mode, for each fault class, a Relation-Direction Probability (RDP) table is computed and stored in a fault library. In online mode, we determine the similarities between the actual RDP and the offline precomputed RDPs. The combination of BBN and fuzzy logic in our introduced method analyzes the dependencies of the signals using Mutual Information (MI) theory. The results show the performance of the combined classifier is comparable to the data-driven method while maintaining the strengths of the knowledge-driven methods. Full article

The relaxation properties of viscous liquids close to their glass transition (GT) have been widely characterised by the statistical tool of time correlation functions. However, the strong influence of ubiquitous non-linearities calls for new, alternative tools of analysis. In this respect, information theory-based observables and, more specifically, mutual information (MI) are gaining increasing interest. Here, we report on novel, deeper insight provided by MI-based analysis of molecular dynamics simulations of molecular and macromolecular glass-formers on two distinct aspects of transport and relaxation close to GT, namely dynamical heterogeneity (DH) and secondary Johari–Goldstein (JG) relaxation processes. In a model molecular liquid with significant DH, MI reveals two populations of particles organised in clusters having either filamentous or compact globular structures that exhibit different mobility and relaxation properties. In a model polymer melt, MI provides clearer evidence of JG secondary relaxation and sharper insight into its DH. It is found that both DH and MI between the orientation and the displacement of the bonds reach (local) maxima at the time scales of the primary and JG secondary relaxation. This suggests that, in (macro)molecular systems, the mechanistic explanation of both DH and relaxation must involve rotation/translation coupling. Full article

Despite the great significance of precisely forecasting the wind speed for development of the new and clean energy technology and stable grid operators, the stochasticity of wind speed makes the prediction a complex and challenging task. For improving the security and economic performance of power grids, accurate short-term wind power forecasting is crucial. In this paper, a deep learning model (Long Short-term Memory (LSTM)) has been proposed for wind speed prediction. Knowing that wind speed time series is nonlinear stochastic, the mutual information (MI) approach was used to find the best subset from the data by maximizing the joint MI between subset and target output. To enhance the accuracy and reduce input characteristics and data uncertainties, rough set and interval type-2 fuzzy set theory are combined in the proposed deep learning model. Wind speed data from an international airport station in the southern coast of Iran Bandar-Abbas City was used as the original input dataset for the optimized deep learning model. Based on the statistical results, the rough set LSTM (RST-LSTM) model showed better prediction accuracy than fuzzy and original LSTM, as well as traditional neural networks, with the lowest error for training and testing datasets in different time horizons. The suggested model can support the optimization of the control approach and the smooth procedure of power system. The results confirm the superior capabilities of deep learning techniques for wind speed forecasting, which could also inspire new applications in meteorology assessment. Full article

This study investigates the information–theoretic waveform design problem to improve radar performance in the presence of signal-dependent clutter environments. The goal was to study the waveform energy allocation strategies and provide guidance for radar waveform design through the trade-off relationship between the information theory criterion and the signal-to-interference-plus-noise ratio (SINR) criterion. To this end, a model of the constraint relationship among the mutual information (MI), the Kullback–Leibler divergence (KLD), and the SINR is established in the frequency domain. The effects of the SINR value range on maximizing the MI and KLD under the energy constraint are derived. Under the constraints of energy and the SINR, the optimal radar waveform method based on maximizing the MI is proposed for radar estimation, with another method based on maximizing the KLD proposed for radar detection. The maximum MI value range is bounded by SINR and the maximum KLD value range is between 0 and the Jenson–Shannon divergence (J-divergence) value. Simulation results show that under the SINR constraint, the MI-based optimal signal waveform can make full use of the transmitted energy to target information extraction and put the signal energy in the frequency bin where the target spectrum is larger than the clutter spectrum. The KLD-based optimal signal waveform can therefore make full use of the transmitted energy to detect the target and put the signal energy in the frequency bin with the maximum target spectrum. Full article

Characterizing and modeling processes at the sun and space plasma in our solar system are difficult because the underlying physics is often complex, nonlinear, and not well understood. The drivers of a system are often nonlinearly correlated with one another, which makes it a challenge to understand the relative effects caused by each driver. However, entropy-based information theory can be a valuable tool that can be used to determine the information flow among various parameters, causalities, untangle the drivers, and provide observational constraints that can help guide the development of the theories and physics-based models. We review two examples of the applications of the information theoretic tools at the Sun and near-Earth space environment. In the first example, the solar wind drivers of radiation belt electrons are investigated using mutual information (MI), conditional mutual information (CMI), and transfer entropy (TE). As previously reported, radiation belt electron flux (J_e) is anticorrelated with solar wind density (n_sw) with a lag of 1 day. However, this lag time and anticorrelation can be attributed mainly to the J_e(t + 2 days) correlation with solar wind velocity (V_sw)(t) and n_sw(t + 1 day) anticorrelation with V_sw(t). Analyses of solar wind driving of the magnetosphere need to consider the large lag times, up to 3 days, in the (V_sw, n_sw) anticorrelation. Using CMI to remove the effects of V_sw, the response of J_e to n_sw is 30% smaller and has a lag time <24 h, suggesting that the loss mechanism due to n_sw or solar wind dynamic pressure has to start operating in <24 h. Nonstationarity in the system dynamics is investigated using windowed TE. The triangle distribution in J_e(t + 2 days) vs. V_sw(t) can be better understood with TE. In the second example, the previously identified causal parameters of the solar cycle in the Babcock–Leighton type model such as the solar polar field, meridional flow, polar faculae (proxy for polar field), and flux emergence are investigated using TE. The transfer of information from the polar field to the sunspot number (SSN) peaks at lag times of 3–4 years. Both the flux emergence and the meridional flow contribute to the polar field, but at different time scales. The polar fields from at least the last 3 cycles contain information about SSN. Full article

It is well known that the statistical mechanical theory of liquids has been lagging far behind the theory of either gases or solids, See for examples: Ben-Naim (2006), Fisher (1964), Guggenheim (1952) Hansen and McDonald (1976), Hill (1956), Temperley, Rowlinson and Rushbrooke (1968), O’Connell (1971). Information theory was recently used to derive and interpret the entropy of an ideal gas of simple particles (i.e., non-interacting and structure-less particles). Starting with Shannon’s measure of information (SMI), one can derive the entropy function of an ideal gas, the same function as derived by Sackur (1911) and Tetrode (1912). The new deviation of the same entropy function, based on SMI, has several advantages, as listed in Ben-Naim (2008, 2017). Here we mention two: First, it provides a simple interpretation of the various terms in this entropy function. Second, and more important for our purpose, this derivation may be extended to any system of interacting particles including liquids and solutions. The main idea is that once one adds intermolecular interactions between the particles, one also adds correlations between the particles. These correlations may be cast in terms of mutual information (MI). Hence, we can start with the informational theoretical interpretation of the entropy of an ideal gas. Then, we add correction due to correlations in the form of MI between the locations of the particles. This process preserves the interpretation of the entropy of liquids and solutions in terms of a measure of information (or as an average uncertainty about the locations of the particles). It is well known that the entropy of liquids, any liquids for that matter, is lower than the entropy of a gas. Traditionally, this fact is interpreted in terms of order-disorder. The lower entropy of the liquid is interpreted in terms of higher degree of order compared with that of the gas. However, unlike the transition from a solid to either a liquid, or to a gaseous phase where the order-disorder interpretation works well, the same interpretation would not work for the liquid-gas transition. It is hard, if not impossible, to argue that the liquid phase is more “ordered” than the gaseous phase. In this article, we interpret the lower entropy of liquids in terms of SMI. One outstanding liquid known to be a structured liquid, is water, according to Ben-Naim (2009, 2011). In addition, heavy water, as well as aqueous solutions of simple solutes such as argon or methane, will be discussed in this article. Full article

Equivalent salt deposit density (ESDD) and non-soluble deposit density (NSDD) measurements are a basic requirement of power systems. In order to predict the site pollution severity (SPS) of insulators, a new method based on random forests (RFs) is proposed. Using mutual information (MI) theory and RFs, the weights of factors related to the SPS of insulators are analyzed. The samples of contaminated insulators are extracted from the transmission lines of high voltage alternating current (HVAC) and high voltage direct current transmission (HVDC). The regression models of RFs and support vector machines (SVM) are constructed and compared, which helps to support the lack of information in predicting NSDD in previous works. The results are as follows: according to the mean decrease accuracy (MDA), mean decrease Gini, (MDG), and MI, the types of the insulators (including surface area, surface orientation, and total length) as well as the hydrophobicity are the main factors affecting both ESDD and NSDD. Compared with NSDD, the electrical parameters have a significant effect on ESDD. For the influence factors of ESDD, the weights of the insulator type, hydrophobicity, and meteorological factors are 52.94%, 6.35%, and 21.88%, respectively. For the influence factors of NSDD, the weights of the insulator type, hydrophobicity, and meteorological factors are 55.37%, 11.04%, and 14.26%, respectively. The influence voltage level (vl), voltage type (vt), polarity/phases (pp) exerted on ESDD are 1.5 times, 3 times, and 4.5 times of NSDD, respectively. The influence that distance from the coastline (d), wind velocity (wv), and rainfall (rf) exert on NSDD are 1.5 times, 2 times, and 2.5 times that of ESDD, respectively. Compared with the natural contamination test and the SVM regression model, the RFs regression model can effectively predict the contamination degree of insulators, and the relative error of the predicted ESDD and NSDD is 8.31% and 9.62%, respectively. Full article

This paper presents a low probability of interception (LPI) radar waveform design method with a fixed average power constraint based on information theory. The Kullback–Leibler divergence (KLD) between the intercept signal and background noise is presented as a practical metric to evaluate the performance of the adversary intercept receiver in this paper. Through combining it with the radar performance metric, that is, the mutual information (MI), a multi-objective optimization model of LPI waveform design is developed. It is a trade-off between the performance of radar and enemy intercept receiver. After being transformed into a single-objective optimization problem, it can be solved by using an interior point method and a sequential quadratic programming (SQP) method. Simulation results verify the correctness and effectiveness of the proposed LPI radar waveform design method. Full article

The module temperature is the most important parameter influencing the output power of solar photovoltaic (PV) systems, aside from solar irradiance. In this paper, we focus on the interdisciplinary research that combines the correlation analysis, mutual information (MI) and heat transfer theory, which aims to figure out the correlative relations between different meteorological impact factors (MIFs) and PV module temperature from both quality and quantitative aspects. The identification and confirmation of primary MIFs of PV module temperature are investigated as the first step of this research from the perspective of physical meaning and mathematical analysis about electrical performance and thermal characteristic of PV modules based on PV effect and heat transfer theory. Furthermore, the quantitative description of the MIFs influence on PV module temperature is mathematically formulated as several indexes using correlation-based feature selection (CFS) and MI theory to explore the specific impact degrees under four different typical weather statuses named general weather classes (GWCs). Case studies for the proposed methods were conducted using actual measurement data of a 500 kW grid-connected solar PV plant in China. The results not only verified the knowledge about the main MIFs of PV module temperatures, more importantly, but also provide the specific ratio of quantitative impact degrees of these three MIFs respectively through CFS and MI based measures under four different GWCs. Full article