Entropy

We consider a retrospective change-point detection problem for multidimensional time series of arbitrary nature (in particular, panel data). Change-points are the moments at which the changes in generating mechanism occur. Our method is based on the new theory of $ϵ$-complexity of individual continuous vector functions and is model-free. We present simulation results confirming the effectiveness of the method. Full article

U-model, which is a control-oriented model set with the property of generally facilitate nonlinearity dynamic inversion/cancellation, has been introduced to the Disturbance Observer-Based control (DOBC) methods to improve the performance of the nonlinear systems in this paper. A general DOB based U-Control (DOBUC) framework is proposed to improve the disturbance attenuation capability of U-controller for both linear and nonlinear systems combined with (based on) the U-model-based dynamic inversion which expands the classical linear disturbance observer control to general nonlinear systems. The proposed two-step DOBUC design procedures in which the design of DOB and U-controller are totally independent and separated, enables the establishment of global exponential stability without being subject to disturbances and uncertainties. Comparative simulation experiments with Nonlinear DOBC in controlling Wind Energy Conversion Systems (WECS) and Permanent Magnet Synchronous Motors (PMSM) demonstrated the proposed method. Full article

We discuss the application of the Simple Equations Method (SEsM) for obtaining exact solutions of non-linear differential equations to several cases of equations containing non-polynomial non-linearity. The main idea of the study is to use an appropriate transformation at Step (1.) of SEsM. This transformation has to convert the non-polynomial non- linearity to polynomial non-linearity. Then, an appropriate solution is constructed. This solution is a composite function of solutions of more simple equations. The application of the solution reduces the differential equation to a system of non-linear algebraic equations. We list 10 possible appropriate transformations. Two examples for the application of the methodology are presented. In the first example, we obtain kink and anti- kink solutions of the solved equation. The second example illustrates another point of the study. The point is as follows. In some cases, the simple equations used in SEsM do not have solutions expressed by elementary functions or by the frequently used special functions. In such cases, we can use a special function, which is the solution of an appropriate ordinary differential equation, containing polynomial non-linearity. Specific cases of the use of this function are presented in the second example. Full article

We consider learning as an undirected graphical model from sparse data. While several efficient algorithms have been proposed for graphical lasso (GL), the alternating direction method of multipliers (ADMM) is the main approach taken concerning joint graphical lasso (JGL). We propose proximal gradient procedures with and without a backtracking option for the JGL. These procedures are first-order methods and relatively simple, and the subproblems are solved efficiently in closed form. We further show the boundedness for the solution of the JGL problem and the iterates in the algorithms. The numerical results indicate that the proposed algorithms can achieve high accuracy and precision, and their efficiency is competitive with state-of-the-art algorithms. Full article

Thanks to technological advances leading to near-continuous time observations, emerging multivariate point process data offer new opportunities for causal discovery. However, a key obstacle in achieving this goal is that many relevant processes may not be observed in practice. Naïve estimation approaches that ignore these hidden variables can generate misleading results because of the unadjusted confounding. To plug this gap, we propose a deconfounding procedure to estimate high-dimensional point process networks with only a subset of the nodes being observed. Our method allows flexible connections between the observed and unobserved processes. It also allows the number of unobserved processes to be unknown and potentially larger than the number of observed nodes. Theoretical analyses and numerical studies highlight the advantages of the proposed method in identifying causal interactions among the observed processes. Full article

In the era of the Internet of Things and big data, we are faced with the management of a flood of information. The complexity and amount of data presented to the decision-maker are enormous, and existing methods often fail to derive nonredundant information quickly. Thus, the selection of the most satisfactory set of solutions is often a struggle. This article investigates the possibilities of using the entropy measure as an indicator of data difficulty. To do so, we focus on real-world data covering various fields related to markets (the real estate market and financial markets), sports data, fake news data, and more. The problem is twofold: First, since we deal with unprocessed, inconsistent data, it is necessary to perform additional preprocessing. Therefore, the second step of our research is using the entropy-based measure to capture the nonredundant, noncorrelated core information from the data. Research is conducted using well-known algorithms from the classification domain to investigate the quality of solutions derived based on initial preprocessing and the information indicated by the entropy measure. Eventually, the best 25% (in the sense of entropy measure) attributes are selected to perform the whole classification procedure once again, and the results are compared. Full article

Multiscale entropy (MSE) analysis is a fundamental approach to access the complexity of a time series by estimating its information creation over a range of temporal scales. However, MSE may not be accurate or valid for short time series. This is why previous studies applied different kinds of algorithm derivations to short-term time series. However, no study has systematically analyzed and compared their reliabilities. This study compares the MSE algorithm variations adapted to short time series on both human and rat heart rate variability (HRV) time series using long-term MSE as reference. The most used variations of MSE are studied: composite MSE (CMSE), refined composite MSE (RCMSE), modified MSE (MMSE), and their fuzzy versions. We also analyze the errors in MSE estimations for a range of incorporated fuzzy exponents. The results show that fuzzy MSE versions—as a function of time series length—present minimal errors compared to the non-fuzzy algorithms. The traditional multiscale entropy algorithm with fuzzy counting (MFE) has similar accuracy to alternative algorithms with better computing performance. For the best accuracy, the findings suggest different fuzzy exponents according to the time series length. Full article

This theoretical paper explores the affect-logic approach to schizophrenia in light of the general complexity theories of cognition: embodied cognition, Haken’s synergetics, and Friston’s free energy principle. According to affect-logic, the mental apparatus is an embodied system open to its environment, driven by bioenergetic inputs of emotions. Emotions are rooted in goal-directed embodied states selected by evolutionary pressure for coping with specific situations such as fight, flight, attachment, and others. According to synergetics, nonlinear bifurcations and the emergence of new global patterns occur in open systems when control parameters reach a critical level. Applied to the emergence of psychotic states, synergetics and the proposed energetic understanding of emotions lead to the hypothesis that critical levels of emotional tension may be responsible for the transition from normal to psychotic modes of functioning in vulnerable individuals. In addition, the free energy principle through learning suggests that psychotic symptoms correspond to alternative modes of minimizing free energy, which then entails distorted perceptions of the body, self, and reality. This synthetic formulation has implications for novel therapeutic and preventive strategies in the treatment of psychoses, among these are milieu-therapeutic approaches of the Soteria type that focus on a sustained reduction of emotional tension and phenomenologically oriented methods for improving the perception of body, self, and reality. Full article

The Khinchin–Shannon generalized inequalities for entropy measures in Information Theory, are a paradigm which can be used to test the Synergy of the distributions of probabilities of occurrence in physical systems. The rich algebraic structure associated with the introduction of escort probabilities seems to be essential for deriving these inequalities for the two-parameter Sharma–Mittal set of entropy measures. We also emphasize the derivation of these inequalities for the special cases of one-parameter Havrda–Charvat’s, Rényi’s and Landsberg–Vedral’s entropy measures. Full article

With the rapid growth of the Internet, the curse of dimensionality caused by massive multi-label data has attracted extensive attention. Feature selection plays an indispensable role in dimensionality reduction processing. Many researchers have focused on this subject based on information theory. Here, to evaluate feature relevance, a novel feature relevance term (FR) that employs three incremental information terms to comprehensively consider three key aspects (candidate features, selected features, and label correlations) is designed. A thorough examination of the three key aspects of FR outlined above is more favorable to capturing the optimal features. Moreover, we employ label-related feature redundancy as the label-related feature redundancy term (LR) to reduce unnecessary redundancy. Therefore, a designed multi-label feature selection method that integrates FR with LR is proposed, namely, Feature Selection combining three types of Conditional Relevance (TCRFS). Numerous experiments indicate that TCRFS outperforms the other 6 state-of-the-art multi-label approaches on 13 multi-label benchmark data sets from 4 domains. Full article

Several methods have been recently proposed to calculate configurational entropy, based on Boltzmann entropy. Some of these methods appear to be fully thermodynamically consistent in their application to landscape patch mosaics, but none have been shown to be fully generalizable to all kinds of landscape patterns, such as point patterns, surfaces, and patch mosaics. The goal of this paper is to evaluate if the direct application of the Boltzmann relation is fully generalizable to surfaces, point patterns, and landscape mosaics. I simulated surfaces and point patterns with a fractal neutral model to control their degree of aggregation. I used spatial permutation analysis to produce distributions of microstates and fit functions to predict the distributions of microstates and the shape of the entropy function. The results confirmed that the direct application of the Boltzmann relation is generalizable across surfaces, point patterns, and landscape mosaics, providing a useful general approach to calculating landscape entropy. Full article

Many regions around the globe are subjected to precipitation-data scarcity that often hinders the capacity of hydrological modeling. The entropy theory and the principle of maximum entropy can help hydrologists to extract useful information from the scarce data available. In this work, we propose a new method to assess sub-daily precipitation features such as duration and intensity based on daily precipitation using the principle of maximum entropy. Particularly in arid and semiarid regions, such sub-daily features are of central importance for modeling sediment transport and deposition. The obtained features were used as input to the SYPoME model (sediment yield using the principle of maximum entropy). The combined method was implemented in seven catchments in Northeast Brazil with drainage areas ranging from 10⁻³ to 10⁺² km² in assessing sediment yield and delivery ratio. The results show significant improvement when compared with conventional deterministic modeling, with Nash–Sutcliffe efficiency (NSE) of 0.96 and absolute error of 21% for our method against NSE of −4.49 and absolute error of 105% for the deterministic approach. Full article

We propose an algorithm to generate graphical summarising of longer text passages using a set of illustrative pictures (TIPS). TIPS is an algorithm using a voting process that uses results of individual “weak” algorithms. The proposed method includes a summarising algorithm that generates a digest of the input document. Each sentence of the text summary is used as the input for further processing by the sentence transformer separately. A sentence transformer performs text embedding and a group of CLIP similarity-based algorithms trained on different image embedding finds semantic distances between images in the illustration image database and the input text. A voting process extracts the most matching images to the text. The TIPS algorithm allows the integration of the best (highest scored) results of the different recommendation algorithms by diminishing the influence of images that are a disjointed part of the recommendations of the component algorithms. TIPS returns a set of illustrative images that describe each sentence of the text summary. Three human judges found that the use of TIPS resulted in an increase in matching highly relevant images to text, ranging from 5% to 8% and images relevant to text ranging from 3% to 7% compared to the approach based on single-embedding schema. Full article

The accurate description of a complex process should take into account not only the interacting elements involved but also the scale of the description. Therefore, there can not be a single measure for describing the associated complexity of a process nor a single metric applicable in all scenarios. This article introduces a framework based on multiscale entropy to characterize the complexity associated with the most identifiable characteristic of songs: the melody. We are particularly interested in measuring the complexity of popular songs and identifying levels of complexity that statistically explain the listeners’ preferences. We analyze the relationship between complexity and popularity using a database of popular songs and their relative position in a preferences ranking. There is a tendency toward a positive association between complexity and acceptance (success) of a song that is, however, not significant after adjusting for multiple testing. Full article

This study proposes a framework to diagnose stock market crashes and predict the subsequent price rebounds. Based on the observation of anomalous changes in stock correlation networks during market crashes, we extend the log-periodic power-law model with a metric that is proposed to measure network anomalies. To calculate this metric, we design a prediction-guided anomaly detection algorithm based on the extreme value theory. Finally, we proposed a hybrid indicator to predict price rebounds of the stock index by combining the network anomaly metric and the visibility graph-based log-periodic power-law model. Experiments are conducted based on the New York Stock Exchange Composite Index from 4 January 1991 to 7 May 2021. It is shown that our proposed method outperforms the benchmark log-periodic power-law model on detecting the 12 major crashes and predicting the subsequent price rebounds by reducing the false alarm rate. This study sheds light on combining stock network analysis and financial time series modeling and highlights that anomalous changes of a stock network can be important criteria for detecting crashes and predicting recoveries of the stock market. Full article

Spectrum sensing is an important function in radio frequency spectrum management and cognitive radio networks. Spectrum sensing is used by one wireless system (e.g., a secondary user) to detect the presence of a wireless service with higher priority (e.g., a primary user) with which it has to coexist in the radio frequency spectrum. If the wireless signal is detected, the second user system releases the given frequency to maintain the principle of not interfering. This paper proposes a machine learning implementation of spectrum sensing using the entropy measure as a feature vector. In the training phase, the information about the activity of the wireless service with higher priority is gathered, and the model is formed. In the classification phase, the wireless system compares the current sensing report to the created model to calculate the posterior probability and classify the sensing report into either the presence or absence of wireless service with higher priority. This paper proposes the novel application of the Fluctuation Dispersion Entropy (FDE) measure recently introduced in the research community as a feature vector to build the model and implement the classification. An improved implementation of the FDE (IFDE) is used to enhance the robustness to noise. IFDE is further enhanced with an adaptive method (AIFDE) to automatically select the hyper-parameter introduced in IFDE. Then, this paper combines the machine learning approach with the entropy measure approach, which are both recent developments in spectrum sensing research. The approach is compared to similar approaches in literature and the classical energy detection method using a generated radar signal data set with different conditions of SNR(dB) and fading conditions. The results show that the proposed approach is able to outperform the approaches from literature based on other entropy measures or the Energy Detector (ED) in a consistent way across different levels of SNR and fading conditions. Full article

Fixed-time synchronization problem for delayed dynamical complex networks is explored in this paper. Compared with some correspondingly existed results, a few new results are obtained to guarantee fixed-time synchronization of delayed dynamical networks model. Moreover, by designing adaptive controller and discontinuous feedback controller, fixed-time synchronization can be realized through regulating the main control parameter. Additionally, a new theorem for fixed-time synchronization is used to reduce the conservatism of the existing work in terms of conditions and the estimate of synchronization time. In particular, we obtain some fixed-time synchronization criteria for a type of coupled delayed neural networks. Finally, the analysis and comparison of the proposed controllers are given to demonstrate the validness of the derived results from one numerical example. Full article

We introduce an index based on information theory to quantify the stationarity of a stochastic process. The index compares on the one hand the information contained in the increment at the time scale $\tau$ of the process at time t with, on the other hand, the extra information in the variable at time t that is not present at time $t-\tau$. By varying the scale $\tau$, the index can explore a full range of scales. We thus obtain a multi-scale quantity that is not restricted to the first two moments of the density distribution, nor to the covariance, but that probes the complete dependences in the process. This index indeed provides a measure of the regularity of the process at a given scale. Not only is this index able to indicate whether a realization of the process is stationary, but its evolution across scales also indicates how rough and non-stationary it is. We show how the index behaves for various synthetic processes proposed to model fluid turbulence, as well as on experimental fluid turbulence measurements. Full article

Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model’s uncertainty is evaluated using point-prediction metrics, such as the negative log-likelihood (NLL), expected calibration error (ECE) or the Brier score on held-out data. Marginal coverage of prediction intervals or sets, a well-known concept in the statistical literature, is an intuitive alternative to these metrics but has yet to be systematically studied for many popular uncertainty quantification techniques for deep learning models. With marginal coverage and the complementary notion of the width of a prediction interval, downstream users of deployed machine learning models can better understand uncertainty quantification both on a global dataset level and on a per-sample basis. In this study, we provide the first large-scale evaluation of the empirical frequentist coverage properties of well-known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that coverage is not maintained on out-of-distribution data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and reinforce coverage as an important metric in developing models for real-world applications. Full article