Information Dynamics in Complex Systems Negates a Dichotomy between Chance and Necessity

Abstract

Entropy increases in the execution of linear physical processes. At equilibrium, all uncertainty about the future is removed and information about the past is lost. Complex systems, on the other hand, can lead to the emergence of order, sustain uncertainty about the future, and generate new information to replace all old information about the system in finite time. The Kolmogorov–Sinai entropy for events and the Kolmogorov–Chaitin complexity for strings of numbers both approximate Shannon’s entropy (an indicator for the removal of uncertainty), indicating that information production is equivalent to the degree of complexity of an event. Thus, in the execution of non-linear processes, information entropy is inseparably tied to thermodynamic entropy. Therein, the critical decision points (bifurcations), which can exert lasting impact on the evolution of the future (the “butterfly effect”), defy the definition of being either born from randomness or from determination. Nevertheless, their information evolution and degree of complexity are amenable to measurement and can meaningfully replace the dichotomy of chance versus necessity. Common anthropomorphic perceptions do not accurately account for the transient durability of information, the potential for major consequences by small actions, or the absence of a discernible opposition between coincidence and inevitability.

The terms “chance” and “necessity” are part of the daily vocabulary. However, they have been difficult to define rigorously [1]. Simply the delineation of what is lawful and what is random has been subject to debate. Karl R. Popper wrote about the differentiation of planetary movements as lawful and of throws of dice as chance events that, thus far, prognoses of planetary orbits have been successful while prognoses of individual throws of dice have not. Extending a rationale developed earlier by Kurt Gödel, Popper demonstrated that self-prediction from within a system is impossible, and he was consecutively led to become an indeterminist [2].

The concept of randomness may be characterized in various ways as an event without a perceivable design (Appendix A(1)). The lack thereof can be reflected in the “random” mechanism of generation or in the “random” pattern of output.

-

Uncertainty in the mechanism that generates specific outcomes may produce randomness. The throwing of dice is such a mechanism of uncertainty. Likewise, a string of numbers is random if it is generated by an unpredictable mechanism [3]. The randomness resides in the disorder of the generating process.

-

The arrangement of a pattern or a string of numbers may reflect randomness. A string is random because there is no inherent design that allows the prediction of consecutive digits or because there are no simple rules that enable the description of the string in compressed form [4]. The randomness lies in the arrangement itself.

Information (the removal of uncertainty, Appendix A(2)) and complexity (non-linearity, not amenable to closed-form solutions) are intricately linked to our everyday understanding of chance, but unlike “chance” or “randomness” these physical entities can be subjected to rigorous definition and analysis. Whereas no information is yielded by equilibrium (stasis), information flow (more accurately information evolution [5]) is tied to an occurrence (Section 1). The complexity of that occurrence underlies the amount of new information generated (Section 2 and Section 3), and it produces emergent phenomena (Section 4). At the bifurcations that are inherent in non-linear systems, infinitesimal influences can yield dramatically divergent results (Section 4), but any analysis of deterministic versus indeterministic foundations for complex processes is bound to be unsuccessful (Section 5 and Section 6). Substantial implications derive from the transience of information, the potential for major consequences by small actions at decision points (bifurcations), and the breakdown of the dichotomy between chance and necessity.

1. Information is the Removal of Uncertainty

The information content of an event is principally quantifiable in the algorithms developed by Claude Shannon [6]. An event with a certain outcome (a probability of 1) yields no new information. The information content of a process is interpretable as the removal of the extent of uncertainty that exists before its execution. This uncertainty constitutes the random or chance component inherent in this process. With its execution, the random element is removed, and new information is communicated.

In Shannon’s model, communication takes place between a sender and a receiver via a channel, the capacity of which is a critical determinant that is calculated from its noise characteristics. For all communication rates below channel capacity, the probability of error can in principle become arbitrarily small. However, theoretically optimized communication schemes may be computationally impractical. Random processes have an irreducible complexity, below which the signal cannot be compressed. The ultimate data compression is dubbed the entropy (Appendix A(3) and Appendix A(4)). Entropy and mutual information are functions of the probability distributions that underlie the process of communication.

2. Information Has a Finite Lifespan

A swinging pendulum must eventually come to a halt. Once at rest, the history of its swing cannot be reconstructed by any means. The second law of thermodynamics states that a closed system progresses toward a state of maximal entropy. Once this state is reached, all information on the path that led to it is lost, as is any uncertainty about its future. Thermodynamics (Appendix A(5)) has introduced the asymmetry of time irreversibility into physical law, and it has established the inescapability of losing information about the past (a form of “collective forgetting”).

The energy of physical systems is describable on the macroscale, which in classical mechanics is completely intelligible and can be captured with deterministic algorithms, versus the micro-scale of thermal motion, which is randomly distributed and can be expressed only as averages over large numbers. In the epochs preceding non-linear systems research, the states of the microscales were assumed to be uniform or stochastic and to constitute a lower limit of feasible explanation [7]. An adjustment of this concept is required in non-linear events that arise far from equilibrium, where energy is believed to emerge from the microscales and affect the macroscopic outcome [8].

In dynamics, motion can be depicted as a flow along trajectories in a phase space that often maps momentum versus position variables (sometimes velocity versus position) [9,10]. This phase space has as many dimensions as the system has degrees of freedom. In laminar flow, motion is governed by boundary and initial conditions, and no new information is generated, as outlined above. In turbulent flow, information is continuously generated by the flow itself through an exchange between the micro- and macro-scales. The average rate of information production (bits of information per unit time, denoted as $\overline{\lambda }$; the term is identical to the Lyapunov characteristic exponent [11]) is a measure for the rate of divergence of nearby trajectories (Appendix A(6)). The transition of a system from laminar to turbulent behavior is reflected in a change of the Lyapunov characteristic exponent from negative to positive, corresponding to a change of the system from an information sink to an information source, even if this turbulent behavior is governed by simple system equations. The Lyapunov characteristic exponent is a system state function that remains invariant under coordinate transformations. The information generated anew by turbulent systems precludes prediction past a certain time, when new information has accumulated to entirely displace the initial data. “New information is continuously being injected into the macroscopic degrees of freedom of the world by every puff of wind and every swirl of water” [8]. The loss of information about the initial state of a system is a physical reality, whether the occurring process strives toward equilibrium or toward emergence.

3. Information and Complexity are Equivalent

An occurrence in phase space, state space, or event space is characterized by its occupancy of the attributed space. In essence, such an analysis of occurrences in abstract space removes the definition of dimensions from to the axes of concrete space and relieves its restriction to whole numbers. The application of the Lyapunov characteristic exponent $\overline{\lambda }$ to such a conceptual space yields a measure for the information flow (or the information evolution) associated with each step in the execution of the event (Appendix A(7)). It lays the foundation for the calculation of a fractal dimensionality, the Lyapunov dimension, that characterizes the information evolution of the event in its entirety. This dimensionality gives an estimate of the Kolmogorov–Sinai entropy [12], which defines the information change inherent in the execution of the process. The Lyapunov dimension D_KY represents an upper bound for the information dimension of the system (Appendix A(8)).

$$D_{KY}=k+{\displaystyle \sum }_{i=1}^k\frac{{\overline{\mathsf{\lambda }}}_i}{\left|{\overline{\mathsf{\lambda }}}_{k+1}\right|}$$

k = maximum integer such that the sum of the k largest exponents is non-negative; $\overline{\lambda }$ = Lyapunov characteristic exponent. The insertion of the Lyapunov characteristic exponent into this formula achieves the connection between the complexity of a process, the execution of which impacts thermodynamic entropy development, to its information entropy.

As non-linear systems dynamics studies algorithms that do not have closed-form solutions (Appendix A(9)), its analyses draw heavily on information theory [13]. Complexity has been measured in simple strings of 0s and 1s. A string is random if its shortest description is obtained by writing it out in its entirety—the shortest description of the string is the string itself. The more compressible the string, the less random it is. Connecting complexity and information content, the Kolmogorov–Chaitin complexity (K) of a string is approximately equal to the Shannon entropy (H), if the sequence of the string under study is drawn at random from a distribution that has the entropy H [14]. Specifically, for almost all infinite sequences produced by a stationary process, the growth rate of the Kolmogorov–Chaitin complexity is equal to the Shannon entropy rate. More broadly, diverse measures for information change, entropy and dimensionality in conceptual space are interdependent (

Table 1. Entropy, information and dimension. Characteristics describing the measures of complexity. The far-right column describes connections among the diverse measures.

Measure	Formalism	Area	Description	Explanation	Interpretation	Commonalities
Boltzmann entropy	$S=-k_B{\displaystyle \sum }{p}_i\ln p_i$	thermodynamics	entropy = degree of disorder	dissipation of energy, arrow of time	thermodynamic entropy equivalent to thermodynamic information
Shannon entropy	$S=-{\displaystyle \sum }_i{P}_i\log P_i=-E_P\left[\log P\right]$	communication	entropy = ultimate data compression	removal of the uncertainty that exists before communication	uncertainty constitutes the random or chance component inherent in this process	with Boltzmann: common origin in probability theory, common requirements
Rényi entropy	$D_{\alpha }=\underset{\epsilon \to 0}{\lim }\frac{\frac{1}{\alpha -1}\log \left({{\displaystyle \sum }}_i{p}_i^{\alpha }\right)}{\log \epsilon }$	abstract space	uncertainty = dimension	connects uncertainty to dimensionality of the space, in which it is measured	quantifies diversity, uncertainty or randomness of a system	generalization of the Shannon entropy
Kolmogorov-Sinai entropy	$H_{KS}=\underset{\epsilon \to 0}{\lim }\underset{t\to 0}{\lim }\left(H_{∆t}/t\right))$	dynamical systems	information change inherent in the execution of a process	metric invariant of a dynamical system	maximum capacity of information that can be generated by a dynamical system	essentially Shannon entropy per unit time
Kolmogorov-Chaitin complexity	$K_T\left(s\right)=min\left\{\left|p\right|,T\left(p\right)=s\right\}$	number strings	complexity = non-compressibility	connects complexity and information content	reflective of the content of information	growth rate often equal to Shannon entropy rate
Lyapunov dimension	$D_{KY}=\mathrm{k}+{\displaystyle {\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{k}}}\frac{{\overline{\lambda }}_{\mathrm{i}}}{\left|{\overline{\lambda }}_{\mathrm{k}+1}\right|}$	abstract space	upper bound for the information dimension of a system	estimates fractal dimension of attractors	function of the Lyapunov characteristic exponents	estimate of the Kolmogorov-Sinai entropy
information dimension	$D_1=\underset{\epsilon \to 0}{\lim }\frac{-\log p_{\epsilon }}{\log \frac{1}{\epsilon }}$	probability	fractal dimension of a probability distribution	information measure for random vectors in Euclidean space	measure for the fractal dimension of a probability distribution	characterizes growth rate of Shannon entropy with fine-graining of space
Mandelbrot dimension	${\log }_{\epsilon }N=-D=\frac{\log N}{\log \epsilon }$	geometry	statistical index of complexity	ratio of the change in detail to the change in scale	measure for the space-filling capacity of a pattern	similar to box-counting dimension

), suggesting that these modalities of assessment largely equate to each other.

4. Complexity Drives Emergence via Critical Decision Points

Complex systems are characterized by non-periodic flow with sensitive dependence on the starting conditions, emergence of properties from non-equilibrium states, and generation of new information to fully replace old information. Hence, there are limits to how precisely we can describe the evolution of non-linear processes. Uncertainty about the future grows rapidly. Further, in the progression of complex systems there exist decision points, at which infinitesimal influences can propel events onto one of several very different future paths. Proceeding in this manner, the emergence of ordered structures from non-equilibrium conditions has been described in physics [15], chemistry [16,17] and evolution [18].

States in thermodynamic equilibrium (or states that equate to a minimal entropy production in the linear thermodynamics of non-equilibrium) are stable states that are principally reversible. Yet, irreversible processes play a fundamental and constructive role in the physical world. The laws of irreversible processes [10] have embedded dynamics in a more comprehensive formalism that includes unstable states. Non-equilibrium can lead to dissipative structures, wherein fluctuations introduce a stochastic description into the macroscopic level. Instabilities far from equilibrium are essential elements for emerging systems [19].

Evolution has been investigated as occurring in a conceptual state space, the shape of which is defined by the distribution of properties across an ensemble (a fitness landscape; Appendix A(10)). The initiating event for every step in evolution is a change (a mutation). Once a mutation has taken place, its penetration of the population is subjected to the rule of selection. However, simple and complex systems can exhibit powerful self-organization, and the effects of mutation and selection are diminished when operating on systems that have their own robust self-ordered properties. Spontaneous order is maintained despite selection, not because of it. Selection may only be able to mitigate the tendency for adaptive processes to become trapped on continuously lower local optima of fitness as complexity increases. While the selective force is stronger than the mutational force below a critical complexity of an organism, above this critical complexity, the dispersing mutational pressure increases, and the population falls from the global optimum to a suboptimal stationary steady state. Hence, the outcome of selection is context dependent. It drives or maintains complex systems on the boundary between order and chaos (sometimes dubbed “the edge of chaos” or “the onset of chaos”). At that boundary (“the poised state”), systems are best able to coordinate complex tasks and evolve in a complex environment [18].

A lack of periodicity is very common in nature. It is one of the distinguishing features of complex systems, for which progression and advanced states are unpredictable [20]. Events that are governed by non-linear differential equations have traditionally been considered deterministic on the basis of their driving algorithms, even though the slightest immeasurable deviations in their initial states can lead to dramatically different outcomes. For a finite system of ordinary differential equations representing forced dissipative flow, frequently, all of its solutions are ultimately confined within the same bounds. For these equations, non-periodic solutions cannot readily be determined, except by numerical procedures. The evolution of dissipative systems under these equations is commonly modeled with trajectories in phase space. Prediction of the sufficiently distant future is not feasible by any method, unless the initial conditions are known exactly (an impossible feat, as detailed in Section 5 below). There is an eventual necessity for any bounded system of finite dimensionality to come arbitrarily close to acquiring a state it has previously assumed. Only if the system is stable will its future development then remain arbitrarily close to its past history, and it will be quasi-periodic. Unstable systems display the now-famous “butterfly effect”: one flap of a butterfly’s wings may change the future course of the weather in a place far away. “The result […] implies that two states differing by imperceptible amounts may eventually evolve into two considerably different states. If, then, there is any error whatever in observing the present state–and in any real system such errors seem inevitable—an acceptable prediction of an instantaneous state in the distant future may well be impossible.” [21].

5. At the Critical Decision Points, Randomness Versus Determination Is Undecidable

It is principally impossible to assess whether decision points, such as bifurcations, are governed by deterministic or indeterministic mechanisms. Phase space models have minimally discernible resolution (Appendix A(11)), description beyond which is infeasible [8]. Thermodynamic considerations striving to incorporate irreversibility into physical law have developed event space models, wherein internal time and entropy become operators. The resolution of the description is restrained by the eigenvectors and eigenvalues of the operators [10].

Non-linear systems research often describes events as trajectories in phase space. Such phase space is divisible into blocks of minimum size that represent states. Their expansions constitute the limits to the precision of obtainable knowledge. Progressive attempts to more precisely measure the system eventually lead to the microscopic scale, wherein the uncertainty principle assures that the more accurately the position in phase space is determined, the less accurately the velocity can be assessed [22]. The analogous relationship exists between energy and time. Two trajectories become indistinguishable after they have approached each other below the distance of a block size. At the heart of non-linear systems are bifurcation points, where a system can evolve either toward one state or toward another, yielding very different outcomes. The infinite accuracy of measurement at the bifurcation point that would be required to predict which state a system will assume is impossible. Hence, the unpredictability of complex systems is rooted in the physical limit on the precision of obtainable knowledge [23].

In the vicinity of decision points (bifurcations), predictability under the law of large numbers is not valid anymore. While the molecular interactions in chemical reactions or mechanical motion far from equilibrium do not change from those in equilibrium, they do become dependent on global conditions. The transition from the time-reversible formulas of chemistry or mechanics to the algorithms of emerging processes is accomplished through a unique form of a non-local transformation, in which the homogeneity of the space-time structure is eliminated and both entropy and time become operators in an event space. This transition involves an internal time that is derived from the uncertainty associated with the trajectories in unstable dynamic systems. The transformation leads to a spatiotemporally non-local description [19]. Intelligibility is limited to the eigenvectors and eigenvalues of the operators.

6. Complexity and Chance Are Mirror-Images of the Same Events

Any discrete process, which maps the interval onto itself and whose reverse process is double- or multiple-valued, fulfills the Li and Yorke criterion for chaos. Sequences of numbers obtained by iteration of such processes (t indicates iteration steps) are mirror images of paths of random walks in the reverse, branched processes [24]. A deterministic system may be defined as one whose future and past are both unique functions of the present. On this basis, the difference equation (the logistic map equation) is

$$x_{t+1}=k·x_t·\left(1-x_t\right),$$

the inverse being

$$x_t=\frac{1}{2}\pm \frac{1}{2}\sqrt{1-\frac{4x_{t+1}}{k}},$$

which is not deterministic because there are two different pasts of equal probability, even though the future is a unique function of the present. Iteration of the latter equation is still possible if rules are defined to choose one of the two values. Generally, for any one-dimensional map, what is unstable in the iteration of the forward process is stable in the iteration of the inverse process and vice versa. If iteration of the inverse equation is performed with random choices at each step, there will be no convergence toward a point or a cycle. The mirror image of this iteration will be a chaotic sequence produced by the initial difference equation, as is evidenced if the last number of a sequence produced by the inverse equation with random decision making at each iteration is used as the first number of an iteration of the initial difference equation, which will yield the same sequence in reverse. That is, the same sequence of numbers can be produced by a stochastic as well as by a deterministic process. Chaos in a one-dimensional difference equation can be viewed as reversed random walk [25].

7. Conclusions

The present analysis shows that the perception of a dichotomy between determined and undetermined elements of the world is an artifact, generated by the human mind. A distinction between a “random” occurrence and a complex “deterministic” occurrence is impossible to make as it has no correlate in nature. Models in phase space (typically plotting impulse versus time) or event space (an operator space) inherently have insufficient resolution to enable an assessment in favor of one or the other (Section 5). Strings of numbers can reflect deterministic chaos (a form of complexity) or randomness, depending on the order in which they are encountered (Section 6). By contrast, we are enabled to measure new information produced by an event (through the Lyapunov characteristic exponent) and the degree of complexity inherent in the event (through the information dimension) in order to gain relevant, quantitative readouts (Section 3). Rather than a dichotomy between chance and necessity, quantifiable information evolution and information dimension, as well as entropy, are suitable scientific measures for the complexity of an occurrence, which can be interpreted as being reflective of the “ordered” and “chance” components that contribute to this occurrence [5].

The universe is one coherent entity. Nevertheless, the traditional, reductionist description of nature has typically categorized observations and created opposites that are purportedly mutually independent, thus generating partial entities of the world that have been treated as detached from one another. Such categorizations had the benefit of achieving tractability through a quasi-linearization of descriptions [26]. The construct of a dichotomy between necessary events and chance events enabled us—especially in the eras preceding computers and complexity research—to make meaningful observations about parts of the world. However, it has given rise to centuries of discussions about world views that span the entire spectrum from deterministic fatalism to indeterministic nihilism. The results from non-linear systems research must lead to the conclusion that the dichotomous perception itself is untenable. There is no meaningful answer to the question whether a complex event has originated in chance or necessity. Only readouts for its extent of complexity are meaningful assessments.

Human societal value systems are often rooted in the idea of name recognition as a modality of acknowledgement for achievement. For fame we compete. Yet, the present analysis has shown that information generally has a finite lifespan. Eventually, name recognition—like all information—will be replaced and the name will turn into smoke. Societies are emergent systems wherein new information will eventually entirely replace old information. By contrast, miniscule actions at decision points (bifurcations) of complex systems can have an irreversible and durable impact on the future. In keeping with natural laws, actions—even small representations thereof—may deserve priority over attribution of credit as the measure for contributions to progress.