Local Compositional Complexity: How to Detect a Human-Readable Message

Abstract

Data complexity is an important concept in the natural sciences and related areas, but lacks a rigorous and computable definition. This paper focusses on a particular sense of complexity that is high if the data is structured in a way that could serve to communicate a message. In this sense, human speech, written language, drawings, diagrams and photographs are high complexity, whereas data that is close to uniform throughout or populated by random values is low complexity. I describe a general framework for measuring data complexity based on dividing the shortest description of the data into a structured and an unstructured portion, and taking the size of the former as the complexity score. I outline an application of this framework in statistical mechanics that may allow a more objective characterisation of the macrostate and entropy of a physical system. Then, I derive a more precise and computable definition geared towards human communication, by proposing local compositionality as an appropriate specific structure. Experimental evaluation shows that this method can distinguish meaningful signals from noise or repetitive signals in auditory, visual and text domains, and could potentially help determine whether an extra-terrestrial signal contained a message.

1. Introduction

Complexity as a concept arises in various forms in various fields of study. The domain we are interested in here consists of static pieces of data that do not evolve and change over time, meaning that some characterisations of complexity do not apply, such as that from complex systems theory, which involves the interaction of parts and behaviour prediction [1]. Common approaches to the task we are interested in, namely quantifying complexity of static data, include Shannon entropy and Kolmogorov complexity from algorithmic information theory. These methods quantify complexity as the amount of information/length of the algorithm required to represent the data perfectly. A significant drawback of these measures, with respect to the present task, is that they fail to identify random data as low complexity, and in fact give them close to a maximum score. This observation has prompted some to suggest using the inverse of these measures to detect communication signals [2], which correctly gives noise a low score, but only at the cost of giving a very high score to uniform repetitive data [3], which are not able to convey a complex message either. This is a fundamental problem in measuring complexity: there are several effective methods that define a scale from uniformity at one extreme to random noise on the other, but the data we would like to give the highest score to, such as human language and real-world images, sits in the middle, not close to either extreme. One might try to define a Goldilocks zone where, e.g., Shannon entropy is neither too high nor too low, as the location of communicative signals. This is the wrong approach for two reasons. Firstly, there is no principled way to select the boundaries of such a zone. Secondly, it is easy to construct low-complexity, meaningless data that would fall inside it. Consider a bit-string consisting of all 0s, which has a minimum Shannon entropy of zero, and imagine replacing the first p% of it with independent uniformly random bits, i.e., coin flips, and varying p from 0 to 100%. The bit-string remains meaningless throughout, but because its Shannon entropy increases steadily from 0 to the maximum score of 1, it would cross any intermediary zone.

While the description length needed for perfect reconstruction is related to meaningful complexity, it does not tell the full story, and so Kolmogorov complexity, Shannon entropy and related measures are incomplete for quantifying meaningful complexity.

The solution is to consider descriptions of the data divided into two parts, with only one counting toward complexity. For communication, portion A, which is the meaningful portion, conforms to a locally compositional structure, meaning a tree-like structure where the leaves are the atomic constituents of the data such as image pixels or text characters. Portion A does not (normally) specify the given data exactly, but rather describes an approximation. Portion B, then, fills in the missing details. Portion B conforms to a default structure that makes minimal assumptions about the data and that we do not regard as meaningful. Both parts have a well-defined length, equal to the number of bits they comprise. The optimal description is chosen as that with the overall smallest length (sum of portion A and portion B), and the local compositional complexity (LCC) score is then the length of portion A within this optimal description. Effective complexity [4] employs essentially the same solution, except it generally requires specifying the meaningfully structured features from the outset, and then separately describing these features and the remaining, meaningless, portion. In our framework, on the other hand, we first compute a single optimal description for the entire data, and then analyse it as two separate portions. This means that whether a feature is regarded as being present at all depends on all aspects of the data. The same data could admit either a structured or unstructured description, and which one is selected depends on which one is more efficient. Portion A is analogous to the description of the macrostate of a physical system, and Section 2.2 outlines how this framework could be used to provide a more objective characterisation of a macrostate. A comparison can also be drawn to compression: the representation consisting of both portions A and B corresponds to lossless compression, because it can exactly reproduce the given data, while the representation with just portion A corresponds to lossy compression, because it reconstructs an approximation with the less important aspects being ignored.

The contributions of this paper are:

• a novel approach to measuring complexity by dividing the description into structured and unstructured portions;
• a sketch of how this idea can be used to more objectively quantify the entropy and macrostate of a physical system;
• the proposal of local compositionality as the structure that, in the case of communicative signals, the structured portion should conform to;
• the development of the LCC score, a computable metric for complexity that can be applied to a variety of domains;
• the empirical testing of the LCC on text, image and audio data, showing that it agrees with our intuitions regarding meaningful complexity;
• a demonstration of the potential of the LCC score for detecting non-human communication, showing that it can identify the Arecibo message as meaningful, as well as determine its correct aspect ratio.

2. Local Compositional Complexity (LCC) Score

The two novel elements of the LCC score are the method of dividing descriptions into two parts, and the choice of local compositionality as the appropriate structure for portion A. The following sections describe both in detail.

2.1. Two-Part Descriptions

The division of descriptions into parts one and two is similar in spirit to two-part codes in the statistical frameworks of minimum description length (MDL) and minimum message length (MML). Both MDL [5,6] and MML [7] descriptions consist of a model (portion A) and the data input to the model (portion B). The standard division in MDL into model and input is slightly different to the one made here. Here, the first part of the description generally contains both something like a statistical model and the use of that model to approximate the given data; the second corrects the errors and fills in missing details in that approximation. Two part divisions are also made in the theoretical measures ‘Sophistication’ [8,9] and ‘effective complexity’ [4,10], which are discussed in Section 4.

For example, if the data consists of a set of points in ${\mathbb{R}}^n$, and we consider a fit clustering model as the model of the data, then portion A will consist of the cluster centroids in the fit model and the cluster labels for each data point. This gives an approximation to the given data where every point is replaced with its assigned cluster centroid. Portion B, then, is the residual portion that corrects this approximation to the precise data values. We search for the description that has the overall shortest length, adding together portion A and portion B. Assuming 32-bit precision, each cluster centroid would require 32n bits, and each cluster label would require roughly $logK$ bits, where K is the number of clusters (though this can be further optimised, as discussed below). For some data points, this two-part description of approximation plus correction is more expensive than a simple one-part description that specifies data points directly. This simple one-part description can just be to describe the floating point values of the data point directly, which would take 32n bits. For each data point, we use whichever is smaller, the two-part or one-part description. All one-part descriptions fall into portion B, because they do not contribute to any consistent structure or pattern in the data. We will see in the experiments in Section 3, that the total description length for random noise is very high, higher than for data such as speech or real-world images, but almost all the description resides in portion B, so the LCC score is close to zero. For uniform, repetitive data, on the other hand, the overall description length is very low, so even when it mostly resides in portion A, the LCC score still ends up being low. Note the trade-off of adding more clusters: each additional cluster increases the model length, by 32n in our example, and increases the length of each cluster label by roughly $logk+1-logk\approx \frac{1}{k+1}$, but it means there are more cluster centroids to better cover the data, and so will tend to reduce the average residual cost. The LCC score uses the MDL principle and selects the number of clusters that minimises the overall cost. This particular usage of the MDL principle, that of of a criterion for selecting the number of subsets in a partition, has been employed by a number of previous works  [11,12,13].

2.2. Entropy, Macrostates and Microstates

This division of descriptions into a structured portion A and an unstructured portion B is a general framework not specific to communication. Indeed, an analogy can be drawn to describing the state of a physical system in statistical mechanics: portion A corresponds to the macrostate and portion B to the microstate given the macrostate.

Typically, the choice of what form the macrostate should take is made depending on what ‘state variables’, such as temperature and pressure, the experimenter is interested in or can observe/control. The apparent subjectivity in the choice of what the state variables are, plus the fact that the entropy of the system depends on this choice, has led to the characterization of entropy as an “anthropomorphic” property [14]. In our framework, there is a precise criterion for whether to include a particular variable in the macro-description, namely whether it decreases the overall description length of the microstate. (For simplicity, we assume the microstate itself is known, but this could be replaced with the expected description length over the set of possible microstates). Once we pick a set of possible state variables, for example all conserved or invariant quantities, we can use this criterion to select, for a particular state, which subset of the variables should be used, and how many bits of precision should be devoted to them. Note, this refers to the accuracy with which we represent a quantity, not to the accuracy with which we can measure it.

For example, suppose temperature was one of the possible state variables, and assume that, for any precisely known temperature, we can calculate a distribution over microstates, then, for a given distribution over temperatures, we can marginalise to calculate another distribution over microstates. Let $q\left(s\right|\tau )$ be the distribution over microstates s resulting from considering the temperature of system up to $\tau$ bits of precision. Then, under the optimal encoding scheme as given by the Kraft-McMillan inequality [15,16], the number of bits needed to select the single correct microstate from this distribution is $-{log}_{2}q\left(s\right|\tau )$. (See Appendix B for further details.) Thus, our theory suggests that the correct number of bits of precision to use, ${\tau }^{*}$, is given by

$${\tau }^{*}=\underset{\tau }{argmin}\tau -{log}_2{q}_s(\tau ,s),$$

(1)

where ${\tau }^{*}=0$ means not including temperature as a state variable at all. This would imply that temperature is not an inherent feature of the system. Just because a feature can be measured and described, does not mean it is correct to do so. For example, the feature ‘stripe pattern’ could be measured and described of certain visual objects even when it is not warranted. One could choose to regard, not just zebras, but all Equidae (including horses) as striped, and say that what appears as a fully white horse, has white stripes on a white background. This would not exactly be false, but it unnecessarily complicates the description of the white horse, and so it seems wrong to include. The two-part framework used in this paper suggests that, if ${\tau }^{*}=0$, then temperature is to the system as stripes are to the white horse. In practice, however, other factors, such as measurability, may affect the optimal choice of state variables. In an experiment specifically investigating temperature, there is of course no problem considering it even if ${\tau }^{*}=0$.

• Entropy vs. Complexity

Entropy and complexity are two important concepts in information theory and science more broadly. Given the optimal (i.e., shortest) description of a system, where portion A consists of the macrostate as we have outlined, it is easy to see (shown in Appendix A) that the expected length of portion B is proportional to the Gibbs entropy. Thus, one could claim that a decrease in Gibbs entropy is equivalent to an increase in complexity, assuming the overall description length remains constant. However, it is likely that this assumption would often be violated in practice: even the same system can have a varying description length as it moves through different configurations. This is shown graphically in Figure 1. Comparing row two to row one, we see increasing entropy means decreasing complexity, because the two rows happen to have the same total description length. However, rows three and four show that, in the realistic case that total length can vary, high entropy can correspond to either high or low complexity. Entropy is the length of the non-meaningfully structured portion of the optimal description of the system, whereas complexity is the length of the meaningfully-structured portion. According to this argument, the two notions of entropy and complexity are, in general, independent, and this suggests, among other things, that an increase in entropy in accordance with the second law of thermodynamics does not necessarily imply a reduction in complexity.

2.3. Local Compositionality

Turning specifically to human communication, I claim that the appropriate structure for portion A is local compositionality. Compositional means that the structure should be tree-like, in that it represents multiple different parts of the data as ‘children’ of some shared ‘parent’, and that this parent can then be a child of another parent etc. Local means that, assuming the data has a notion of distance between points, so that some parts are close to each other and some far away, then only parts that are close together should share a parent. An example of this structure is grammatical parse trees of natural language text. We can regard natural language as represented by a locally compositional structure extending from phonemes, through morphemes, syntactic constituents, sentences and discourse elements, where the large majority of compositions (by some theories, all compositions [17]) are local. For example, if the words ‘the’ and ‘cat’ are adjacent, then they can be composed into a single grammatical unit ‘the cat’. Local compositionality is also the structure found in human perception of visual objects, where lines combine to form shapes, and then objects etc. [18,19].

We do not want a single tree that encompasses all of the data, as multiple different parts of the data should be able to correspond to the same node in the tree: the combination of ‘the’ and ‘cat’ can appear multiple times in the data but should only occur once in the tree. To allow for this, we first describe a tree-like structure, then invoke the nodes it contains to specify the data. We term the former the ‘model’, and the latter, the ‘index’. For human-readable messages, this is the structure that portion A descriptions should conform to. Portion B then consists of an encoding scheme that makes minimal assumptions about the data structure. Examples of these default encodings are detailed below. Generally, they involve encoding each part independently and assuming a normal or uniform distribution.

In the clustering example from Section 2.1, the model consisted of the fit cluster centroids, and the index consisted of the sequence of cluster labels. In the physical state example from Section 2.2, the model consisted of a set of macroscopic quantities (state variables), along with the physical equations by which they are related, probabilistically, to microstates, and the index consisted of values for these state variables, up to finite precision. Portion A need not necessarily comprise tree-like structures, in principle one could choose any sort of model. However, for many choices of model, it is possible to find data which gets a very short description, but appears random and meaningless to humans. An example is given in Appendix D. In order to align with what is meaningful to humans, I propose that tree-like structures, specifically local compositionality, is an appropriate choice of model type.

The models used to compute the LCC score are not assumed to be like the process that, in some sense, ‘really’ generated the data. The framework is just that we are given a piece of data, with no prior ideas about it, and then must search among a set of possible representations for the most efficient one. This is in keeping with the MDL philosophy of focussing on finding efficient descriptions, rather than a true distribution or generating function [6]. The LCC score is therefore not a property of a distribution but of a given piece of data or dataset. It generally increases as the amount of data is larger, which reflects the fact that it is a version of information content and that more data can generally convey more information.

We do not have a formal proof that the LCC score will always give a high score to data if and only if it contains a human-readable message. A proof is given in [20] that, for their very similar method, white noise images will, with high probability, get a score of zero, but even that is not a formal guarantee against false positives, because white noise does not constitute all non-meaningful-to-human images. It is difficult to see how any such metric could have a formal proof of correctness because we do not have a formal account of what makes something a human-readable message–in fact one way to interpret the LCC score is as an attempt at such a formalisation. Instead, the LCC score is evaluated empirically. Experiments in the following sections show that it gives a high score to a set of things we know to be meaningful to us, and a low score to a set of things we know not to be.

3. Implementation and Results

This section describes more concretely the implementation of the LCC score for data from different domains. It then presents experimental results showing that it consistently gives a low score to random data and to repetitive/uniform data, and that the only data that gets a high LCC score is that which we know to be meaningful in the sense that it carries a human-readable message.

3.1. Discrete Data

In the domain of strings, such as natural language text, the model is a codebook mapping single characters to sequences of characters, and the index is a string composed of characters that appear as keys in the codebook, along with a special character x, indicating ‘unspecified’. The data is specified by proceeding from the top of the codebook, and replacing all occurrences of each key that appear in the index string with the corresponding value, then replacing all x characters with specific values as given in a third string, the residual string. The reverse process of replacing all occurrences of a subsequence with an index to the code, we refer to as ‘aliasing’. This conforms to the local compositionality structure, and allows an efficient approximate search for the optimal representation (see Appendix E). The fact that values in the dictionary can be multiple characters long means the tree is not restricted to be binary and can express n-ary relations, as discussed for example in [21]. The complexity score is the total number of bits needed to represent the codebook and the index string, under the optimal encoding, ignoring the residual string.

Adding more entries to the codebook reduces the number of indices in the input string, causing a reduction in cost, but increases the average length of each index and size of the codebook itself. For random sequences of characters, the decrease will almost always be outweighed by the increase, meaning almost nothing is added to the codebook and all the information is contained in the residual string, so the LCC score will be low. Repetitive uniform data, on the other hand, can be represented accurately by only a few codebook entries, so the codebook will be small and will admit efficient indexing, so it will also get a low score.

3.1.1. Discrete Worked Example

Let $\mathcal{A}=\{a,b,c\}$, and

$$\begin{array}{c}S=ccabacbbbcaacbcccbcaaacbbcccabbbaacbaaabcabbbcabbbcacbb.\end{array}$$

Then, one possible encoding is given by

$$\begin{array}{c}C=\left(\right(g,ace),(f,cb),(e,bb),(d,aa\left)\right)\\ I=xxxxgxxdfxxfxdgxxxxexdfdxxxxexxxexxg\\ X=ccabbcccccccababcabcabc.\end{array}$$

If we begin with I, then replace each codebook character with the corresponding string, followed by replacing the x’s with the residual string X, we get the following sequence of strings:

$$\begin{array}{cc}\hfill I=& xxxxgxxdfxxfxdgxxxxexdfdxxxxexxxexxg\hfill \\ \hfill & xxxxacexxdfxxfxdacexxxxexdfdxxxxexxxexxace\hfill \\ \hfill & xxxxacexxdcbxxcbxdacexxxxexdcbdxxxxexxxexxace\hfill \\ \hfill & xxxxacbbxxdcbxxcbxdacbbxxxxbbxdcbdxxxxbbxxxbbxxacbb\hfill \\ \hfill & xxxxacbbxxaacbxxcbxacbbxxxxbbxaacbaaxxxxbbxxxbbxxacbb\hfill \\ \hfill & ccabacbbbcaacbcccbcaaacbbcccabbbaacbaaabcabbbcabbbcacbb=S.\hfill \end{array}$$

The number of bits needed to represent S under this encoding is then $L\left(C\right)+L\left(I\right)+L\left(X\right)=102.84+59.81+33.90=196.55$ and the computed LCC score is then $L\left(C\right)+L\left(I\right)=102.84+59.81=162.65$. This particular encoding is the one found by our search algorithm, slightly simplified for demonstration purposes. However, this search is approximate only, so we cannot guarantee that the above encoding is optimal for the present example.

3.1.2. Complexity of Natural Language Text

Figure 2 shows the score for the first 7500 characters of 10 random Wikipedia articles (precise list in Appendix F) each of different natural languages: English, German and Irish, along with the score for other types of strings. The total height of the bar for each string type is its total description length. This is broken down as model cost (dark green), index cost (light green) and residual cost (orange), with the LCC score (written in green text) being the sum of the first two. Random strings, ‘rand’, get a score of 0, despite requiring the highest number of bits to represent exactly, because the cost resides entirely in the residual portion. Simple, artificial English, ‘simp-en’, consisting of the same five English words selected randomly, gets a significantly lower score, and very repetitive strings, ‘repeat[2510]’, consisting solely of repetitions of the same random string of length 2, 5 and 10 respectively, get a very low score. The slightly lower score for Irish compared to German and English is expected, given that its alphabet uses only 18 letters, vs 26 for English and 30 for German. One may then expect German to receive a proportionally higher score than English. The fact that it instead receives only a slightly higher score than English is, we expect, due to some feature of the morphology, or perhaps syntax, of the three languages. One possibility is consonant clustering, which occurs more strongly in German and Irish than in English, so e.g. “sch” in German or “bhf” in Irish contains less information than the sum of the constituent letters, thus lowering the scores for these languages.

3.2. Continuous Data

Unlike discrete data, such as text, continuous data, such as images and audio, allow two different values to be arbitrarily close to each other. For continuous data, we cluster progressively larger localised regions, i.e., image patches, using a multivariate Gaussian mixture model (GMM). The representation model then consists of a sequence of GMMs, specified by the means and covariances of each cluster. The index consists of an n-dimensional tensor (for images $n=2$, and for audio $n=1$), where each location is either an index to a cluster of the GMM or the special symbol x. The residual consists of the encoding of each point in the input, with respect to its assigned cluster in the GMM, plus the values of each x.

A similar idea was used by [20], except that their method only applied to the image domain, and included a complicated entropy calculation that was not justified in a theoretical framework and that did not transfer to other data types.

Further details, including a full formal description for the continuous calculation of LCC score, is given in Appendix C.2.

3.2.1. Continuous Worked Example

Here we present a simplified worked example for the continuous domain. Suppose the input image is as in Figure 3 (top). The first step is to run MDL clustering on the set of pixel values across the whole image. In this case, the result is two clusters corresponding to blue-ish and green-ish pixels respectively. Some pixels are not assigned any cluster, and are instead encoded directly as part of the residual, because they do not fit efficiently into a pattern with the rest of the data. Writing x, for such pixels, we get the assignment as shown in Figure 3 (middle). Most of the pixels in the blue cluster are in the bottom left and most of those in the green cluster are in the top right. In total, there are 19 blues, 18 greens and 27 zeros. The Shannon information content of this distribution of labels is

$$-19{log}_2{\displaystyle \frac{19}{64}}-18{log}_2{\displaystyle \frac{18}{64}}-27log{\displaystyle \frac{27}{64}}\approx 33.29+32.94+33.61=99.59.$$

The model cost, meanwhile, comprises the bits for the RGB values for each of the two clusters. At 32-bit precision, this gives $2\times 3\times 32=192$. The next step is to cluster patches in the image. By ‘patch’, we here mean a $2\times 2$ region of pixels, see Appendix C.2 for a full description of the computation of patch cluster labels. In the full method, we consider overlapping patches, but for simplicity here, we will show non-overlapping of size $2\times 2$. This would give three clusters, those mostly composed of 0 s, 1 s, 2 s. The result is as in Figure 3 (bottom) (These are for illustrative purposes only and may differ from those output by our method). There are seven 0 s, three 3 s, three 4 s and three 5 s, giving information content

$$-3{log}_2{\displaystyle \frac{3}{16}}-3{log}_2{\displaystyle \frac{3}{16}}-3log{\displaystyle \frac{3}{16}}-7log{\displaystyle \frac{7}{16}}\approx 7.25+7.25+7.25+8.35=30.08.$$

Here, there are three cluster centroids, which takes $3\times 3\times 32=288$. The total score for this image is then 192 + 99.59 + 288 + 30.08 = 609.67. Because this toy image is small, the cost is dominated by the model cost, but for larger images, it is dominated by the index cost, as can be seen in Figure 4.

Figure 3. (Top): Toy example image. (Middle) The MDL cluster labels for this image on the first level. One this level, two clusters are found, corresponding to blue-ish and green-ish pixels, and the meaningful portions, i.e., those pixels assigned a cluster centroid and not marked with ‘x’, are mostly in the bottom left and top right. (Bottom) The MDL cluster labels on the first level. One this level, there are again two clusters, now corresponding to green-ish and blue-ish patches, and the meaningful portions are entirely in the bottom left and top right.

Figure 3. (Top): Toy example image. (Middle) The MDL cluster labels for this image on the first level. One this level, two clusters are found, corresponding to blue-ish and green-ish pixels, and the meaningful portions, i.e., those pixels assigned a cluster centroid and not marked with ‘x’, are mostly in the bottom left and top right. (Bottom) The MDL cluster labels on the first level. One this level, there are again two clusters, now corresponding to green-ish and blue-ish patches, and the meaningful portions are entirely in the bottom left and top right.

3.2.2. Continuous Results

Figure 4 shows the LCC score for various types of images: Imagenet and Cifar10 are two datasets of natural images depicting a single, identifiable object. ‘halves’, and ‘stripes’ are two simple uniform types of image we create, the former are half black and half white, with the dividing line being at various angles, and the latter consists of black and white stripes of varying thickness and angles. We also include white noise images, ‘rand’. The natural images consistently get the highest score, and a very similar one across the two datasets, Imagenet and Cifar10; the simple uniform patterns get a moderate score, and random noise gets the lowest score of close to zero, even though the total cost is the highest of all image types. It is also striking that the fraction of the cost occupied by the residual description is much higher for images than for text. In order to make visible the differences in the LCC score portion, the residual portion of the bars in Figure 4 has been reduced by a factor of 5. The actual ratio of residual cost to LCC score (model cost plus index cost), is 20:1, even for the real-world images in Imagenet and Cifar10, and even higher for the other images. For natural language text, in contrast, this ratio is over 1:3. This is consistent with the conception of text as information dense: if we regard ‘information density’ as the ratio of the length of the meaningful portion of the description to the total length of the description, then our results find natural language text to be ∼60 times as information dense as real-world images.

Our method can be applied to audio in the same way as it can be to images, except that we first convert the audio signal to a spectrogram.

Figure 5 shows the resulting scores for various types of audio: human speech in English, Irish and German; random noise, sampled from a uniform distribution; simple, repetitive signals, a tuning fork and a continuously ringing electronic bell; and some examples of ambient noise: rainfall and muffled crowded human speech (walla). We see the same patterns as before: human speech receives the highest score, random noises have a high total description length but almost all of this is in portion B, so they receive a very low score, and simple repetitive signals have a score in between the two. It is striking that the three natural languages receive such a similar score to each other, which is consistent with the general principle that, phonetically, all human languages are roughly equally efficient at conveying meaning [22,23,24].

3.2.3. Use in Compression

The problem of efficiently representing data is studied in the context of compression, which is the process of encoding data to reduce its size by exploiting redundancy and patterns. The entire description, portions A and B, constitutes a sort of lossless compression, while discarding portion B would constitute a lossy compression. The original image can be approximately reconstructed from portion A alone, by replacing the information in portion B with random samples from the corresponding cluster distributions. Figure 6 shows such reconstructions for an example real image and noise image.

The LCC representation has not been optimized for compression. Specific compression algorithms, such as JPEG, carefully preserve information most salient to the human eye. For the real-world image, JPEG provides a much better quality-compression trade-off than the reconstruction in Figure 6, reaching over 200 before a similar degradation in quality. For the noise image, the LCC compression ratio is 114, while the JPEG is only around 30 for a similar quality. Carefully examining the noise reconstruction from the LCC method, the pixel values are actually completely different, yet the LCC representation correctly identified that these differences are not structurally meaningful, giving a large compression ratio. The intention of the LCC representation is not compression, but rather to be theoretically optimal among a certain class of representations, enabling reasoning and quantification of meaningful structure. Clearly, JPEG affords better compression overall than the simple compression method shown in Figure 6. However, it is interesting that it provides a reasonable compression as a by-product, and this may suggest a direction for future work, especially in the context of compressing noisy data.

3.3. Arecibo Message

If the earth were to receive a communication signal from elsewhere in the universe, even before decoding what it meant, an interesting and important question is whether we would be able to detect that it contained a message at all. We cannot assume the language of this message would be especially similar to any found on Earth. However, if we make no assumptions about its possible structure, then any bit-string could be associated with any meaning and the problem would be impossible. We need to assume enough about a possible alien language to have a chance of detecting it, but not so much that we rule out those that would be very unfamiliar to us but still potentially decipherable. The assumptions of locality and compositionality are reasonable choices in this respect, and the results presented so far already show that the present method has a strong ability to detect signals of this sort.

Now, we directly consider the case of interplanetary communication, by computing the LCC score for the Arecibo message, a sequence of 1679 bits broadcast by a group of Cornell and Arecibo scientists, including Carl Sagan and Frank Drake, in 1975 [25]. When depicted as a 73 × 23 bitmap image, as shown in Figure 7 (top left), it shows a series of simple shapes and patterns designed to convey information about earth and humankind. When depicted with an aspect ratio that differs by more than +/−1 from this, these shapes are lost and they become unintelligible to human eyes. The signal length of 1679 was chosen to be semi-prime so it only exactly factors in two unique ways: 73 × 23 or 23 × 73. We could imagine that the receiver is unsure of the exact start and end point, which introduces error in the possible length and greatly increases the set of possible aspect ratios. In any case, the problem of singling out the correct one of 73 × 23 based only on the resulting image contents is an instructive one for the present purposes.

Figure 8 shows the LCC score given to the image for varying aspect ratios. For each ratio, we add the minimum number of bits to the end to allow the number to factor in that ratio, with values independent and random with the same on probability as the rest of the signal. This is a small number of additional bits, and often only 1 is needed; the average is 22, which is ∼1.3% of the total.

The red dotted line shows the maximum score that could be given to a random bit-string (obtained at Bernoulli p = 0.5). The correct aspect ratio receives the highest score, of over 9000, and clearly exceeds the threshold, while almost all others do not. (Note that we are representing the message as a continuous image, meaning the total number of bits is 32 × 1679 = 53,728). The second highest, and also above the red line, is 1 pixel in width away from the correct one, with a ratio of 77 × 22 (with 15 random bits at the end). The image at 77 × 22, shown in Figure 7 (top right), is in fact still somewhat intelligible. The human shape midway down the image, and the antenna shape at the bottom, have merely been slanted to the left. Given that the LCC score does not know what a human or antenna or any other particular shape is, it is reasonable that it considers the 77 × 22 aspect ratio to be similarly meaningful to 73 × 23. Indeed, it is striking that, without any of this information on the shape of familiar objects, or previous examples of such objects, the LCC score can identify that the signal for the Arecibo message is likely to contain a human-readable message, and the particular way the data should be represented for this message to be human-readable. If humans received an extra-terrestrial message in the same form as the Arecibo message, we could not expect it to depict objects we recognise, such as human bodies and radio antennas, but if it were organised in a locally compositional structure, then the LCC score may be able to identify it meaningful, as well as the aspect ratio in which to read it.

The complexity measure of [26] has also been applied to the Arecibo message, and was also able to detect the correct aspect ratio. That measure gave the lowest score to the correct aspect ratio, in contrast to ours which gives the highest. This is because the overall description is indeed shortest for the correct aspect ratio. However, in the LCC framework, a higher fraction of that description is structured, and so the overall score is higher. This again highlights how the LCC score is measuring meaningful structure, rather than, as existing methods do, approximating Kolmogorov complexity. As I have argued, approximating Kolmogorov complexity is not able to make the correct three-way distinction between simple, meaningful and random. For example, a method like that of [26] will give the lowest score to uniform data.

4. Discussion

The first main idea behind the LCC score is to divide an information-theoretic description into two parts, and to take the length of only the former, structured part as a measure of complexity. This is similar to some purely theoretical computer science work that seeks to modify Kolmogorov complexity to not give a high score to random strings. ‘Sophistication’ [8,9] and ‘effective complexity’ [4,10], propose to divide the generative algorithm for a given string into a section that describes a set of which it is a typical member, and a portion that selects it from within that set, and take the length of the former as the complexity. Our method differs in that it determines the boundary between the structured and unstructured section dynamically, as part of a single optimisation of the overall description length. Additionally, while these existing methods are purely theoretical and often uncomputable, ours can be implemented to actually compute accurate complexity scores for a variety of data, as shown by our results.

The second aspect of the LCC score, local compositionality, has a connection to assembly theory (AT) for measuring molecular complexity [27,28]. AT measures the complexity of a molecule as the number of steps needed to assemble it from atomic constituents, allowing reuse of previously assembled intermediary structures at the cost of only a single step. This is analogous, in our framework, to aliasing, where we pay the cost of a substring as it appears in the codebook, and then, every time it appears in the input, pay only the cost of indexing that entry. AT says you can reuse any substring as a single step, which is equivalent to assuming constant lookup time in the codebook, however strictly speaking, the amortized lookup space (and hence, time) is always bounded below by the Shannon entropy of the codebook entries in the input string [29], so can grow arbitrarily. In our framework, the index length is proportional to the substring’s negative log frequency in the input string, which means we respect this lower bound.

Similarly, when AT is applied to collections of molecules, some of which may have been assembled by chance, it assumes that any appearing more than once could not have been due to chance [30]. Specifically, the $-1$ in the numerator of Equation (1) in [30] is so that molecules contribute to the assembly index of the ensemble if and only if they have two or more copies. This may work to exclude chance occurrences in the case of molecular complexity, where the space of possibilities is far larger than the given set, but in general items can occur many times due to chance alone. Rather than a threshold of 1, our framework produces a threshold strictly greater than 1, given by

$$1+{\displaystyle \frac{{\sum }_i{log}_2\frac{f_i}{f_i^{\prime }}}{l}},$$

(2)

where l is the description length of the substring, and $f_i$ and $f_i^{\prime }$, are, respectively, the the frequency of the ith character in the non-aliased portion before and after aliasing (derivation in Appendix H). In the case of random strings, as measured in

Table A1. Full, raw results for text with std. dev. in parentheses.

	Model Cost	Idx Cost	LCC Score	Residual Cost
text-en	1442.38 (58.03)	13,195.02 (225.99)	14,637.40 (195.98)	9393.79 (457.05)
text-de	1375.24 (46.37)	13,342.95 (122.85)	14,718.19 (101.97)	9653.65 (168.47)
text-ie	1756.35 (142.88)	11,756.63 (335.73)	13,512.98 (220.23)	8647.10 (219.83)
simp-en	20.26 (0.00)	5801.94 (0.70)	5822.21 (0.70)	0.00 (0.00)
rand	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	46,801.71 (3.60)
repeat2	2.00 (0.00)	0.00 (0.00)	2.00 (0.00)	0.00 (0.00)
repeat5	11.21 (0.27)	0.00 (0.00)	11.21 (0.27)	0.00 (0.00)
repeat10	29.13 (0.52)	0.00 (0.00)	29.13 (0.52)	0.00 (0.00)

, we observe that substrings of length up to $\sim 10$ characters often occur multiple times by chance alone. If we used a fixed threshold of 1, these substrings would mistakenly be regarded as meaningful. However, we also observe that the number of times they occur is always less than the threshold in (2), so our method still correctly identifies that they did not arise by any meaningful process. The inverse dependence on l means that, for larger substrings, (2) approximately recovers the threshold of 1, as used by [30].

Entropic slope [31], is another method aiming to recognise communicative data. It measures the decrease in entropy of each term in a sequence of symbols as you condition on more and more previous terms. That is, in the terminology of language modelling, the dependence of the perplexity on the size of the left context (though entropy is simply calculated empirically, without any specific language model). A score close to -1 indicates high complexity, because it means the symbols individually are non-uniform, but yet can be predicted more and more accurately with more and more left context. Random strings see no drop in entropy with increasing left context, so have a slope of zero, while repetitive sequences have a low entropy for even a small left context, so cannot drop much further. Thus, in the text domain, entropic slope is able to make the three-way distinction between random, uniform and meaningful. However, it has also been applied to the Arecibo message, and here it gives it a relatively low score [32]. The unique feature of the LCC score is that it is able to identify meaningful data across a variety of domains, as well as its clear theoretical foundations in minimum description length and Occam‘s razor.

5. Conclusions

This paper described a way to measure the amount of meaningful complexity, of the sort found in human communication, present in a piece of data. First, a general framework was described for constructing two-part descriptions that can be used to quantify any sort of complexity or structure in data. This framework connects to the concept of complexity in statistical mechanics, implying that complexity should not be equated with low entropy, and suggesting a method for a more objective characterisation of the macrostate of a physical system. Then, the particular structure of local compositionality was proposed as being appropriate for modelling human communication and perception. Coupled with the framework for two-part descriptions, this led to a metric for meaningful complexity, termed the LCC score. Experimental results showed that the LCC score correctly distinguishes meaningful data from both simple uniform data and random noise, across the domains of text, images and audio. Finally, when applied to a message broadcast from Earth constructed by human cosmologists, the LCC score identified that this signal contained a message, as well as the appropriate form in which to read the message. This suggests a use for the LCC score in the study of non-human communication.