E-Mail Alert

Add your e-mail address to receive forthcoming issues of this journal:

Journal Browser

Journal Browser

Special Issue "Kolmogorov Complexity"

Quicklinks

A special issue of Entropy (ISSN 1099-4300). This special issue belongs to the section "Complexity".

Deadline for manuscript submissions: closed (31 October 2011)

Special Issue Editor

Guest Editor
Prof. Dr. Paul M. B. Vitányi (Website)

1 CWI (Centrum Wiskunde & Informatica), Science Park 123, 1098 XG, Amsterdam, The Netherlands
2 University of Amsterdam, 1012 WX Amsterdam, The Netherlands
Fax: + 31 20 5924199
Interests: cellular automata; computational complexity; distributed and parallel computing; machine learning and prediction; physics of computation; Kolmogorov complexity; information theory; quantum computing

Special Issue Information

Dear Colleagues,

This Special Issue deals with Kolmogorov complexity in its many facets:

Pointwise Randomness:

Kolmogorov complexity is a modern notion of randomness dealing with the quantity of information in individual objects; that is, pointwise randomness rather than average randomness as produced by a random source. It was proposed by A.N. Kolmogorov in 1965 to quantify the information and randomness of individual objects in an objective and absolute manner. This is impossible by classical probability theory (a branch of measure theory satisfying the so-called Kolmogorov axioms formulated in 1933). Kolmogorov complexity is known variously as "algorithmic information", "algorithmic entropy", "Kolmogorov-Chaitin complexity", "descriptional complexity", "shortest program length", "algorithmic randomness", and others.

Incompressibility Method:

In some parts of the research interests mentioned on Paul Vitanyi's home page a new mathematical proof technique was developed, now known as the `incompressibility method'. The incompressibility method is a basic general technique such as the "pigeon hole" argument, "the counting method" or the "probabilistic method". The new method is based on Kolmogorov complexity.

Absolute Information.

The Kolmogorov complexity of an object is a form of absolute information of the individual object. This is not possible to do by C.E. Shannon's information theory. Unlike Kolmogorov complexity, information theory is only concerned with the average information of a random source.

Universal Induction and Data Compression.

Traditional wisdom has it that the better a theory compresses the learning data concerning some phenomenon under investigation, the better we learn, generalize, and the better the theory predicts unknown data. This belief is vindicated in practice but before the advent of Kolmogorov complexity has not been rigorously proved in a general setting. Making these ideas rigorous involves the length of the shortest effective description of an individual object: its Kolmogorov complexity. Ray Solomonoff invented the notion of universal prediction using the Kolmogorov complexity based universal distribution. Universal prediction is related to optimal effective compression. The latter is almost always a best strategy in hypotheses identification (an ideal form of the minimum description length (MDL) principle invented for real Statistics by J.J. Rissanen). Whereas the single best hypothesis does not necessarily give the best prediction, it can be demonstrated that nonetheless compression is almost always the best strategy in prediction methods in the style of R. Solomonoff.

Prof. Dr. Paul M. B. Vitányi
Guest Editor

Keywords

  • Kolmogorov complexity
  • algorithmic information theory
  • randomness of individual object
  • information in individual object
  • incompressibility method
  • universal induction and data compression
  • similarity by compression

Published Papers (4 papers)

View options order results:
result details:
Displaying articles 1-4
Export citation of selected articles as:

Research

Open AccessArticle A Philosophical Treatise of Universal Induction
Entropy 2011, 13(6), 1076-1136; doi:10.3390/e13061076
Received: 20 April 2011 / Revised: 24 May 2011 / Accepted: 27 May 2011 / Published: 3 June 2011
Cited by 21 | PDF Full-text (425 KB)
Abstract
Understanding inductive reasoning is a problem that has engaged mankind for thousands of years. This problem is relevant to a wide range of fields and is integral to the philosophy of science. It has been tackled by many great minds ranging from [...] Read more.
Understanding inductive reasoning is a problem that has engaged mankind for thousands of years. This problem is relevant to a wide range of fields and is integral to the philosophy of science. It has been tackled by many great minds ranging from philosophers to scientists to mathematicians, and more recently computer scientists. In this article we argue the case for Solomonoff Induction, a formal inductive framework which combines algorithmic information theory with the Bayesian framework. Although it achieves excellent theoretical results and is based on solid philosophical foundations, the requisite technical knowledge necessary for understanding this framework has caused it to remain largely unknown and unappreciated in the wider scientific community. The main contribution of this article is to convey Solomonoff induction and its related concepts in a generally accessible form with the aim of bridging this current technical gap. In the process we examine the major historical contributions that have led to the formulation of Solomonoff Induction as well as criticisms of Solomonoff and induction in general. In particular we examine how Solomonoff induction addresses many issues that have plagued other inductive systems, such as the black ravens paradox and the confirmation problem, and compare this approach with other recent approaches. Full article
(This article belongs to the Special Issue Kolmogorov Complexity)
Figures

Open AccessArticle Algorithmic Relative Complexity
Entropy 2011, 13(4), 902-914; doi:10.3390/e13040902
Received: 3 March 2011 / Revised: 31 March 2011 / Accepted: 1 April 2011 / Published: 19 April 2011
Cited by 5 | PDF Full-text (120 KB) | HTML Full-text | XML Full-text
Abstract
Information content and compression are tightly related concepts that can be addressed through both classical and algorithmic information theories, on the basis of Shannon entropy and Kolmogorov complexity, respectively. The definition of several entities in Kolmogorov’s framework relies upon ideas from classical [...] Read more.
Information content and compression are tightly related concepts that can be addressed through both classical and algorithmic information theories, on the basis of Shannon entropy and Kolmogorov complexity, respectively. The definition of several entities in Kolmogorov’s framework relies upon ideas from classical information theory, and these two approaches share many common traits. In this work, we expand the relations between these two frameworks by introducing algorithmic cross-complexity and relative complexity, counterparts of the cross-entropy and relative entropy (or Kullback-Leibler divergence) found in Shannon’s framework. We define the cross-complexity of an object x with respect to another object y as the amount of computational resources needed to specify x in terms of y, and the complexity of x related to y as the compression power which is lost when adopting such a description for x, compared to the shortest representation of x. Properties of analogous quantities in classical information theory hold for these new concepts. As these notions are incomputable, a suitable approximation based upon data compression is derived to enable the application to real data, yielding a divergence measure applicable to any pair of strings. Example applications are outlined, involving authorship attribution and satellite image classification, as well as a comparison to similar established techniques. Full article
(This article belongs to the Special Issue Kolmogorov Complexity)
Figures

Open AccessArticle Quantum Kolmogorov Complexity and Information-Disturbance Theorem
Entropy 2011, 13(4), 778-789; doi:10.3390/e13040778
Received: 17 January 2011 / Revised: 9 March 2011 / Accepted: 24 March 2011 / Published: 29 March 2011
Cited by 2 | PDF Full-text (150 KB) | HTML Full-text | XML Full-text
Abstract
In this paper, a representation of the information-disturbance theorem based on the quantum Kolmogorov complexity that was defined by P. Vit´anyi has been examined. In the quantum information theory, the information-disturbance relationship, which treats the trade-off relationship between information gain and its [...] Read more.
In this paper, a representation of the information-disturbance theorem based on the quantum Kolmogorov complexity that was defined by P. Vit´anyi has been examined. In the quantum information theory, the information-disturbance relationship, which treats the trade-off relationship between information gain and its caused disturbance, is a fundamental result that is related to Heisenberg’s uncertainty principle. The problem was formulated in a cryptographic setting and the quantitative relationships between complexities have been derived. Full article
(This article belongs to the Special Issue Kolmogorov Complexity)
Open AccessArticle Entropy Measures vs. Kolmogorov Complexity
Entropy 2011, 13(3), 595-611; doi:10.3390/e13030595
Received: 14 January 2011 / Revised: 25 February 2011 / Accepted: 26 February 2011 / Published: 3 March 2011
Cited by 9 | PDF Full-text (172 KB) | HTML Full-text | XML Full-text
Abstract
Kolmogorov complexity and Shannon entropy are conceptually different measures. However, for any recursive probability distribution, the expected value of Kolmogorov complexity equals its Shannon entropy, up to a constant. We study if a similar relationship holds for R´enyi and Tsallis entropies of [...] Read more.
Kolmogorov complexity and Shannon entropy are conceptually different measures. However, for any recursive probability distribution, the expected value of Kolmogorov complexity equals its Shannon entropy, up to a constant. We study if a similar relationship holds for R´enyi and Tsallis entropies of order α, showing that it only holds for α = 1. Regarding a time-bounded analogue relationship, we show that, for some distributions we have a similar result. We prove that, for universal time-bounded distribution mt(x), Tsallis and Rényi entropies converge if and only if α is greater than 1. We also establish the uniform continuity of these entropies. Full article
(This article belongs to the Special Issue Kolmogorov Complexity)

Journal Contact

MDPI AG
Entropy Editorial Office
St. Alban-Anlage 66, 4052 Basel, Switzerland
entropy@mdpi.com
Tel. +41 61 683 77 34
Fax: +41 61 302 89 18
Editorial Board
Contact Details Submit to Entropy
Back to Top