Kolmogorov Complexity - Definition

Definition

To define the Kolmogorov complexity, we must first specify a description language for strings. Such a description language can be based on any computer programming language, such as Lisp, Pascal, or Java Virtual Machine bytecode. If P is a program which outputs a string x, then P is a description of x. The length of the description is just the length of P as a character string, multiplied by the number of bits in a character (e.g. 7 for ASCII).

We could, alternatively, choose an encoding for Turing machines, where an encoding is a function which associates to each Turing Machine M a bitstring <M>. If M is a Turing Machine which, on input w, outputs string x, then the concatenated string <M> w is a description of x. For theoretical analysis, this approach is more suited for constructing detailed formal proofs and is generally preferred in the research literature. In this article, an informal approach is discussed.

Any string s has at least one description, namely the program:

If a description of s, d(s), is of minimal length (i.e. it uses the fewest number of characters), it is called a minimal description of s. Thus, the length of d(s) (i.e. the number of characters in the description) is the Kolmogorov complexity of s, written K(s). Symbolically,

We now consider how the choice of description language affects the value of K, and show that the effect of changing the description language is bounded.

Theorem: If K₁ and K₂ are the complexity functions relative to description languages L₁ and L₂, then there is a constant c – which depends only on the languages L₁ and L₂ chosen – such that

Proof: By symmetry, it suffices to prove that there is some constant c such that for all bitstrings s

Now, suppose there is a program in the language L₁ which acts as an interpreter for L₂:

where p is a program in L₂. The interpreter is characterized by the following property:

Running InterpretLanguage on input p returns the result of running p.

Thus, if P is a program in L₂ which is a minimal description of s, then InterpretLanguage(P) returns the string s. The length of this description of s is the sum of

1. The length of the program InterpretLanguage, which we can take to be the constant c.
2. The length of P which by definition is K₂(s).

This proves the desired upper bound.

See also invariance theorem.