Information Bottleneck Method - Gaussian Information Bottleneck - Density Estimation

Density Estimation

Since the bottleneck method is framed in probabilistic rather than statistical terms, we first need to estimate the underlying probability density at the sample points . This is a well known problem with a number of solutions described by Silverman in . In the present method, joint probabilities of the samples are found by use of a Markov transition matrix method and this has some mathematical synergy with the bottleneck method itself.

Define an arbitrarily increasing distance metric between all sample pairs and distance matrix . Then compute transition probabilities between sample pairs for some . Treating samples as states, and a normalised version of as a Markov state transition probability matrix, the vector of probabilities of the ‘states’ after steps, conditioned on the initial state, is . We are here interested only in the equilibrium probability vector given, in the usual way, by the dominant eigenvector of matrix which is independent of the initialising vector . This Markov transition method establishes a probability at the sample points which is claimed to be proportional to the probabilities densities there.

Other interpretations of the use of the eigenvalues of distance matrix are discussed in .