The information bottleneck method is a technique introduced by Naftali Tishby et al. for finding the best tradeoff between accuracy and complexity (compression) when summarizing (e.g. clustering) a random variable X, given a joint probability distribution between X and an observed relevant variable Y. Other applications include distributional clustering, and dimension reduction. In a well defined sense it generalized the classical notion of minimal sufficient statistics from parametric statistics to arbitrary distributions, not necessarily of exponential form. It does so by relaxing the sufficiency condition to capture some fraction of the mutual information with the relevant variable Y.
The compressed variable is and the algorithm minimises the following quantity
where are the mutual information between and respectively.
Read more about Information Bottleneck Method: Gaussian Information Bottleneck, Defining Decision Contours, Bibliography
Famous quotes containing the words information and/or method:
“We hear a great deal of lamentation these days about writers having all taken themselves to the colleges and universities where they live decorously instead of going out and getting firsthand information about life. The fact is that anybody who has survived his childhood has enough information about life to last him the rest of his days.”
—Flannery O’Connor (1925–1964)
“A method of child-rearing is not—or should not be—a whim, a fashion or a shibboleth. It should derive from an understanding of the developing child, of his physical and mental equipment at any given stage, and, therefore, his readiness at any given stage to adapt, to learn, to regulate his behavior according to parental expectations.”
—Selma H. Fraiberg (20th century)