The information bottleneck method is a technique introduced by Naftali Tishby et al. for finding the best tradeoff between accuracy and complexity (compression) when summarizing (e.g. clustering) a random variable X, given a joint probability distribution between X and an observed relevant variable Y. Other applications include distributional clustering, and dimension reduction. In a well defined sense it generalized the classical notion of minimal sufficient statistics from parametric statistics to arbitrary distributions, not necessarily of exponential form. It does so by relaxing the sufficiency condition to capture some fraction of the mutual information with the relevant variable Y.
The compressed variable is and the algorithm minimises the following quantity
where are the mutual information between and respectively.
Read more about Information Bottleneck Method: Gaussian Information Bottleneck, Defining Decision Contours, Bibliography
Famous quotes containing the words information and/or method:
“So while it is true that children are exposed to more information and a greater variety of experiences than were children of the past, it does not follow that they automatically become more sophisticated. We always know much more than we understand, and with the torrent of information to which young people are exposed, the gap between knowing and understanding, between experience and learning, has become even greater than it was in the past.”
—David Elkind (20th century)
“You know, I have a method all my own. If you’ll notice, the coat came first, then the tie, then the shirt. Now, according to Hoyle, after that the pants should be next. There’s where I’m different. I go for the shoes next. First the right, then the left. After that, it’s every man for himself.”
—Robert Riskin (1897–1955)