Problems With High Dimensionality
Even if the scale parameter is well-tuned, as the dimensionality of the problem increases, progress can still remain exceedingly slow. To see this, again consider . In one dimension, this corresponds to a Gaussian distribution with mean 0 and variance 1. For one dimension, this distribution has a mean step of zero, however the mean squared step size is given by
As the number of dimensions increases, the expected step size becomes larger and larger. In dimensions, the probability of moving a radial distance is related to the Chi distribution, and is given by
This distribution is peaked at which is for large . This means that the step size will increase as the roughly the square root of the number of dimensions. For the MH algorithm, large steps will almost always land in regions of low probability, and therefore be rejected.
If we now add the scale parameter back in, we find that to retain a reasonable acceptance rate, we must make the transformation . In this situation, the acceptance rate can now be made reasonable, but the exploration of the probability space becomes increasingly slow. To see this, consider a slice along any one dimension of the problem. By making the scale transformation above, the expected step size is any one dimension is not but instead is . As this step size is much smaller than the "true" scale of the probability distribution (assuming that is somehow known a priori, which is the best possible case), the algorithm executes a random walk along every parameter.
Read more about this topic: Multiple-try Metropolis
Famous quotes containing the words problems with, problems and/or high:
“In many ways, life becomes simpler [for young adults]. . . . We are expected to solve only a finite number of problems within a limited range of possible solutions. . . . It’s a mental vacation compared with figuring out who we are, what we believe, what we’re going to do with our talents, how we’re going to solve the social problems of the globe . . .and what the perfect way to raise our children will be.”
—Roger Gould (20th century)
“We have heard all of our lives how, after the Civil War was over, the South went back to straighten itself out and make a living again. It was for many years a voiceless part of the government. The balance of power moved away from it—to the north and the east. The problems of the north and the east became the big problem of the country and nobody paid much attention to the economic unbalance the South had left as its only choice.”
—Lyndon Baines Johnson (1908–1973)
“A novel is a mirror carried along a high road. At one moment it reflects to your vision the azure skies at another the mire of the puddles at your feet. And the man who carries this mirror in his pack will be accused by you of being immoral! His mirror shews [sic] the mire, and you blame the mirror! Rather blame that high road upon which the puddle lies, still more the inspector of roads who allows the water to gather and the puddle to form.”
—Stendhal [Marie Henri Beyle] (1783–1842)