Problems With High Dimensionality
Even if the scale parameter is well-tuned, as the dimensionality of the problem increases, progress can still remain exceedingly slow. To see this, again consider . In one dimension, this corresponds to a Gaussian distribution with mean 0 and variance 1. For one dimension, this distribution has a mean step of zero, however the mean squared step size is given by
As the number of dimensions increases, the expected step size becomes larger and larger. In dimensions, the probability of moving a radial distance is related to the Chi distribution, and is given by
This distribution is peaked at which is for large . This means that the step size will increase as the roughly the square root of the number of dimensions. For the MH algorithm, large steps will almost always land in regions of low probability, and therefore be rejected.
If we now add the scale parameter back in, we find that to retain a reasonable acceptance rate, we must make the transformation . In this situation, the acceptance rate can now be made reasonable, but the exploration of the probability space becomes increasingly slow. To see this, consider a slice along any one dimension of the problem. By making the scale transformation above, the expected step size is any one dimension is not but instead is . As this step size is much smaller than the "true" scale of the probability distribution (assuming that is somehow known a priori, which is the best possible case), the algorithm executes a random walk along every parameter.
Read more about this topic: Multiple-try Metropolis
Famous quotes containing the words problems with, problems and/or high:
“She has problems with separation; he has trouble with unity—problems that make themselves felt in our relationships with our children just as they do in our relations with each other. She pulls for connection; he pushes for separateness. She tends to feel shut out; he tends to feel overwhelmed and intruded upon. It’s one of the reasons why she turns so eagerly to children—especially when they’re very young.”
—Lillian Breslow Rubin (20th century)
“It’s so easy during those first few months to think that the problems will never end. You feel as if your son will never sleep through the night, will always spit up food after eating, and will never learn to smile—even though you don’t know any adults or even older children who still act this way.”
—Lawrence Kutner (20th century)
“We do not prove the existence of the poem.
It is something seen and known in lesser poems.
It is the huge, high harmony that sounds
A little and a little, suddenly,
By means of a separate sense. It is and it
Is not and, therefore, is.”
—Wallace Stevens (1879–1955)