The "curse of Dimensionality" As Open Problem
The "curse of dimensionality" is often used as a blanket excuse for not dealing with high-dimensional data. However, the effects are not yet completely understood by the scientific community, and there is ongoing research. On one hand, the notion of intrinsic dimension refers to the fact that any low-dimensional data space can trivially be turned into a higher dimensional space by adding redundant (e.g. duplicate) or randomized dimensions, and in turn many high-dimensional data sets can be reduced to lower dimensional data without significant information loss. This is also reflected by the effectiveness of dimension reduction methods such as principal component analysis in many situations. For distance functions and nearest neighbor search, recent research also showed that data sets that exhibit the curse of dimensionality properties can still be processed unless there are too many irrelevant dimensions, while relevant dimensions can make some problems such as cluster analysis actually easier. Secondly, methods such as Markov chain Monte Carlo or shared nearest neighbor methods often work very well on data that were considered intractable by other methods due to high dimensionality.
Read more about this topic: Curse Of Dimensionality
Famous quotes containing the words curse, open and/or problem:
“Do you still persist in your integrity? Curse God, and die.”
—Bible: Hebrew, Job 2:9.
Job’s wife to Job.
“...there are important considerations in the world beyond plain sewing and teaching dull little boys the alphabet. Any woman who has brains and willing hands finds twenty remunerative occupations open to her where formerly she would have found merely the inevitable two—plain sewing, or the dull little boys. All she had to do is to make her choice and then buckle on her armor of perseverance, while the world applauds.”
—Clara (Marquise)
“From cradle to grave this problem of running order through chaos, direction through space, discipline through freedom, unity through multiplicity, has always been, and must always be, the task of education, as it is the moral of religion, philosophy, science, art, politics and economy; but a boy’s will is his life, and he dies when it is broken, as the colt dies in harness, taking a new nature in becoming tame.”
—Henry Brooks Adams (1838–1918)