The "curse of Dimensionality" As Open Problem
The "curse of dimensionality" is often used as a blanket excuse for not dealing with high-dimensional data. However, the effects are not yet completely understood by the scientific community, and there is ongoing research. On one hand, the notion of intrinsic dimension refers to the fact that any low-dimensional data space can trivially be turned into a higher dimensional space by adding redundant (e.g. duplicate) or randomized dimensions, and in turn many high-dimensional data sets can be reduced to lower dimensional data without significant information loss. This is also reflected by the effectiveness of dimension reduction methods such as principal component analysis in many situations. For distance functions and nearest neighbor search, recent research also showed that data sets that exhibit the curse of dimensionality properties can still be processed unless there are too many irrelevant dimensions, while relevant dimensions can make some problems such as cluster analysis actually easier. Secondly, methods such as Markov chain Monte Carlo or shared nearest neighbor methods often work very well on data that were considered intractable by other methods due to high dimensionality.
Read more about this topic: Curse Of Dimensionality
Famous quotes containing the words curse, open and/or problem:
“In the past, as now, Haiti’s curse has been her politicians. There are still too many men of influence in the country who believe that a national election is a mandate from the people to build themselves a big new house in Petionville and Kenscoff and a trip to Paris.”
—Zora Neale Hurston (1891–1960)
“It’s fairly obvious that American education is a cultural flop. Americans are not a well-educated people culturally, and their vocational education often has to be learned all over again after they leave school and college. On the other hand, they have open quick minds and if their education has little sharp positive value, it has not the stultifying effects of a more rigid training.”
—Raymond Chandler (1888–1959)
“If we parents accept that problems are an essential part of life’s challenges, rather than reacting to every problem as if something has gone wrong with universe that’s supposed to be perfect, we can demonstrate serenity and confidence in problem solving for our kids....By telling them that we know they have a problem and we know they can solve it, we can pass on a realistic attitude as well as empower our children with self-confidence and a sense of their own worth.”
—Barbara Coloroso (20th century)