Curse of Dimensionality - The "curse of Dimensionality" As Open Problem

The "curse of Dimensionality" As Open Problem

The "curse of dimensionality" is often used as a blanket excuse for not dealing with high-dimensional data. However, the effects are not yet completely understood by the scientific community, and there is ongoing research. On one hand, the notion of intrinsic dimension refers to the fact that any low-dimensional data space can trivially be turned into a higher dimensional space by adding redundant (e.g. duplicate) or randomized dimensions, and in turn many high-dimensional data sets can be reduced to lower dimensional data without significant information loss. This is also reflected by the effectiveness of dimension reduction methods such as principal component analysis in many situations. For distance functions and nearest neighbor search, recent research also showed that data sets that exhibit the curse of dimensionality properties can still be processed unless there are too many irrelevant dimensions, while relevant dimensions can make some problems such as cluster analysis actually easier. Secondly, methods such as Markov chain Monte Carlo or shared nearest neighbor methods often work very well on data that were considered intractable by other methods due to high dimensionality.

Read more about this topic:  Curse Of Dimensionality

Famous quotes containing the words curse, open and/or problem:

    They [twin beds] are the most stupid, the most perfidious, and the most dangerous invention in the world. Shame and a curse on who thought of them.
    HonorĂ© De Balzac (1799–1850)

    There is all the difference in the world between the criminal’s avoiding the public eye and the civil disobedient’s taking the law into his own hands in open defiance. This distinction between an open violation of the law, performed in public, and a clandestine one is so glaringly obvious that it can be neglected only by prejudice or ill will.
    Hannah Arendt (1906–1975)

    The problem of culture is seldom grasped correctly. The goal of a culture is not the greatest possible happiness of a people, nor is it the unhindered development of all their talents; instead, culture shows itself in the correct proportion of these developments. Its aim points beyond earthly happiness: the production of great works is the aim of culture.
    Friedrich Nietzsche (1844–1900)