Winner-take-all - Neural Networks

Neural Networks

In the theory of artificial neural networks winner-take-all networks are a case of competitive learning in recurrent neural networks. Output nodes in the network mutually inhibit each other, while simultaneously activating themselves through reflexive connections. After some time, only one node in the output layer will be active, namely the one corresponding to the strongest input. Thus the network uses nonlinear inhibition to pick out the largest of a set of inputs. Winner-take-all is a general computational primitive that can be implemented using different types of neural network models, including both continuous-time and spiking networks (Grossberg, 1973; Oster et al. 2009).

Winner-take-all networks are commonly used in computational models of the brain, particularly for distributed decision-making in the cortex. Important examples include hierarchical models of vision (Riesenhuber et al. 1999), and models of selective attention and recognition (Carpenter and Grossberg, 1987; Itti et al. 1998). They are also common in artificial neural networks and neuromorphic analog VLSI circuits. It has been formally proven that the winner-take-all operation is computationally powerful compared to other nonlinear operations, such as thresholding (Maass 2000).

In many practical cases, there is not only a single neuron which becomes the only active one but there are exactly k neurons which become active for a fixed number k. This principle is referred to as k-winners-take-all.