Overcomplete ICA: more sources than sensors

In an overcomplete basis, the number of basis vectors is greater than the dimensionality of the input. The representation of an input is not a unique combination of basis vectors, however, overcomplete representations have greater robustness in the presence of noise, are more sparse, and have greater flexibility in matching structure in the data. Overcomplete codes have also been proposed as a model of some of the response properties of neurons in primary visual cortex. In this paper, we present a learning algorithm for inferring an overcomplete basis by viewing it as probabilistic model of the observed data. We show that overcomplete bases allow for better approximation of the underlying statistical density. Redundancy in the overcomplete representation is removed by using a Laplacian prior on the basis coefficients which leads to representations that are sparse and are a nonlinear function of the data. This work generalizes the technique of independent component analysis and provides a method for the identification of a greater number of sources than inputs. We also demonstrate the advantage of learned representations using natural speech and show that compared to the traditional Fourier basis the learned basis has greater coding efficiency.
Lewicki, M.S. and Sejnowski, T.J. (1997) ``Learning nonlinear overcomplete representations for efficient coding.'' Advances in Neural and Information Processing Systems 10. abstract, compressed postscript .

Michael Lewicki, Bruno Olshausen

Efficient Coding of Natural Images

We apply a general technique for learning overcomplete bases to the problem of finding efficient image codes. The bases learned by the algorithm are localized, oriented, and bandpass, consistent with earlier results obtained using related methods. We show that the learned bases are Gabor-like in structure and that higher degrees of overcompleteness produce greater sampling density in position, orientation, and scale. The efficient coding framework provides a method for comparing different bases objectively by calculating their probability given the observed data or by measuring the entropy of the basis function coefficients. Compared to complete and overcomplete Fourier and wavelet bases, the learned bases have much better coding efficiency. We demonstrate the improvement in the representation of the learned bases by showing superior performance in image denoising and filling-in of missing pixels.
Lewicki, M.S. and Olshausen, B.A. (1998) ``Inferring sparse, overcomplete image codes using an efficient coding framework.'' J. Opt. Soc. Am. A: Optics, Image Science, and Vision (submitted). abstract, compressed postscript .