Learning temporal patterns : the example of Echo State Networks.

Abstract. The brain has to learn and produce appropriate temporal patterns of activity in many tasks, a prominent case being the production of movements. A prototypical example of how this could be achieved is provided by Echo-state-networks (ESN). ESNs are known for their remarkable property of producing prescribed autonomous dynamics by learning a simple feedback to a large recurrent random network. They serve as useful conceptual models for movement production and for the role of the thalamo-cortical loop. However, the principles that underly ESN success have remained incompletely understood. I will describe our recent work with Alain Karma (Northeastern U) that shed light on this question by focusing on the regime where the recurrent neural network evolution is stable and the feedback is weak. The network dynamics are then controlled by a finite number of modes, the nonlinear interactions of which, can be described by normal forms. Function approximation in the considered regime can be precisely understood as a Fourier decomposition with nonlinearly determined amplitudes. The description is found to extend to feedbacks of moderate amplitude and to recurrent neural networks with several unstable modes. We expect the theory to be applicable to various generalizations of the simplest setting of randomly connected firing rate units of relevance to neural dynamics, with, for instance, more complex dynamical units, or more structured networks with different unit classes.