Optimal with room to vary

Abstract: Normative theories of neural coding, and efficient coding in particular, have been successful across many sensory systems, yet they typically predict a single optimal solution. In practice, neural populations show substantial variability across individuals, time, and stimulus conditions. Reconciling this tension requires understanding not just the optimal code, but how sensitive performance is to perturbations away from that optimum. Directions in coding space where small deviations cause large performance losses should be tightly constrained while directions where large deviations cause little loss can support low-cost variability. This "stiff-sloppy" structure, well known in statistical physics and machine learning, provides a natural framework for asking where theory should agree with data. We test this framework using large-scale recordings of larval tiger salamander retinal ganglion cells responding to naturalistic movies as well as a noise stimulus. We fit low-rank Ising models, selected for their interpretability, and find that population activity is well described by a small number of collective modes tightly coupled to the visual input. The spatial receptive fields of these leading latent variables closely match the principal components of natural images, suggesting there is efficient coding of natural motion at the population level. Agreement with the efficient coding optimum tracks perturbation sensitivity: it is strongest along stiff directions and weaker along sloppy ones.  These same stiff modes form a stable skeleton of interactions across diverse visual environments and support both efficient reconstruction of the current stimulus and accurate prediction of the future.