Abstract: Modern machine learning optimizes highly nonconvex, high-dimensional losses with exponentially many stationary points, yet simple methods like stochastic gradient descent (SGD) work remarkably well. In many settings, the loss landscape consists of high-dimensional noise and a low-dimensional signal, and SGD’s dynamics can be tracked by a few summary statistics capturing alignment with the signal. I will illustrate this phenomenon using the exactly solvable model of multi-spiked tensor estimation, where the goal is to recover multiple signal vectors on a high-dimensional sphere from noisy Gaussian tensor observations. I will present recent results on the geometry of the associated loss landscape and the dynamics of single-pass SGD, showing how signal recovery emerges despite the apparent complexity of the landscape. This is based partly on joint work with G. Ben Arous and C. Gerbelot.

Host: Prof. Wit Ernst-Jan Camiel