Abstract: Characterising deep learning in terms of probabilistic inference gives us a theoretical framework for understanding the information content of their internal representations. This Bayesian perspective is made computationally efficient with variational approximations, including the Variational Information Bottleneck (VIB) regulariser for vector-space representations. With attention-based models such as Transformers, attention functions use internal representations which are arbitrarily long sequences of vectors. We can still define variational Bayesian methods for these variable-sized representations using Bayesian nonparametrics, giving us the Nonparametric Variational Information Bottleneck (NVIB) regulariser. We will explain these methods and outline some of the main empirical results for this theoretically motivated approach to deep learning architectures.

Talk in the framework of the course Advanced Topics in NLP

Host: Prof. Lonneke van der Plas