ECTS : 3
Description du contenu de l'enseignement :
Optimization is at the heart of most recent advances in machine learning. Indeed, it not only plays a major role in linear regression, SVM and kernel methods, but it is also the key to the recent explosion of deep learning for supervised and unsupervised problems in imaging, vision and natural language processing. This course will review the mathematical foundations, the underlying algorithmic methods and showcase modern applications of a broad range of optimization techniques.
The course consists of several lectures with numerical illustrations in Python. It will begin with the basic components of smooth optimization (optimality conditions, gradient-type methods), then move to methods that are particularly relevant in a machine learning setting such as the celebrated stochastic gradient descent algorithm and its variants. More advanced algorithms related to non-smooth and constrained optimization, that encompass known characteristics of learning problems such as the presence of regularizing terms, will also be described. The various algorithms studied during the lectures will be tested on real and synthetic datasets: these sessions will also address several practical features of optimization codes such as automatic differentiation, and built-in optimization routines within popular machine learning libraries such as PyTorch.
Compétence à acquérir :
- Identify the characteristics of an optimization problem given its formulation.
- Know the theoretical and practical properties of the most popular optimization techniques.
- Find the best optimization algorithm to handle a particular feature of a machine learning problem.
Mode de contrôle des connaissances :
Written exam+Course project.
Bibliographie, lectures recommandées :
L. Bottou, F. E. Curtis, and J. Nocedal. Optimization Methods for Large-Scale Machine Learning, 2018.
J. Wright and Y. Ma, High-Dimensional Data Analysis with Low-Dimensional Models, 2022.
S. J. Wright and B. Recht. Optimization for Data Analysis, 2022.