Optimization for Machine Learning

ECTS : 4

Volume horaire : 24

Description du contenu de l'enseignement :

Optimization has long been a fundamental component for modeling and  solving classical machine learning problems such as linear regression  and SVM classification. It also plays a key role in the training of  neural networks, thanks to the development of efficient numerical tools  tailored to deep learning. This course is concerned with developing optimization algorithms for  learning tasks, and will consist of both lectures and hands-on sessions  in Python. The course will begin by an introduction to the various  problem formulations arising in machine and deep learning, together with  a refresher on key mathematical concepts (linear algebra, convexity,  smoothness). The course will then describe the main algorithms for  optimization in data science (gradient descent, stochastic gradient) and  their theoretical properties. Finally, the course will focus on the  challenges posed by implementing these methods in a deep learning and  large-scale environment (automatic differentiation, distributed  calculations, regularization).

Compétence à acquérir :

• Understand the nature and structure of optimization problems arising in machine learning.
• Select an algorithm tailored to solving a particular instance among those seen in class based on theoretical and practical concerns.
• Experience the practical challenges in implementing an optimization scheme in a learning setting.

Bibliographie, lectures recommandées :

• L. Bottou, F. E. Curtis and J. Nocedal. Optimization Methods for Large-Scale Machine Learning. SIAM Review, 2018.
• S. J. Wright and B. Recht. Optimization for Data Analysis. Cambridge University Press, 2022.