Lectures related to Gradient Descent: Optimization and Regularization

Gradient Descent

Covers the concept of gradient descent, a universal algorithm used to find the minimum of a function.

Gradient Descent: Optimization Techniques

Explores gradient descent, loss functions, and optimization techniques in neural network training.

Proximal Gradient Descent: Optimization Techniques in Machine Learning

Discusses proximal gradient descent and its applications in optimizing machine learning algorithms.

Optimization Methods: Theory Discussion

Explores optimization methods, including unconstrained problems, linear programming, and heuristic approaches.

Gradient Descent

Covers the algorithm of gradient descent, aiming to minimize a function by iteratively moving in the direction of the steepest decrease.

Optimization algorithms

Covers optimization algorithms, focusing on Proximal Gradient Descent and its variations.

Quasi-newton optimization

Covers gradient line search methods and optimization techniques with an emphasis on Wolfe conditions and positive definiteness.

Choosing a Step Size

Explores choosing a step size in optimization on manifolds, including backtracking line-search and the Armijo method.

Optimization Techniques: Stochastic Gradient Descent and Beyond

Discusses optimization techniques in machine learning, focusing on stochastic gradient descent and its applications in constrained and non-convex problems.

Descent methods and line search: Finiteness of the line search algorithm

Explores the Wolfe conditions for line search algorithms and proves the finiteness of the line search parameter.

Optimality of Convergence Rates: Accelerated/Stochastic Gradient Descent

Covers the optimality of convergence rates in accelerated and stochastic gradient descent methods for non-convex optimization problems.

Polynomial Regression and Gradient Descent

Covers polynomial regression, gradient descent, overfitting, underfitting, regularization, and feature scaling in optimization algorithms.

Structures in Non-Convex Optimization

Covers non-convex optimization, deep learning training problems, stochastic gradient descent, adaptive methods, and neural network architectures.

Simplex Algorithm: Basics

Introduces the Simplex algorithm for solving flow problems and handling negative cost cycles.

Suspension Optimisation: Performance Analysis and Optimization

Explores the design and optimization of car suspensions for improved performance.

Proximal and Subgradient Descent: Optimization Techniques

Discusses proximal and subgradient descent methods for optimization in machine learning.

Differentiable Ranking and Sorting

Explores differentiable ranking and sorting techniques for machine learning applications.

Understanding Generalization: Implicit Bias & Optimization

Explores the trade-off between complexity and risk in machine learning models, the benefits of overparametrization, and the implicit bias of optimization algorithms.

Matrix Factorization: Optimization and Evaluation

Explores matrix factorization optimization, evaluation methods, and challenges in recommendation systems.

Deep Learning Building Blocks

Covers tensors, loss functions, autograd, and convolutional layers in deep learning.