Lecture

Gradient Descent: Proximal Operator and Step-Size Strategies

Related lectures (29)

Richardson Method: Preconditioned Iterative Solvers

Covers the Richardson method for solving linear systems with preconditioned iterative solvers and introduces the gradient method.

Optimization Methods in Machine Learning

Explores optimization methods in machine learning, emphasizing gradients, costs, and computational efforts for efficient model training.

Linear Systems: Convergence and Methods

Explores linear systems, convergence, and solving methods with a focus on CPU time and memory requirements.

Gradient Descent

Covers the concept of gradient descent in scalar cases, focusing on finding the minimum of a function by iteratively moving in the direction of the negative gradient.

Proximal Gradient Descent: Optimization Techniques in Machine Learning

Discusses proximal gradient descent and its applications in optimizing machine learning algorithms.

Optimization Methods

Covers optimization methods without constraints, including gradient and line search in the quadratic case.

Convergence Analysis: Stochastic Gradient Algorithms

Explores the convergence analysis of stochastic gradient algorithms under various operational modes and step-size sequences.

Optimality of Convergence Rates: Accelerated Gradient Descent

Explores the optimality of convergence rates in convex optimization, focusing on accelerated gradient descent and adaptive methods.

Linear Systems: Iterative Methods

Explores linear systems and iterative methods like gradient descent and conjugate gradient for efficient solutions.

Newton Method: Convergence and Quadratic Care

Covers the Newton method and its convergence properties near the optimal point.

Gradient Descent: Linear Regression

Covers the concept of gradient descent for linear regression, explaining the iterative process of updating parameters.

Uniform Integrability and Convergence

Explores uniform integrability, convergence theorems, and the importance of bounded sequences in understanding the convergence of random variables.

Descent Methods and Newton Method with Line Search

Covers the comparison between descent methods and Newton's method, modifications, line search, and convergence analysis.

Double Descent Curves: Overparametrization

Explores double descent curves and overparametrization in machine learning models, highlighting the risks and benefits.

Coordinate Descent: Optimization Strategies

Explores coordinate descent optimization strategies, emphasizing simplicity in optimization through one-coordinate updates and discussing the implications of different approaches.

Stochastic Optimization: Algorithms and Methods

Explores stochastic optimization algorithms and methods for convex problems with smooth and nonsmooth risks.

Optimization in Machine Learning: Gradient Descent

Covers optimization in machine learning, focusing on gradient descent for linear and logistic regression, stochastic gradient descent, and practical considerations.

Gradient Descent

Covers the algorithm of gradient descent, aiming to minimize a function by iteratively moving in the direction of the steepest decrease.

Optimization Techniques: Gradient Method Overview

Discusses the gradient method for optimization, focusing on its application in machine learning and the conditions for convergence.