Publications de Kimon Antonakopoulos

Training Deep Learning Models with Norm-Constrained LMOs

Volkan Cevher, Thomas Michaelsen Pethick, Zhenyu Zhu, Kimon Antonakopoulos, Wanyun Xie

In this work, we study optimization methods that leverage the linear minimization oracle (lmo) over a norm-ball. We propose a new stochastic family of algorithms that uses the lmo to adapt to the geometry of the problem and, perhaps surprisingly, show that ...

2025

Adaptive Bilevel Optimization

Volkan Cevher, Luca Viano, Kimon Antonakopoulos

We propose a new adaptive optimization algorithm based on mirror descent for a class of possibly non-convex smooth bilevel optimization problems. The bilevel optimization template is broadly applicable in machine learning as it features two coupled problem ...

Association for Computing Machinery (ACM)2025

Layer-wise Quantization for Quantized Optimistic Dual Averaging

Dan Alistarh, Volkan Cevher, Kimon Antonakopoulos

Modern deep neural networks exhibit heterogeneity across numerous layers of various types such as residuals, multi-head attention, etc., due to varying structures (dimensions, activation functions, etc.), distinct representation characteristics, which impa ...

2025

On the Generalization of Stochastic Gradient Descent with Momentum

Volkan Cevher, Kimon Antonakopoulos

While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that th ...

Microtome Publishing2024

Universal Gradient Methods for Stochastic Convex Optimization

Volkan Cevher, Ali Kavis, Yongtao Wu, Kimon Antonakopoulos

We develop universal gradient methods for Stochastic Convex Optimization (SCO). Our algorithms automatically adapt not only to the oracle's noise but also to the Hölder smoothness of the objective function without a priori knowledge of the particular setti ...

2024

Improving SAM Requires Rethinking its Optimization Formulation

Volkan Cevher, Fabian Ricardo Latorre Gomez, Thomas Michaelsen Pethick, Kimon Antonakopoulos, Wanyun Xie

This paper rethinks Sharpness-Aware Minimization (SAM), which is originally formulated as a zero-sum game where the weights of a network and a bounded perturbation try to minimize/maximize, respectively, the same differentiable loss. We argue that SAM shou ...

2024

Distributed Extra-Gradient With Optimal Complexity And Communication Guarantees

Volkan Cevher, Igor Krawczuk, Kimon Antonakopoulos

We consider monotone variational inequality (VI) problems in multi-GPU settings where multiple processors/workers/clients have access to local stochastic dual vectors. This setting includes a broad range of important problems from distributed convex minimi ...

2023

UNDERGRAD: A Universal Black-Box Optimization Method with Almost Dimension-Free Convergence Rate Guarantees

Volkan Cevher, Kimon Antonakopoulos

Universal methods for optimization are designed to achieve theoretically optimal convergence rates without any prior knowledge of the problem’s regularity parameters or the accurarcy of the gradient oracle employed by the optimizer. In this regard, existin ...

2022

ADAGRAD Avoids Saddle Points

Kimon Antonakopoulos, Xiao Wang

Adaptive first-order methods in optimization are prominent in machine learning and data science owing to their ability to automatically adapt to the landscape of the function being optimized. However, their convergence guarantees are typically stated in te ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimization

Volkan Cevher, Ali Kavis, Leello Tadesse Dadi, Efstratios Panteleimon Skoulakis, Kimon Antonakopoulos

We propose an adaptive variance-reduction method, called AdaSpider, for minimization of L-smooth, non-convex functions with a finite-sum structure. In essence, AdaSpider combines an AdaGrad-inspired [Duchi et al., 2011, McMahan & Streeter, 2010], but a fai ...

2022

No-regret learning in games with noisy feedback: Faster rates and adaptivity via learning rate separation

Volkan Cevher, Ya-Ping Hsieh, Kimon Antonakopoulos

We examine the problem of regret minimization when the learner is involved in a continuous game with other optimizing agents: in this case, if all players follow a no-regret algorithm, it is possible to achieve significantly lower regret relative to fully ...

2022

Sifting through the Noise: Universal First-Order Methods for Stochastic Variational Inequalities

Volkan Cevher, Ali Kavis, Thomas Michaelsen Pethick, Kimon Antonakopoulos

We examine a flexible algorithmic framework for solving monotone variational inequalities in the presence of randomness and uncertainty. The proposed template encompasses a wide range of popular first-order methods, including dual averaging, dual extrapola ...

2021