حميدرضا جعفريان

In recent years, modern machine learning—particularly deep learning—has increasingly relied on highly overparameterized models where the number of parameters significantly exceeds the number of training samples. This thesis investigates the fast convergence of Stochastic Gradient Descent (SGD) in such regimes. Unlike classical theoretical analyses that require small learning rates for convergence, this research introduces conditions that enable SGD to use large step sizes while achieving a fast linear convergence rate comparable to deterministic Gradient Descent (GD). Two main regularity conditions are proposed: Quadratic Growth an‎d the Aiming Condition. It is proven that these conditions not only suffice locally, but also hold during the training of sufficiently wide feedforward neural networks with linear output layers. Under these assumptions, SGD iterates converge to the set of global minimizers with high probability, exhibiting a convergence rate of O(exp(-t/k)) that aligns well with empirical observations. In the final part, a boosting strategy is presented to raise the algorithm’s success probability from a constant level to arbitrarily high levels using techniques such as small ball assumptions an‎d rejection sampling.

يادگيري ماشين , شبكه هاي عصبي عميق , بيش پارامتري , گراديان تصادفي , نرخ همگرايي خطي , شرط هدف گيري , نمونه گيري رد شونده

Machine Learning , Deep Neural Networks , Overparameterization , Stochastic Gradient Descent , Linear Convergence Rate , Condition, Aiming Condition , Rejection Sampling , Polyak-Łojasiewicz (PŁ) Condition

Hamidreza Jafarian

Dr, Javad Vahidi