صالح وطن خواه

Gradient clipping is a fundamental tool in Deep Learning, improving the high-probability convergence of stochastic first-order methods like SGD, AdaGrad, an‎d Adam under heavy-tailed noise, which is common in training large language models. It is also a crucial component of Differential Privacy (DP) mechanisms. However, existing high-probability convergence analyses typically require the clipping threshold to increase with the number of optimization steps, which is incompatible with stan‎dard DP mechanisms like the Gaussian mechanism. In this work, we close this gap by providing the first high-probability convergence analysis for DP-Clipped-SGD with a fixed clipping level, applicable to both convex an‎d non-convex smooth optimization under heavy-tailed noise, characterized by a bounded central α-th moment assumption, α ∈ (1, 2]. Our results show that, with a fixed clipping level, the method converges to a neighborhood of the optimal solution with a faster rate than the existing ones. The neighborhood can be balanced against the noise introduced by DP, providing a refined trade-off between convergence speed an‎d privacy guarantees.

محرمانگي تفاضلي , نويز دم سنگين , همگرايي با احتمال بالا , برش گراديان

Differential Privacy , Heavy-tailed noise , High Probability Convergence , Gradient Clipping

Saleh Vatan Khah

Shahrokh Farahmand