Dimension-Independent Convergence Rate for Adagrad with Heavy-Ball Momentum

초록

In this study, we analyze the convergence rate of Adagrad with momentum for non-convex optimization problems. We establish the first dimension-independent convergence rate under the (L0,L1)-smoothness assumption, which is a generalization of the standard L-smoothness. We show the O(1/T) convergence rate under bounded noise in stochastic gradients, where the bound can scale with the current optimality gap and gradient norm.

키워드