The fundamental goal of training a deep network is to minimize a cost function $J(\theta)$, where $\theta$ represents the model parameters. In a perfect world, we would use Gradient Descent (GD), calculating the gradient across the entire dataset. However, for modern datasets with millions of samples, this is computationally prohibitive. Stochastic Gradient Descent (SGD) solves this by approximating the true gradient using a single sample or a small 'mini-batch'. The intuition is that while a single sample's gradient is noisy, the expected value of this noise aligns with the true gradient, allowing the model to converge while significantly reducing the per-iteration computational cost.
Mathematically, for a mini-batch of size $m$, the SGD update rule is defined as: $$\theta_{t+1} = \theta_t - \eta \nabla_{\theta} J(\theta_t; x^{(i:i+m)}, y^{(i:i+m)})$$ where $\eta$ is the learning rate. While SGD is efficient, it suffers from slow convergence in 'ravines'—areas where the surface curves much more steeply in one dimension than another. The updates oscillate across the narrow ravine while making very little progress along the floor toward the optimum, necessitating a careful balance of $\eta$ to avoid divergence or stagnation.
To overcome the limitations of constant learning rates, we introduce Momentum. Instead of relying solely on the current gradient, Momentum accumulates a velocity vector $v$ that smooths out oscillations. This is analogous to a ball rolling down a hill, gaining momentum in directions of consistent descent. The updates are formulated as: $$v_{t+1} = \gamma v_t + \eta \nabla_{\theta} J(\theta_t)$$ $$\theta_{t+1} = \theta_t - v_{t+1}$$ where $\gamma ∈ [0, 1)$ is the momentum coefficient. This approach accelerates convergence in the presence of high curvature and dampens the noise inherent in stochastic sampling.
The Adam (Adaptive Moment Estimation) optimizer evolves this concept by maintaining individual learning rates for every parameter. Adam tracks both the first moment (the mean) and the second moment (the uncentered variance) of the gradients. The first moment $m_t$ acts like momentum, while the second moment $v_t$ scales the update based on the magnitude of recent gradients. This ensures that parameters with frequent, large gradients are updated cautiously, while parameters with sparse gradients receive larger updates to escape plateaus.
The mathematical machinery of Adam involves bias-correction to account for the fact that moment estimates are initialized at zero. The update steps are: $$m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$$ $$v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$$ $$\hat{m}_t = \frac{m_t}{1-\beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1-\beta_2^t}$$ $$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$$ Here, $\beta_1$ and $\beta_2$ are decay rates for the moments, and $\epsilon$ is a small constant to prevent division by zero.
Despite the power of Adam, the choice of the global learning rate $\eta$ remains critical. Learning Rate Scheduling involves adjusting $\eta$ during training to stabilize convergence. Early in training, a high learning rate allows the model to traverse the landscape quickly and escape local minima. As training progresses, decaying the learning rate—via step decay, exponential decay, or cosine annealing—allows the model to 'settle' into a sharper, deeper minimum without overshooting it.
A common sophisticated technique is the 'Warm-up' phase, where $\eta$ starts very small and increases linearly for a few thousand steps before decaying. This prevents the model from diverging early on due to the high variance of initial gradients in deep architectures. The synthesis of an adaptive optimizer like Adam with a well-tuned scheduler often represents the state-of-the-art approach for optimizing massive transformer-based models, balancing the need for rapid exploration with the requirement for precise convergence.