At its core, training a deep network is an optimization problem where we seek to minimize a loss function $J(\theta)$ by adjusting the model parameters $\theta$. Standard Gradient Descent computes the gradient across the entire dataset, which is computationally prohibitive for large-scale data. Stochastic Gradient Descent (SGD) solves this by estimating the gradient using a single sample or a small 'minibatch'. This introduces noise into the optimization process, which, paradoxically, helps the model escape shallow local minima and find broader, more generalizable regions of the loss landscape.
Mathematically, the SGD update rule is defined as $\theta_{t+1} = \theta_t - \eta \nabla_{\theta} J(\theta_t; x^{(i)}, y^{(i)})$. Here, $\eta$ represents the learning rate, a hyperparameter that controls the step size. While SGD is powerful, it suffers from oscillations in directions of high curvature and slows down in flat regions. This is because a single global learning rate is applied to all parameters regardless of their individual gradients' magnitudes or frequency of updates.
To address the inefficiencies of SGD, we introduce Momentum. Instead of relying solely on the current gradient, Momentum accumulates a moving average of past gradients, acting like a heavy ball rolling down a hill. This dampens oscillations and accelerates convergence. The update is formulated as $v_t = \gamma v_{t-1} + \eta \nabla_{\theta} J(\theta_t)$ and $\theta_{t+1} = \theta_t - v_t$, where $\gamma$ is the momentum coefficient, typically set to 0.9. This essentially filters out noise and emphasizes the consistent directional trend of the gradient.
Adam (Adaptive Moment Estimation) takes this further by maintaining per-parameter learning rates. It tracks both the first moment (mean) and the second moment (uncentered variance) of the gradients. The first moment $m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$ handles acceleration, while the second moment $v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$ scales the updates. By dividing the update by $\sqrt{\hat{v}_t} + \epsilon$, Adam ensures that parameters with frequent, large gradients receive smaller updates, and those with sparse gradients receive larger steps.
The full Adam update involves a bias-correction step to prevent the moments from being skewed toward zero during early iterations. The corrected estimates are $\hat{m}_t = \frac{m_t}{1-\beta_1^t}$ and $\hat{v}_t = \frac{v_t}{1-\beta_2^t}$. The final parameter update is $\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$. This adaptive nature makes Adam the default choice for many architectures, as it requires significantly less manual tuning of the initial learning rate compared to SGD.
Despite the power of adaptive optimizers, the 'Learning Rate Schedule' remains critical. A fixed learning rate $\eta$ is rarely optimal; too high, and the model diverges; too low, and it plateaus. Learning rate scheduling involves decaying $\eta$ over time. Common strategies include 'Step Decay', where the rate drops by a factor every few epochs, and 'Cosine Annealing', which follows a half-cosine curve to smoothly transition from a high rate to nearly zero, allowing the model to converge precisely into the global minimum.
A sophisticated approach is the 'Warm-up' phase, where the learning rate starts very small and increases linearly for a few thousand steps before decaying. This prevents the model from diverging in the early stages of training when the weights are randomly initialized and gradients are volatile. Combining a warm-up period with a Cosine Decay schedule often yields the state-of-the-art results in Large Language Models and Vision Transformers.
In summary, the choice of optimizer represents a trade-off between convergence speed and final generalization. While Adam offers rapid initial progress through adaptive moments, tuned SGD with a carefully crafted learning rate schedule often reaches a slightly superior minimum. Understanding the interplay between the gradient $\nabla J$, the moment $m_t$, and the schedule $\eta_t$ is essential for any practitioner aiming to master the training of deep neural architectures.