To understand optimization in deep learning, we must first envision the 'loss landscape'—a high-dimensional surface where the height represents the error of our model. The goal is to find the global minimum of this surface. While Batch Gradient Descent computes the gradient over the entire dataset, it is computationally prohibitive for large data. Stochastic Gradient Descent (SGD) solves this by updating model parameters $\theta$ using only a small batch of samples. This introduces 'noise' into the optimization path, which, counter-intuitively, helps the model escape shallow local minima and saddle points.
Mathematically, the SGD update rule is defined as $\theta_{t+1} = \theta_t - \eta \nabla J(\theta_t)$, where $\eta$ represents the learning rate and $\nabla J(\theta_t)$ is the gradient of the cost function with respect to the parameters. While SGD is efficient, it struggles with 'ravines'—areas where the surface curves much more steeply in one dimension than another. In such cases, SGD oscillates across the narrow ravine without making significant progress along the floor toward the minimum, necessitating a more sophisticated approach to momentum.
To combat oscillation, we introduce Momentum, which accumulates a moving average of past gradients to smooth out updates. The update becomes $v_t = \gamma v_{t-1} + \eta \nabla J(\theta_t)$ and $\theta_{t+1} = \theta_t - v_t$, where $\gamma$ is the momentum coefficient (typically $0.9$). By treating the optimization like a ball rolling down a hill, the velocity $v_t$ builds up in directions of consistent gradient, accelerating convergence and dampening the noise inherent in stochastic batches.
Adam (Adaptive Moment Estimation) takes this further by maintaining individual learning rates for every single parameter. It tracks both the first moment (the mean) and the second moment (the uncentered variance) of the gradients. Specifically, it computes $m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$ and $v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$. These estimates are then bias-corrected to account for their initialization at zero: $\hat{m}_t = \frac{m_t}{1-\beta_1^t}$ and $\hat{v}_t = \frac{v_t}{1-\beta_2^t}$.
The final parameter update in Adam is formulated as $\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$. Here, the term $\sqrt{\hat{v}_t}$ acts as a preconditioner; if a parameter has a consistently large gradient, its effective learning rate is scaled down, and if it has a small gradient, the rate is scaled up. This allows Adam to navigate complex landscapes with minimal manual tuning of the initial learning rate $\eta$, although $\epsilon$ (a tiny constant) is added to prevent division by zero.
Despite the power of Adam, the choice of $\eta$ over time remains critical. Learning Rate Scheduling involves adjusting $\eta$ as training progresses. A common strategy is 'Step Decay,' where the learning rate is reduced by a factor every few epochs, or 'Cosine Annealing,' which follows a curve defined by $\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})(1 + \cos(\frac{t \pi}{T}))$. Scheduling ensures that the model takes large steps early on to explore the space and tiny steps at the end to converge precisely into the narrowest part of the minimum.
In practice, the choice between SGD with Momentum and Adam often depends on the task. While Adam converges faster and is more robust to hyperparameters, SGD often generalizes better to unseen data in computer vision tasks. Many practitioners now use a hybrid approach: starting training with Adam for rapid progress and switching to SGD in the final stages to 'fine-tune' the weights into a sharper, more stable optimum.