To understand optimization in deep learning, we must first visualize the loss landscape. Imagine a high-dimensional surface where the height represents the error of our model. Our goal is to find the global minimum of this surface. Gradient Descent achieves this by calculating the slope (the gradient) at the current point and taking a small step in the opposite direction. However, calculating the gradient over the entire dataset is computationally prohibitive for large networks. This leads us to Stochastic Gradient Descent (SGD), where we approximate the true gradient using a small, random subset of data called a mini-batch, introducing a 'noise' that can actually help the optimizer escape shallow local minima.
Mathematically, the weight update rule for SGD is defined by the subtraction of the gradient scaled by a learning rate $\eta$. For a weight vector $\theta$, the update at step $t$ is: $$\theta_{t+1} = \theta_t - \eta \nabla J(\theta_t)$$, where $\nabla J(\theta_t)$ is the gradient of the cost function with respect to the parameters. While efficient, vanilla SGD suffers from oscillations in directions of high curvature and slow progress in directions of low curvature, creating a 'zigzagging' effect that delays convergence toward the optimum.
To combat these oscillations, we introduce Momentum. Intuition tells us that if we keep moving in the same direction, we should pick up speed, much like a ball rolling down a hill. Momentum accumulates a moving average of past gradients to smooth out the updates. The velocity $v$ is updated as: $$v_t = \gamma v_{t-1} + \eta \nabla J(\theta_t)$$, and the parameters are updated as $\theta_{t+1} = \theta_t - v_t$. Here, $\gamma$ is the momentum coefficient (typically $0.9$), which ensures that the optimizer maintains a steady trajectory toward the minimum, dampening erratic fluctuations.
Adaptive optimization takes this further by recognizing that not all parameters should be updated with the same learning rate. Adam (Adaptive Moment Estimation) is the industry standard, combining the ideas of Momentum and RMSProp. It maintains an estimate of both the first moment (the mean) and the second raw moment (the uncentered variance) of the gradients. The first moment $m_t$ tracks the direction, while the second moment $v_t$ tracks the scale of the gradients: $$m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$$ and $$v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$$, where $g_t$ is the gradient at time $t$.
Because $m_t$ and $v_t$ start at zero, they are biased toward the origin. Adam applies a bias-correction step to ensure the estimates are accurate during the early stages of training: $$\hat{m}_t = \frac{m_t}{1-\beta_1^t}$$ and $$\hat{v}_t = \frac{v_t}{1-\beta_2^t}$$. Finally, the parameter update is performed as: $\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$. By dividing by the square root of the second moment, Adam effectively shrinks the learning rate for parameters with large, volatile gradients and boosts it for those with sparse, small gradients.
Despite the power of Adam, the global learning rate $\eta$ remains a critical hyperparameter. Learning Rate Scheduling is the practice of adjusting $\eta$ over time. Early in training, a high learning rate allows the model to traverse the landscape quickly. As we approach the minimum, however, a high rate can cause the model to overshoot. Common strategies include 'Step Decay', where the rate is dropped by a factor every $X$ epochs, and 'Cosine Annealing', which follows a cosine curve: $\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})(1 + \cos(\frac{T_{cur}}{T_{max}}\pi))$.
In summary, the choice of optimizer represents a trade-off between convergence speed and generalization. While Adam converges rapidly, some research suggests that fine-tuned SGD with a carefully crafted learning rate schedule may achieve better final generalization on certain benchmarks. The modern practitioner typically starts with Adam for rapid prototyping and may switch to SGD with momentum and a decay schedule for the final production-grade model training to 'polish' the weights into a flatter, more robust minimum.