The foundation of training deep neural networks lies in minimizing a loss function $J(\theta)$, where $\theta$ represents the model parameters. While Batch Gradient Descent computes the gradient using the entire dataset, it is often computationally prohibitive and prone to getting stuck in saddle points. The core intuition behind Stochastic Gradient Descent (SGD) is to approximate the true gradient using a single data point or a small mini-batch, introducing noise that helps the optimizer escape local minima and traverse the error surface more efficiently.
Mathematically, standard SGD updates the parameters $\theta$ at time step $t$ using the rule $\theta_{t+1} = \theta_t - \eta \nabla_\theta J(\theta_t; x^{(i)}, y^{(i)})$, where $\eta$ is the learning rate and $(x^{(i)}, y^{(i)})$ is a specific training example. This stochasticity means the path to convergence is not a straight line down the steepest descent but rather a zig-zag trajectory. While simple, this method requires careful tuning of $\eta$; if it is too large, the algorithm may diverge, and if too small, convergence becomes impractically slow.
To address the limitations of fixed learning rates and uniform updates for all parameters, adaptive methods like Adam (Adaptive Moment Estimation) were developed. The intuition behind Adam is to maintain an exponentially decaying average of past gradients (first moment) and past squared gradients (second moment). This allows the algorithm to adapt the learning rate for each parameter individually, effectively taking larger steps for sparse features and smaller steps for frequent ones, while also incorporating momentum to smooth out the oscillations inherent in SGD.
The mathematical formulation of Adam involves computing biased first and second moment estimates, $m_t$ and $v_t$, followed by bias correction. Specifically, $m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$ and $v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$, where $g_t$ is the gradient at step $t$. The parameters are then updated via $\theta_{t+1} = \theta_t - \eta \\· \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$, where $\hat{m}_t$ and $\hat{v}_t$ are the bias-corrected estimates and $\epsilon$ is a small constant for numerical stability. This formulation combines the benefits of Momentum and RMSProp into a single robust optimizer.
Even with advanced optimizers like Adam, the choice of the initial learning rate $\eta$ remains critical, and a static value is often suboptimal throughout the entire training process. Learning rate scheduling involves dynamically adjusting $\eta$ during training based on the epoch number or validation performance. The intuition is to start with a larger learning rate to make rapid progress in the early stages when the weights are far from optimal, and then gradually reduce it to allow the model to settle into a sharp or flat minimum without overshooting.
Common scheduling strategies include Step Decay, where the learning rate is dropped by a factor every few epochs, and Exponential Decay, defined as $\eta_t = \eta_0 e^{-kt}$. Perhaps the most effective modern approach is the Cosine Annealing schedule, where the learning rate follows a cosine curve: $\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})(1 + \cos(\frac{t}{T}\pi))$. This smooth reduction prevents the abrupt changes in gradient magnitude that can destabilize training, allowing for a more refined convergence near the end of the optimization landscape.
In practice, the interplay between the optimizer choice and the learning rate schedule defines the success of deep network training. While Adam is often the default choice due to its robustness to hyperparameter settings, recent research suggests that SGD with momentum and a carefully tuned learning rate schedule can sometimes generalize better on specific computer vision tasks. Understanding the mathematical underpinnings of both the update rules and the scheduling functions allows practitioners to diagnose training failures, such as vanishing gradients or oscillating loss, and adjust their strategy accordingly.
Ultimately, mastering these optimization techniques requires viewing them not as black-box tools but as mechanisms controlling the trajectory through high-dimensional non-convex spaces. Whether utilizing the adaptive scaling of Adam or the precise control of SGD with cosine annealing, the goal remains the same: to efficiently navigate the loss landscape to find parameters $\theta^*$ that minimize generalization error. As networks grow deeper and more complex, these foundational algorithms continue to evolve, yet their core principles of gradient exploitation and moment estimation remain central to the field of machine learning.