At its core, optimizing a deep network is an exercise in finding the global minimum of a high-dimensional, non-convex cost function $J(\theta)$. The most fundamental approach is Stochastic Gradient Descent (SGD). Unlike batch gradient descent, which computes the gradient over the entire dataset, SGD approximates the gradient using a single sample or a small mini-batch. This introduces 'noise' into the optimization path. While this noise might seem detrimental, it often helps the optimizer escape shallow local minima and saddle points, allowing it to explore the loss landscape more effectively.
Mathematically, the SGD update rule is defined as follows. For a learning rate $\\eta$, the parameters $ heta$ are updated in the opposite direction of the gradient of the cost function: $$\theta_{t+1} = \theta_t - \eta \nabla_{\theta} J(\theta_t)$$ Here, $\nabla_{\theta} J(\theta_t)$ represents the partial derivatives of the loss with respect to the weights. The primary challenge with vanilla SGD is the choice of $\eta$; too large, and the model diverges; too small, and convergence becomes prohibitively slow, especially in regions where the curvature of the loss surface is uneven.
To overcome the limitations of a fixed learning rate, we introduce Momentum. Momentum simulates a physical ball rolling down a hill, accumulating 'velocity' from previous gradients to smooth out oscillations in narrow valleys. This prevents the optimizer from zig-zagging across a ravine and instead accelerates it toward the minimum. The update incorporates a velocity vector $v_t$, calculated as: $$v_t = \gamma v_{t-1} + \eta \nabla_{\theta} J(\theta_t)$$ followed by the update $$\theta_{t+1} = \theta_t - v_t$$, where $\gamma$ is the momentum coefficient, typically set around $0.9$.
Adam (Adaptive Moment Estimation) represents a paradigm shift by providing an individual learning rate for every single parameter in the network. Adam maintains an exponential moving average of both the gradients (the first moment, $m_t$) and the squared gradients (the second moment, $v_t$). The first moment tracks the mean direction (momentum), while the second moment tracks the uncentered variance, allowing the algorithm to dampen updates for parameters with high-frequency volatility and amplify updates for those with sparse gradients.
The Adam update process is mathematically rigorous. First, we compute the biased estimates: $$m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$$ and $$v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$$. To counteract the fact that these moments start at zero, we apply bias correction: $\hat{m}_t = \frac{m_t}{1-\beta_1^t}$ and $\hat{v}_t = \frac{v_t}{1-\beta_2^t}$. Finally, the parameters are updated via: $$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$$, where $\epsilon$ is a small constant to prevent division by zero.
Despite the sophistication of Adam, the global learning rate $\eta$ still requires tuning. Learning rate scheduling is the practice of adjusting $\eta$ during training to ensure stability and precision. A common strategy is 'Step Decay', where the learning rate is multiplied by a factor (e.g., $0.1$) every $N$ epochs. Another approach is 'Cosine Annealing', which follows a cosine curve, starting high to explore the landscape and ending very low to settle deeply into a local optimum: $\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})(1 + \cos(\frac{T_{cur}}{T_{max}}\pi))$.
In summary, the progression from SGD to Adam and the integration of scheduling reflects the need to balance exploration and exploitation. SGD provides a stochastic baseline, Momentum adds directional persistence, and Adam introduces per-parameter sensitivity. When combined with a strategic decay schedule, these tools allow us to train networks with millions of parameters, ensuring that we don't just find a minimum, but a robust and generalizable one.