At its core, training a deep network is a non-convex optimization problem where the goal is to find a set of parameters $\theta$ that minimizes a cost function $J(\theta)$. The intuition behind Gradient Descent is to imagine the cost function as a landscape of hills and valleys; to find the lowest point, we must move in the opposite direction of the steepest ascent. In Stochastic Gradient Descent (SGD), instead of calculating the gradient over the entire dataset—which is computationally prohibitive—we compute the gradient using a single randomly chosen sample or a small mini-batch, introducing a 'stochastic' noise that can actually help the model escape shallow local minima.
Mathematically, the update rule for SGD is expressed as: $$\theta_{t+1} = \theta_t - \eta \nabla_{\theta} J(\theta_t; x^{(i)}, y^{(i)})$$ where $\eta$ is the learning rate and $\nabla_{\theta} J$ represents the gradient of the loss with respect to the parameters. While conceptually simple, SGD suffers from two primary issues: oscillations in steep directions and slow progress in flat regions (plateaus). If $\eta$ is too high, the model diverges; if it is too low, convergence takes an eternity.
To address these instabilities, Adam (Adaptive Moment Estimation) introduces the concept of adaptive learning rates for each individual parameter. Adam maintains an exponential moving average of both the gradients (the first moment) and the squared gradients (the second moment). The first moment $m_t$ acts as momentum, smoothing out oscillations, while the second moment $v_t$ scales the learning rate inversely to the magnitude of past gradients, effectively slowing down updates for frequently changing parameters and speeding them up for sparse ones.
The mathematical formulation of Adam begins with the updates for the moments: $$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$ and $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$ where $g_t = \nabla_{\theta} J(\theta_t)$. To correct for the fact that $m_t$ and $v_t$ are initialized at zero, we apply bias correction: $\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$ and $\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$. The final parameter update is then: $$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$$, where $\epsilon$ is a small constant to prevent division by zero.
Despite the power of Adam, the global learning rate $\eta$ still plays a critical role in the final generalization performance. Learning rate scheduling involves systematically adjusting $\eta$ during training. The intuition is to start with a large learning rate to quickly traverse the loss landscape and later decay it to 'settle' into the global minimum. Common strategies include Step Decay, where $\eta$ drops by a factor every $N$ epochs, and Cosine Annealing, which follows a cosine curve to smoothly reduce the rate.
A rigorous perspective on scheduling often involves the concept of a learning rate schedule $\eta_t$. For example, in an exponential decay schedule, we define $\eta_t = \eta_0 e^{-kt}$. In more advanced 'warm-up' strategies, weights are initialized with a very small $\eta$ and gradually increased to a peak value before decaying. This prevents the gradients from exploding during the very first few iterations when the weights are randomly initialized and the loss is high.
Comparison between SGD and Adam reveals a fundamental trade-off: Adam typically converges much faster and requires less manual tuning of the initial learning rate, but SGD (with momentum) often achieves superior generalization on the test set. This is because Adam's adaptive scaling can cause it to converge to sharp minima, whereas SGD's noise helps it find flatter, more robust minima. Consequently, many state-of-the-art practitioners use Adam for the early phase of training and switch to SGD for the final fine-tuning.
In summary, the journey from SGD to Adam and the integration of scheduling represents an evolution from global, static updates to local, dynamic adjustments. By balancing the momentum of the first moment, the scaling of the second moment, and a carefully timed decay of the learning rate, we can effectively navigate the high-dimensional loss surfaces of modern deep networks.