To understand optimization in deep learning, we must first visualize the 'loss landscape'—a high-dimensional surface where the height represents the error of the model. Our goal is to find the global minimum of this surface. The most fundamental tool for this is Gradient Descent, which moves parameters in the direction of the steepest descent. However, calculating the gradient over the entire dataset (Batch Gradient Descent) is computationally prohibitive for large networks. Stochastic Gradient Descent (SGD) solves this by approximating the gradient using a small, random subset of data called a 'mini-batch', introducing a beneficial amount of noise that can help the optimizer escape local minima.
Mathematically, the update rule for SGD is defined by the subtraction of the gradient scaled by a learning rate $\eta$. For a parameter vector $\theta$, the update at step $t$ is: $$\theta_{t+1} = \theta_t - \eta \nabla J(\theta_t)$$ where $\nabla J(\theta_t)$ is the gradient of the cost function $J$ with respect to the parameters. While simple, SGD faces a critical challenge: the 'oscillation' problem. In regions where the surface is steep in one dimension but flat in another, SGD tends to bounce back and forth across the valley rather than progressing smoothly toward the minimum.
To mitigate these oscillations, we introduce Momentum. Momentum simulates a physical ball rolling down a hill, accumulating velocity from previous gradients to smooth out updates. Instead of relying solely on the current gradient, we maintain a velocity vector $v$. The formulation becomes: $$v_{t+1} = \gamma v_t + \eta \nabla J(\theta_t)$$ $$\theta_{t+1} = \theta_t - v_{t+1}$$ Here, $\gamma$ is the momentum coefficient (typically $0.9$). This allows the optimizer to accelerate in directions of consistent gradients and dampen fluctuations in noisy directions.
While momentum helps, a single learning rate $\eta$ for all parameters is rarely optimal. Some parameters may require large updates to escape plateaus, while others require tiny updates to avoid overshooting. Adam (Adaptive Moment Estimation) addresses this by calculating individual learning rates for every parameter. It tracks both the first moment (the mean) and the second moment (the uncentered variance) of the gradients. This allows Adam to effectively scale the gradient based on how frequently and intensely a parameter is being updated.
The Adam algorithm is formulated using two moving averages. The first moment $m_t$ and second moment $v_t$ are updated as: $$m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$$ $$v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$$ To correct for the fact that these moments start at zero, we apply bias correction: $\hat{m}_t = \frac{m_t}{1-\beta_1^t}$ and $\hat{v}_t = \frac{v_t}{1-\beta_2^t}$. The final parameter update is then: $$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$$ where $\epsilon$ is a small constant to prevent division by zero.
Despite the sophistication of Adam, the choice of the initial learning rate $\eta$ remains the most sensitive hyperparameter. Learning Rate Scheduling involves adjusting $\eta$ during training to ensure convergence. Early in training, a high learning rate allows for rapid exploration of the parameter space. As the model approaches a minimum, reducing the learning rate—known as 'decay'—allows the model to settle precisely into the optimum without jumping out of the valley. Common strategies include Step Decay, where $\eta$ is dropped by a factor every few epochs, and Cosine Annealing, which follows a cosine curve to smoothly lower the rate.
To summarize the relationship between these methods: SGD provides the raw mechanism of descent, Momentum adds physical inertia to overcome noise, Adam provides per-parameter intelligence for efficiency, and Scheduling provides a global strategy for convergence. In modern deep learning pipelines, it is common to start with Adam for fast initial convergence and then switch to SGD with a carefully tuned decay schedule to achieve the best possible generalization on the test set.