Deep learning is fundamentally an optimization problem where we seek to minimize a cost function $J(\theta)$ by iteratively updating the model parameters $\theta$. The core intuition is to imagine the loss landscape as a high-dimensional valley; to find the global minimum, we must move in the direction of the steepest descent. While Batch Gradient Descent computes the gradient over the entire dataset, it is computationally prohibitive for large-scale data. Stochastic Gradient Descent (SGD) solves this by approximating the gradient using a single random sample or a small mini-batch, introducing 'noise' that can actually help the optimizer escape local minima and saddle points.
Mathematically, the update rule for SGD is defined as $\theta_{t+1} = \theta_t - \eta \nabla J(\theta_t; x^{(i)}, y^{(i)})$, where $\eta$ represents the learning rate and $\nabla J$ is the gradient of the cost function with respect to the parameters for a specific sample $(x^{(i)}, y^{(i)})$. The variance introduced by sampling means that the path to the minimum is not a straight line but a stochastic walk. The primary challenge here is the choice of $\eta$: too large, and the system diverges; too small, and convergence becomes excruciatingly slow.
To improve upon SGD, we introduce Momentum, which simulates a ball rolling down a hill by accumulating a velocity vector. Instead of relying solely on the current gradient, momentum keeps a running average of previous gradients to smooth out oscillations in high-curvature directions. The update becomes $v_t = \gamma v_{t-1} + \eta \nabla J(\theta_t)$ and $\theta_{t+1} = \theta_t - v_t$, where $\gamma$ is the momentum coefficient (typically $0.9$). This allows the optimizer to accelerate in directions of consistent descent and dampen erratic movements.
The Adam (Adaptive Moment Estimation) optimizer evolves this concept further by maintaining individual learning rates for every parameter. Adam tracks both the first moment (the mean of gradients) and the second moment (the uncentered variance of gradients). The first moment $m_t$ is calculated as $m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$, and the second moment $v_t$ as $v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$, where $g_t$ is the current gradient. By dividing the update by the square root of the second moment, Adam effectively scales the step size inversely to the magnitude of recent gradients, ensuring that infrequently updated parameters receive larger steps.
To prevent instability during the initial training steps, Adam applies bias correction to the moments: $\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$ and $\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$. The final parameter update is then formulated as $\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$, where $\epsilon$ is a small constant to prevent division by zero. This adaptive mechanism allows Adam to handle sparse gradients and non-stationary objectives more robustly than standard SGD.
Despite the power of Adam, the global learning rate $\eta$ still requires tuning. Learning rate scheduling is the practice of adjusting $\eta$ over time to refine the convergence. Common strategies include 'Step Decay', where the rate is dropped by a factor every few epochs, and 'Cosine Annealing', which follows a cosine curve to smoothly reduce the rate. Mathematically, a decay schedule can be represented as $\eta_t = \eta_0 \\· f(t)$, where $f(t)$ is a monotonically decreasing function. This ensures that the model explores the parameter space aggressively at the start and converges precisely as it approaches the optimum.