In the realm of Reinforcement Learning (RL), an agent learns by interacting with an environment to maximize cumulative reward. However, a naive approach where the policy changes drastically based on a single batch of experience often leads to instability; the agent might unlearn good behaviors or converge to poor local optima. To mitigate this, we constrain how much the new policy can differ from the old one, ensuring updates remain within a 'trust region.'
The mathematical tool we use to measure this difference is the Kullback–Leibler (KL) divergence. Unlike standard distance metrics, KL divergence is not symmetric and does not satisfy the triangle inequality, making it a divergence rather than a true metric. It quantifies the information lost when one probability distribution, $q$, is used to approximate another, $p$.
Formally, for discrete probability distributions $P$ and $Q$ defined on the same probability space, the KL divergence from $Q$ to $P$ is defined as: $$D_{KL}(P || Q) = \sum_{x} P(x) \log \left( \frac{P(x)}{Q(x)} \right)$$. In the context of continuous policies parameterized by $\theta$, we replace the sum with an expectation over the state-action space: $$D_{KL}(\pi_{\theta} || \pi_{\theta_{old}}) = \mathbb{E}_{x \sim \pi_{\theta_{old}}} \left[ \log \frac{\pi_{\theta}(x)}{\pi_{\theta_{old}}(x)} \right]$$.
Why do we care about this specific quantity in policy optimization? When we update our policy parameters from $\theta_{old}$ to $\theta$, we are essentially changing the probability of taking specific actions. If the ratio $\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{old}}(a|s)}$ becomes too large or too small, our estimates of the advantage function become biased because the data was collected under the old policy. The KL divergence penalizes these extreme shifts.
This concept is central to algorithms like Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). In TRPO, the KL divergence is used as a hard constraint in the optimization problem: we maximize the expected surrogate objective subject to $D_{KL}(\pi_{\theta} || \pi_{\theta_{old}}) \le \delta$. This ensures that every step taken in the parameter space results in a policy that is statistically close to the previous one.
Intuitively, you can think of the KL divergence as a 'friction' term that prevents the policy from sliding too far too fast. Without it, a single high-reward outlier could cause the policy to collapse into a deterministic strategy that ignores other potentially valuable actions. By keeping the divergence low, we maintain enough entropy in the policy to continue exploring the environment safely.
In practice, calculating the exact KL divergence can be computationally expensive, so many modern implementations use a first-order or second-order approximation. For small updates, the KL divergence behaves locally like the squared Euclidean distance weighted by the Fisher Information Matrix. This connection allows us to use natural gradient descent, which accounts for the curvature of the policy space.
Ultimately, the KL divergence serves as the guardian of stability in deep reinforcement learning. It transforms the chaotic process of trial-and-error learning into a disciplined optimization procedure. By rigorously bounding the information change between policy iterations, we enable agents to learn complex tasks without forgetting what they have already mastered.