In the realm of Reinforcement Learning, Policy Gradient methods often suffer from instability because a single large update can collapse the policy's performance, making recovery impossible. The core intuition behind Proximal Policy Optimization (PPO) is to constrain the policy update step size, ensuring that the new policy does not deviate too far from the old one, effectively keeping the optimization within a 'trusted region' without the computational complexity of second-order methods.
To formalize this, we first define the probability ratio $r_t(\theta)$, which represents the likelihood of taking action $a_t$ in state $s_t$ under the new policy $\pi_\theta$ relative to the old policy $\pi_{\theta_{old}}$. Mathematically, this is expressed as $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$. If this ratio deviates significantly from 1, it indicates that the policy has changed drastically, which is precisely the behavior we wish to penalize.
The standard policy gradient objective seeks to maximize the expected advantage, formulated as $L^{PG}(\theta) = \mathbb{E}_t [r_t(\theta) \hat{A}_t]$, where $\hat{A}_t$ is the estimated advantage function. However, maximizing this directly can lead to excessively large updates if the advantage is positive and the ratio grows unchecked. PPO modifies this by introducing a clipped surrogate objective function that limits the influence of $r_t(\theta)$ when it moves outside a specific interval $[1-\epsilon, 1+\epsilon]$.
The clipped surrogate objective is defined as $L^{CLIP}(\theta) = \mathbb{E}_t [\min(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t)]$. Here, the $\text{clip}$ function forces the ratio to stay within the bounds $1-\epsilon$ and $1+\epsilon$. The $\min$ operator ensures that the optimizer only considers the unclipped term if it improves the objective within the safe region; otherwise, it defaults to the clipped value, effectively removing the gradient signal for updates that would push the policy too far.
Consider the case where the advantage $\hat{A}_t$ is positive, meaning the action was better than average. If $r_t(\theta)$ exceeds $1+\epsilon$, the term $\text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t$ becomes constant with respect to $\theta$, resulting in a zero gradient. This mechanism prevents the policy from becoming overly confident in a specific action too quickly, which is a common failure mode in vanilla policy gradients that leads to high variance and instability.
Conversely, if the advantage $\hat{A}_t$ is negative, the algorithm penalizes increasing the probability of that action. If the ratio $r_t(\theta)$ drops below $1-\epsilon$, the clipping again activates to stop the gradient from pushing the probability even lower. This symmetric clipping ensures that the policy update remains conservative regardless of whether the action was good or bad, maintaining a stable learning trajectory over many epochs of optimization on the same batch of data.
The hyperparameter $\epsilon$ controls the width of the trusted region, typically set to values like 0.1 or 0.2. A smaller $\epsilon$ enforces stricter constraints, leading to more stable but potentially slower learning, while a larger $\epsilon$ allows for more aggressive updates at the risk of instability. The elegance of PPO lies in this simple clipping mechanism, which approximates the complex trust region constraints of algorithms like TRPO using only first-order optimization techniques.
In summary, Proximal Policy Optimization achieves state-of-the-art performance by balancing exploration and stability through the clipped surrogate objective. By mathematically bounding the probability ratio $r_t(\theta)$, PPO allows for multiple epochs of minibatch updates without the fear of destructively large policy changes, making it the current standard for deep reinforcement learning applications in robotics, game playing, and control systems.