The primary challenge in Reinforcement Learning (RL) is the instability of policy updates. In vanilla Policy Gradient methods, a single gradient step based on a noisy estimate of the return can push the policy parameters $\theta$ into a region of the parameter space where the agent performs poorly. Because the data used for the next update is collected by this now-degraded policy, the agent can enter a 'collapse spiral' from which it never recovers. The core intuition behind Proximal Policy Optimization (PPO) is to constrain how much the policy can change in a single update, ensuring that the new policy stays 'proximal' to the old one.
To achieve this, PPO utilizes an importance sampling ratio. Let $\pi_{\theta}$ be the current policy we are optimizing and $\pi_{\theta_{old}}$ be the policy used to collect the trajectory data. We define the probability ratio as $r_t(\theta) = \frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}$. If $r_t(\theta) > 1$, the action is more likely under the current policy than the old one; if $r_t(\theta) < 1$, it is less likely. This ratio allows us to reuse data collected by the old policy to estimate the gradient of the new policy, effectively turning a policy gradient method into a form of off-policy learning.
The standard surrogate objective for maximizing the expected return is given by $L^{CPI}(\theta) = \hat{E}_t [r_t(\theta) \hat{A}_t]$, where $\hat{A}_t$ is the advantage estimate. While maximizing this objective improves the policy, it fails if the update is too large; a huge change in $\theta$ can lead to a massive change in $r_t(\theta)$, causing the policy to jump to a suboptimal regime. To prevent this, PPO introduces the Clipped Surrogate Objective, which penalizes the objective if the ratio $r_t(\theta)$ moves too far away from 1.
The mathematical formulation of the clipped objective is: $L^{CLIP}(\theta) = \hat{E}_t [ \min(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_{t}) ]$. Here, $\epsilon$ is a hyperparameter (typically $0.1$ or $0.2$). The $\text{clip}$ function restricts the ratio $r_t(\theta)$ to be within the range $[1 - \epsilon, 1 + \epsilon]$. By taking the minimum of the unclipped and clipped objectives, the algorithm removes the incentive for the policy to move the ratio outside this interval, effectively flattening the gradient once the update exceeds the threshold.
To understand why the $\min$ operator is crucial, consider the case where the advantage $\hat{A}_t$ is positive. The objective increases as $r_t(\theta)$ increases, but the clip stops the gain once $r_t(\theta) \\≥ 1 + \epsilon$. Conversely, if $\hat{A}_t$ is negative, the objective increases as $r_t(\theta)$ decreases, but it is capped once $r_t(\theta) \\≤ 1 - \epsilon$. This mechanism ensures that we do not over-optimistically update the policy based on a single batch of experience, acting as a conservative trust-region method without the computational overhead of calculating the Kullback-Leibler (KL) divergence.
In a full PPO implementation, the total loss function combines the clipped surrogate objective with a value function loss and an entropy bonus to encourage exploration. The final objective is typically written as $L^{PPO}(\theta) = \hat{E}_t [ L^{CLIP}(\theta) - c_1 L^{VF}( heta) + c_2 S ]$, where $L^{VF}$ is the squared error of the value function and $S$ is the entropy of the policy. By balancing these three terms, PPO achieves a remarkable trade-off between sample efficiency, ease of tuning, and stability, making it the default algorithm for many modern RL benchmarks.