In Reinforcement Learning (RL), the primary goal is to optimize a policy $\pi_{\theta}(a|s)$ to maximize the expected cumulative reward. A fundamental challenge in policy gradient methods is the 'step size' problem: if the update to the policy parameters $\theta$ is too large, the policy may move into a region of the parameter space where it performs poorly, leading to a catastrophic drop in performance from which the agent cannot recover. This instability occurs because the gradients are estimated from a finite sample of trajectories, meaning a single noisy update can fundamentally break the agent's behavior.
To solve this, early researchers proposed Trust Region Policy Optimization (TRPO), which ensures that the new policy does not deviate too far from the old policy by constraining the Kullback-Leibler (KL) divergence: $\text{KL}(\pi_{\theta_{ ext{old}}}, \pi_{\theta}) \\≤ \delta$. While mathematically sound, TRPO is computationally expensive because it requires calculating a second-order derivative (the Fisher Information Matrix) and solving a constrained optimization problem via conjugate gradient descent. PPO was designed to achieve the same stability as TRPO but using only first-order gradients, making it significantly easier to implement and scale.
The core of PPO is the probability ratio $r_t(\theta)$, which measures how much the current policy differs from the policy used to collect the data: $r_t(\theta) = \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{ ext{old}}}(a_t|s_t)}$. If $r_t(\theta) > 1$, the action is more likely under the current policy; if $r_t(\theta) < 1$, it is less likely. When we multiply this ratio by the advantage estimate $\hat{A}_t$, we get the surrogate objective $L^{CPI}( heta) = \hat{E}_t [r_t(\theta) \hat{A}_t]$. Maximizing this objective encourages the policy to increase the probability of actions that led to better-than-average outcomes.
However, maximizing $L^{CPI}$ without constraints leads to the instability mentioned earlier. If the advantage $\hat{A}_t$ is large, the gradient will push $r_t(\theta)$ to extremes. PPO introduces the Clipped Surrogate Objective to mitigate this. The objective is defined as $L^{CLIP}( heta) = \hat{E}_t [\min(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t)]$. The $\text{clip}$ function limits the ratio $r_t(\theta)$ to remain within a small interval, typically $[0.8, 1.2]$ when $\epsilon = 0.2$.
The intuition behind the $\min$ operator in the clipped objective is crucial. When the advantage $\hat{A}_t$ is positive, the objective increases as $r_t(\theta)$ increases, but it is capped at $1+\epsilon$, preventing the policy from becoming 'overly confident' in a single update. Conversely, when $\hat{A}_t$ is negative, the objective increases as $r_t(\theta)$ decreases, but the clip kicks in at $1-\epsilon$. This effectively removes the incentive to push the probability of a 'bad' action toward zero in one massive step, ensuring the update remains conservative.
Finally, PPO is typically implemented as 'PPO-Clip' and often includes a value function loss to improve the advantage estimation. The total loss function optimized is $L_t^{PPO} = \hat{E}_t [L^{CLIP}( heta) - c_1 L_t^{VF} + c_2 S[\\pi_{\theta}](s_t)]$, where $L_t^{VF}$ is the mean squared error of the value function and $S$ is an entropy bonus to encourage exploration. By combining the clipped objective with value function regularization, PPO achieves a remarkable balance between sample efficiency, ease of tuning, and training stability.