At its core, the challenge of Policy Gradient methods is the 'step size' problem. In standard gradient ascent, a single large update to the policy parameters $\theta$ can lead to a catastrophic drop in performance. Because the data used for the update is collected by the old policy, once the policy changes significantly, the old data becomes misleading. PPO seeks to solve this by ensuring that the new policy does not deviate too far from the old policy, maintaining a 'trust region' where the gradient estimate remains reliable.
To understand PPO, we first define the probability ratio $r_t(\theta)$. This ratio compares the probability of taking an action $a_t$ under the current parameters $\theta$ versus the parameters $\theta_{old}$ that were used to collect the trajectory: $r_t(\theta) = \frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}$. If $r_t > 1$, the action is more likely under the new policy; if $r_t < 1$, it is less likely. This ratio allows us to use importance sampling to estimate the policy gradient using data from the old policy.
The standard surrogate objective is $L^{CPI}( heta) = \hat{E}_t [r_t(\theta) \hat{A}_t]$, where $\hat{A}_t$ is the advantage estimate. The advantage $\hat{A}_t$ tells us if the action was better or worse than the average action in that state. However, maximizing this objective without constraints encourages the ratio $r_t(\theta)$ to grow indefinitely for positive advantages, leading to massive policy shifts and training instability.
PPO introduces the clipped surrogate objective to prevent such volatility. The objective is defined as $L^{CLIP}( heta) = \hat{E}_t [\min(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_t)]$. Here, the $\text{clip}$ function restricts the ratio $r_t(\theta)$ to a narrow window around 1, typically $\epsilon = 0.2$. By taking the minimum of the unclipped and clipped objectives, the algorithm removes the incentive for the policy to move too far away from the old policy.
The mechanism of the CLIP function is nuanced: when the advantage $\hat{A}_t$ is positive, the objective is capped at $1 + \epsilon$, preventing the policy from over-optimistically increasing the action probability. Conversely, when $\hat{A}_t$ is negative, the objective is capped at $1 - \epsilon$, ensuring the policy isn't pushed too aggressively away from an action. This effectively flattens the gradient once the update reaches a certain threshold, creating a soft constraint on the policy change.
In practice, PPO is often implemented as a joint objective that includes a value function loss and an entropy bonus to encourage exploration. The total loss function is $\mathcal{L} = \hat{E}_t [L^{CLIP}( heta) - c_1 L^{VF}( heta) + c_2 S[\\pi_{\theta}](s_t)]$. The value function loss $L^{VF}$ ensures the agent accurately predicts returns, while the entropy term $S$ prevents the policy from collapsing into a single deterministic action too early in training.
The brilliance of PPO lies in its balance between ease of implementation and theoretical grounding. Unlike Trust Region Policy Optimization (TRPO), which requires computing the second-order Hessian matrix and performing conjugate gradient descent, PPO achieves similar stability using only first-order stochastic gradient descent. This makes it the default choice for many industrial RL applications, including the fine-tuning of Large Language Models via RLHF.