In Reinforcement Learning (RL), the primary goal is to find a policy $\pi_{\theta}(a|s)$ that maximizes the expected cumulative reward. A fundamental challenge in policy gradient methods is the 'step size' problem: if the update to the parameters $\theta$ is too large, the policy may move to a region of parameter space where the agent performs poorly, causing a collapse in performance from which it cannot recover. PPO was designed to solve this by ensuring that the new policy does not deviate too far from the old policy, effectively maintaining a 'trust region' without the heavy computational overhead of second-order optimization.
To understand PPO, we first define the probability ratio $r_t(\theta)$ between the current policy and the old policy: $r_t(\theta) = \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{ ext{old}}}(a_t|s_t)}$. When $r_t(\theta) > 1$, the action is more likely under the current policy than the old one; when $r_t(\theta) < 1$, it is less likely. In a standard policy gradient, we maximize the objective $J( heta) = \mathbb{E}_t [r_t(\theta) \hat{A}_t]$, where $\hat{A}_t$ is the estimated advantage function. However, this objective is unconstrained, meaning the gradient can push $r_t(\theta)$ to extremes, leading to the instability mentioned previously.
The core innovation of PPO is the Clipped Surrogate Objective. Instead of simply maximizing the product of the ratio and the advantage, PPO limits the influence of the ratio when it moves too far from 1. The objective function is defined as: $L^{CLIP}( heta) = \mathbb{E}_t [\min(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_t)]$. Here, $\epsilon$ is a hyperparameter (typically 0.1 or 0.2) that defines the clipping range. By taking the minimum of the unclipped and clipped objectives, we ensure that the update is conservative.
Let us examine the mechanics of this clipping. When the advantage $\hat{A}_t$ is positive, the agent wants to increase the probability of the action. The $\min$ operator allows the objective to increase until $r_t(\theta) = 1 + \epsilon$, after which the gradient becomes zero. This prevents the policy from over-optimistically inflating the probability of a single action based on a potentially noisy advantage estimate. Conversely, when $\hat{A}_t$ is negative, the agent wants to decrease the probability; the clipping kicks in at $r_t(\theta) = 1 - \epsilon$, preventing the policy from excessively suppressing an action.
Mathematically, the advantage function $\hat{A}_t$ is often computed using Generalized Advantage Estimation (GAE), which balances bias and variance. The total loss function used during training usually combines the clipped surrogate objective with a value function loss and an entropy bonus to encourage exploration: $L_t^{PPO}( heta) = \mathbb{E}_t [L_t^{CLIP}( heta) - c_1 L_t^{VF}( heta) + c_2 S[\\pi_{\theta}](s_t)]$. Here, $L_t^{VF}$ minimizes the mean squared error of the value function, and $S$ represents the entropy of the policy.
Compared to its predecessor, Trust Region Policy Optimization (TRPO), PPO replaces the complex Kullback-Leibler (KL) divergence constraint with the clipping mechanism. TRPO requires computing the Hessian of the KL divergence and performing conjugate gradient descent, which is computationally expensive. PPO achieves a similar stability guarantee using only first-order gradients, making it significantly easier to implement and more efficient to scale across different architectures, such as deep convolutional networks for visual inputs.