In Reinforcement Learning (RL), the primary goal is to optimize a policy $\pi_{\theta}$ to maximize the expected cumulative reward. A recurring challenge in policy gradient methods is the 'step size' problem. If the gradient update is too large, the policy may move into a region of the parameter space where the agent performs poorly, leading to a collapse in performance from which the model cannot recover. This instability occurs because the gradient is only a local approximation; moving too far from the current policy $\pi_{\theta_{old}}$ makes the advantage estimates unreliable.
To solve this, we introduce the concept of the probability ratio $r_t(\theta)$, which measures how much the new policy differs from the old one. It is defined as: $$r_t(\theta) = \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$$ If $r_t(\theta) > 1$, the action is more likely under the current policy; if $r_t(\theta) < 1$, it is less likely. In a standard policy gradient, we maximize $r_t(\theta) \hat{A}_t$, where $\hat{A}_t$ is the estimated advantage. However, without constraints, the optimizer will push $r_t(\theta)$ to extremes to maximize the objective, causing the aforementioned instability.
Proximal Policy Optimization (PPO) addresses this by implementing a 'Clipped Surrogate Objective'. Instead of allowing the ratio $r_t(\theta)$ to grow unbounded, PPO clips the objective function if the ratio moves too far from $1$. The objective is formulated as: $$L^{CLIP}(\theta) = \hat{E}_t \left[ \min(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_t) \right]$$ Here, $\epsilon$ is a hyperparameter (typically $0.1$ or $0.2$) that defines the 'trust region' around the old policy.
The mechanics of the $\min$ operator are crucial. When the advantage $\hat{A}_t$ is positive, the objective encourages increasing the probability of the action, but only up to a factor of $1 + \epsilon$. Once the ratio exceeds this threshold, the gradient becomes zero, preventing the policy from changing too drastically. Conversely, when $\hat{A}_t$ is negative, the objective encourages decreasing the probability, but the clipping kicks in at $1 - \epsilon$, ensuring we don't 'over-correct' and zero out the action probability too aggressively.
Mathematically, this clipping acts as a first-order approximation of the Trust Region Policy Optimization (TRPO) objective. While TRPO uses a hard constraint on the Kullback-Leibler (KL) divergence between policies—requiring complex second-order optimization involving the Fisher Information Matrix—PPO achieves similar stability using only first-order stochastic gradient ascent. This makes PPO significantly easier to implement and more computationally efficient for high-dimensional action spaces.
To complete the learning framework, PPO typically employs an Actor-Critic architecture. The total loss function combines the clipped policy objective, a value function loss to improve advantage estimation, and an entropy bonus to encourage exploration: $$L^{total}(\theta) = \hat{E}_t [ L^{CLIP}(\theta) - c_1 L^{VF}(\theta) + c_2 S[\\pi_{\theta}](s_t) ]$$ In this equation, $L^{VF}$ is usually the mean squared error of the value function, and $S$ represents the entropy of the policy, ensuring the agent does not converge prematurely to a deterministic sub-optimal policy.