In Reinforcement Learning (RL), the primary goal is to find a policy $\pi_{\theta}$ that maximizes the expected cumulative reward. Standard Policy Gradient methods, such as REINFORCE, often suffer from high variance and instability. The core problem is the 'step size' dilemma: if the gradient update is too small, learning is prohibitively slow; if it is too large, the policy may collapse into a region of parameter space where it can no longer recover, leading to a catastrophic drop in performance. PPO addresses this by ensuring that the updated policy does not deviate too far from the old policy.
To understand PPO, we first define the probability ratio $r_t(\theta)$, which measures the difference between the new policy $\pi_{\theta}$ and the old policy $\pi_{\theta_{old}}$: $$r_t(\theta) = \frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}$$ If $r_t(\theta) > 1$, the action $a_t$ is more likely under the current policy than the old one. If $r_t(\theta) < 1$, it is less likely. This ratio allows us to reuse data collected by the old policy (off-policy learning) while still optimizing the current parameters.
The surrogate objective used in basic policy gradients is $L^{CPI}(\theta) = \mathbb{E}_t [r_t(\theta) \hat{A}_t]$, where $\hat{A}_t$ is the estimated advantage function. The advantage $\hat{A}_t$ tells us whether the action $a_t$ was better or worse than the average action at state $s_t$. While maximizing this objective improves the policy, it encourages the optimizer to push $r_t(\theta)$ to extremes. If $\hat{A}_t$ is positive, the objective increases indefinitely as $r_t(\theta)$ increases, potentially leading to massive, unstable updates that destroy the policy's convergence properties.
To mitigate this, PPO introduces the Clipped Surrogate Objective. The goal is to limit the incentive for the policy to change drastically. The objective function $L^{CLIP}(\theta)$ is defined as: $$L^{CLIP}(\theta) = \mathbb{E}_t [ \min(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_t) ]$$ Here, $\epsilon$ is a hyperparameter (typically $0.1$ or $0.2$). The $\text{clip}$ function restricts the ratio $r_t(\theta)$ to stay within the interval $[1 - \epsilon, 1 + \epsilon]$.
The intuition behind the $\min$ operator is critical. When the advantage $\hat{A}_t$ is positive, the objective increases as $r_t(\theta)$ increases, but it is capped at $1 + \epsilon$. This prevents the policy from becoming 'too greedy' based on a single batch of data. Conversely, when $\hat{A}_t$ is negative, the objective increases as $r_t(\theta)$ decreases, but it is capped at $1 - \epsilon$. This ensures that the policy does not collapse by over-correcting for a bad action.
In practice, PPO is often implemented as an Actor-Critic method. The total loss function combines the clipped surrogate objective, a value function loss to improve state estimation, and an entropy bonus to encourage exploration: $$L^{PPO}(\theta) = \mathbb{E}_t [ L^{CLIP}(\theta) - c_1 (V_{\theta}(s_t) - V_{target})^2 + c_2 S[\pi_{\theta}(s_t)] ]$$ Where $V_{\theta}(s_t)$ is the predicted value, $c_1$ and $c_2$ are coefficients, and $S$ represents the entropy. This tripartite loss ensures that the agent learns a stable policy, an accurate value estimate, and maintains enough randomness to discover new optimal strategies.