At its core, Reinforcement Learning (RL) seeks to optimize a policy $\pi_{\theta}$ to maximize the expected cumulative reward. However, a fundamental challenge in policy gradient methods is the 'step size' problem. If we update the policy parameters $\theta$ too aggressively, we may move into a region of parameter space where the policy collapses, leading to a catastrophic drop in performance from which the agent cannot recover. The central intuition of Proximal Policy Optimization (PPO) is to ensure that the new policy does not deviate too far from the old policy, effectively keeping the update 'proximal' to the current stable behavior.
To understand PPO, we must first look at the probability ratio between the new policy and the old policy. Let $r_t(\theta)$ be defined as the ratio of the probability of taking action $a_t$ given state $s_t$ under the new parameters $\theta$, compared to the old parameters $\theta_{old}$: $$r_t(\theta) = \frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}$$ If $r_t(\theta) > 1$, the action is more likely under the current policy; if $r_t(\theta) < 1$, it is less likely. This ratio allows us to express the objective function in a way that is independent of the original sampling distribution, facilitating multiple epochs of updates on the same batch of data.
In a standard policy gradient approach, we maximize the surrogate objective $L^{CPI}( heta) = \hat{E}_t [r_t(\theta) \hat{A}_t]$, where $\hat{A}_t$ is the estimated advantage function. The advantage $\hat{A}_t$ tells us whether an action was better or worse than the average action in that state. While maximizing this objective improves the policy, it provides no mechanism to prevent $r_t(\theta)$ from growing excessively large. If the gradient update is too large, the policy can shift drastically, leading to high variance and instability in the learning trajectory.
PPO solves this by introducing the 'Clipped Surrogate Objective'. Instead of blindly maximizing the product of the ratio and the advantage, PPO clips the ratio $r_t(\theta)$ within a range defined by a hyperparameter $\epsilon$ (typically $0.1$ or $0.2$). The objective function is formulated as: $$L^{CLIP}( heta) = \hat{E}_t [ \min(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t) ]$$ This formulation ensures that the objective only increases if the action was beneficial and the update is modest. If the ratio moves outside the interval $[1-\epsilon, 1+\epsilon]$, the gradient becomes zero, effectively 'flattening' the objective and preventing further updates in that direction.
The beauty of this clipping mechanism becomes apparent when we analyze the two possible cases for the advantage $\hat{A}_t$. When $\hat{A}_t > 0$, the action was better than average, and we want to increase $r_t(\theta)$; however, the clip prevents it from exceeding $1+\epsilon$. Conversely, when $\hat{A}_t < 0$, the action was worse than average, and we want to decrease $r_t(\theta)$; the clip prevents it from falling below $1-\epsilon$. By taking the minimum of the clipped and unclipped objectives, PPO creates a lower bound (a pessimistic estimate) of the true improvement, ensuring that we do not over-optimistically shift the policy based on a single noisy batch of samples.
In practical implementation, PPO is often used as an Actor-Critic method. The final loss function combines the clipped surrogate objective $L^{CLIP}$, a value function loss $L^{VF}$ to improve state-value estimation, and an entropy bonus $S$ to encourage exploration: $$L^{PPO}( heta) = \hat{E}_t [ L^{CLIP}( heta) - c_1 L^{VF}( heta) + c_2 S[\\pi_{\theta}(s_t)] ]$$ where $c_1$ and $c_2$ are coefficients. This comprehensive loss ensures that the agent learns a robust value function, maintains enough curiosity to explore the environment, and updates its policy in a stable, controlled manner.