lin-0191

15.4 Why the advantage quotient is necessary

Absolute action values can vary smoothly while their ranking is wrong. For policy learning, GIRL quotients the action-common direction:

\begin{align*} A_\theta (s,a) & = Q_\theta (s,a) - \sum _b\pi (b\mid s)Q_\theta (s,b),\\ C_\theta (s;a,a’) & = Q_\theta (s,a)-Q_\theta (s,a’). \end{align*}

Adding the same \(c(s)\) to every action leaves \(C_\theta \) unchanged. The tangent contrast regularizer

\[ \mathcal R_A(\theta ) = \mathbb E_{s,a,a'} \left[ \lVert D_sC_\theta (s;a,a')\rVert ^2 \right] \]

therefore descends to the decision-relevant quotient.

Design principle

Probe the information used by the decision rule. Smooth absolute values are not enough when only action contrasts determine the policy.