lin-0188
15.1 The Bellman factorization
Fix a policy \(\pi \) on a state space \(X\). Let \(P^\pi :\mathbb R^X\to \mathbb R^X\) denote its Markov expectation operator, let \(V:\Theta \to \mathbb R^X\), \(\theta \mapsto V_\theta \), be a parameterized value representation, and let
\[ B^\pi V = R^\pi +\gamma P^\pi V \]
the Bellman operator. The GIRL sketch declares two routes from parameter space to the value-function space:
\[ V_\theta \sim R^\pi +\gamma P^\pi V_\theta . \]
The base obstruction at \(s\) is the Bellman residual
\[ \delta _B(\theta ;s) = V_\theta (s) - \bigl(R^\pi (s)+\gamma P^\pi V_\theta (s)\bigr). \]
Ordinary TD scalarizes sampled instances of this obstruction [ Sutton , 1988 ] . GIRL tangent-lifts the same declared relation.