ora-0142

11.3 Exploration as infinitesimal admissibility

For \(\alpha {\gt}0\), write

\[ \Delta _A^\alpha =\{ p\in \Delta _A:p(a)\geq \alpha \text{ for every }a\in A\} . \]

This is not merely an algorithmic restriction. It is a domain on which the repair map admits controlled tangent variation.

Proposition 11.4 Tangent and second-moment control

Along a differentiable curve \(\varepsilon \mapsto (p_\varepsilon ,y_\varepsilon )\) with \(p_0=p\in \Delta _A^\alpha \), the conditional repaired evidence obeys

\[ \dot R_p(a,y)(b)=\mathbf1\{ a=b\} \left(\frac{\dot y}{p(a)}-\frac{y\dot p(a)}{p(a)^2}\right). \]

For \(0\leq y\leq 1\),

\[ \| \dot R_p(a,y)\| _\infty \leq \frac{|\dot y|}{\alpha } +\frac{\| \dot p\| _\infty }{\alpha ^2}. \]

If \(\widehat\ell =R_p(A,\ell (A))\) with \(A\sim p\), then

\[ \mathbb E\| \widehat\ell \| _2^2 =\sum _{a\in A}\frac{\ell (a)^2}{p(a)}, \qquad \operatorname {Var}(\widehat\ell (b)) =\ell (b)^2\! \left(\frac1{p(b)}-1\right). \]
Proof ▼

Differentiate 11.2. The norm bound follows from \(p(a)\geq \alpha \). The repaired vector has one nonzero coordinate, so averaging \(\ell (A)^2/p(A)^2\) gives the second-moment identity. The variance formula follows from the same coordinatewise calculation and 11.2.

Consequently, mixing any policy \(q\) with a full-support reference law

\[ p=(1-\gamma )q+\gamma u, \qquad \gamma {\gt}0, \]

is a tangent-domain repair. If \(\alpha =\gamma \min _a u(a)\), then the action mechanism lies in \(\Delta _A^\alpha \). The usual exploration–variance tradeoff therefore has a LINCS interpretation: exploration controls whether the evidence reconstruction is infinitesimally admissible.

Proposition 11.5 Barycentric tangent reconstruction

Let \(p_\varepsilon \in \Delta _A^\circ \) and \(\ell _\varepsilon \in \mathcal L\) be differentiable curves. Then

\[ \left.\frac{d}{d\varepsilon }\right|_0 \beta _{\mathcal L}D(R_{p_\varepsilon }) \kappa _{p_\varepsilon }(\ell _\varepsilon ) =\dot\ell . \]

Thus variations of the sampling mechanism and of the loss evidence cancel correctly after barycentric reconstruction, even though neither is stable pathwise near the simplex boundary.

Proof ▼

The composite inside the derivative equals \(\ell _\varepsilon \) for every \(\varepsilon \) by 11.2. Differentiating this identity proves the claim. Coordinatewise, the derivative of the sampling weight cancels the derivative of its reciprocal in the importance repair.