lin-0201

16.1 The scalar-reward declaration

Fix a prompt or context \(x\). Let \(G_x=(V_x,E_x)\) be a connected comparison graph whose vertices are responses. Choose an orientation and let \(B_x\in \mathbb R^{|V_x|\times |E_x|}\) be its signed incidence matrix. If \(H_x(i,j)\) is the probability that response \(i\) is preferred to response \(j\), define the edge log odds

\[ \ell _x(i,j)=\operatorname {logit}H_x(i,j). \]

A Bradley–Terry reward \(r_x:V_x\to \mathbb R\) declares

\[ \ell _x=B_x^\top r_x. \]

This is the representational assumption shared by explicit reward modeling and preference objectives built from reward differences [ Bradley and Terry , 1952 , Christiano et al. , 2017 , Rafailov et al. , 2023 ] .

Commutative diagram illustrating 16.1 The scalar-reward declaration.

Let the columns of \(Z_x\) form a basis of the cycle space \(\ker B_x\).

Definition 16.1 Scalarization obstruction

The obstruction to scalar reward representation is

\[ \Omega _x=Z_x^\top \ell _x, \]

or, independently of a chosen basis,

\[ [\ell _x]\in C^1(G_x;\mathbb R)/\operatorname {im}d_0. \]
Theorem 16.2 Cycle criterion for reward scalarization

For a connected comparison graph, the following are equivalent:

  1. \(\ell _x=B_x^\top r_x\) for some scalar reward \(r_x\);

  2. \(z^\top \ell _x=0\) for every cycle flow \(z\in \ker B_x\); and

  3. \(Z_x^\top \ell _x=0\) for one, hence every, cycle basis.

When these conditions hold, \(r_x\) is unique up to an additive constant.

Proof ▼

The fundamental theorem of linear algebra gives

\[ \operatorname {im}B_x^\top = (\ker B_x)^\perp . \]

Connectedness leaves only the constant vector in \(\ker B_x^\top \), which is the reward gauge.

For a triangle \(i,j,k\), the witness is simply

\[ \Omega _x(i,j,k) = \ell _x(i,j)+\ell _x(j,k)+\ell _x(k,i). \]

This is the cycle component in the graph-Hodge decomposition of pairwise rankings [ Jiang et al. , 2011 ] . LINCS uses it as the obstruction to a declared downstream representation, rather than merely as another fit statistic.

Boundary

Vanishing circulation certifies scalarizability only on the graph that was observed. A tree has no cycle witness: every observed field admits a potential, yet unobserved comparisons can complete it to either a scalar or a cyclic field. “No witness” is unidentifiable, not accepted.