Exploration

Collect preferences over outputs that differ along independently registered criteria, including deliberately inconsistent comparisons. Compare scalar reward modeling with LINCS-RLHF while holding annotations fixed. Evaluate held-out preference prediction, contradiction localization, criterion preservation, and abstention. Does structural preference modeling reveal conflicts that a sufficiently flexible scalar model conceals?

Digital-edition research invitation; not part of the frozen dated PDF.

ch-lincs-rlhf

16 Learning from Structured Preferences

LINCS–RLHF

RLHF usually turns pairwise preferences into a scalar reward and then optimizes that reward. LINCS–RLHF treats the scalar reward as a factorization hypothesis. If comparison log odds circulate around a cycle, no global scalar potential represents the declared preference relation. More reward optimization cannot repair a representation that does not exist.

¹1. The structural question comes before optimizer choice: should the preference relation be represented by a scalar potential at all? ↩