13 Reinforcement Learning with Infinitesimal Creativity

RELIC: From Policy Optimization to Structural Accommodation

In a standard reinforcement-learning formulation, an agent improves a policy inside a declared decision problem. The state variables, action types, transition interface, reward representation, and temporal grammar are fixed by that formulation before policy learning begins. Exploration changes the evidence available to the agent; optimization changes its value function or policy. Neither operation, by itself, changes what a policy is allowed to be.

RELIC—Reinforcement Learning with Infinitesimal Creativity—asks a different question. What should an agent do when its failure is evidence that the current decision language is inadequate? The answer developed in this chapter is not to identify creativity with a larger exploration bonus. The agent must detect a typed obstruction, localize the missing composition, propose a structural repair, test it on evidence not used to propose it, and retain it only after admission.

The experiments below provide the sequential-decision bridge from the DIAL-X architecture of Chapter 10 to an embodied learning system, complementing the causal-learning and skill-construction bridges developed by CLIC and OPTIC in Chapters 11 and 12. They form a progression rather than one headline comparison. Generic novelty changes learning without producing a task theory. Isolated goal-relevant actions also fail to compose. A temporally extended pickup-and-place schema, however, can be triggered by an observed obstruction, admitted on disjoint games, and transported to untouched houses. The result is a controlled mechanism demonstration for structural accommodation. It is not yet a claim that the agent autonomously invented the schema.

¹1. Chapter roadmap. RELIC first separates policy learning from declaration change and fixes its target card. The ALFWorld ladder then tests persistent artifacts, novelty controls, supplied temporal composition, constructed composition, and finally construction of a missing stage. ↩