5.5.1 Options as persistent compositional objects

5.5.1 Options as persistent compositional objects

The options framework gives classical reinforcement learning a revealing fragment of compositional persistence. An ordinary task policy \(\pi :S\to \mathcal D A\) is usually valued only through the return it obtains on its parent task. Even when part of that policy performs a recognizable subtask, the learned parameters do not by themselves declare where the behavior is applicable, when it has completed, or how another decision maker may invoke it. The useful composite remains implicit in the solution.

An option promotes such a composite into a named, action-like object

where \(I_o\subseteq S\) is its initiation set, \(\pi _o\) is its closed-loop policy, and \(\beta _o:S\to [0,1]\) is its termination rule [ Sutton et al. , 1999 ] . Executing \(o\) generates a distribution over termination states, accumulated rewards, and durations. This induced semi-Markov model is the option’s external interface. A planner or learner can therefore treat the entire temporally extended behavior as one action while retaining primitive actions at the lower level.

Categorically, an option is not merely a long path. Its policy selects a controlled family of paths from each admissible starting state, while its termination rule quotients their internal histories into an observable exit distribution. The option model is consequently a macro-morphism in a stochastic action category. Two options compose when the exit support of the first lies within the initiation region of the second; Kleisli composition integrates over the intermediate termination state. The triple \((I_o,\pi _o,\beta _o)\) is thus an interface declaration that makes a learned behavior available for later composition.

This explains why an option can outlive the task on which it was discovered. A “navigate to doorway” option learned while delivering one object may be reused while searching, cleaning, or escaping, even when none of those tasks contains the original delivery objective. What persists is not the parent policy but a promoted substructure with a name and invocation contract. In ORACLE language, the learner has moved a useful composite from an ephemeral realization into its storehouse of compositional knowledge.

The promotion is not automatically valid across tasks. A policy optimized under the parent reward may exploit incidental dynamics; its initiation set may omit states encountered elsewhere; and its termination predicate may not match the boundary required by a new composition. Reusing a label despite these changes is syntactic persistence, not semantic transport. Genuine persistence requires an observer-relative equivalence under which the option model, safety conditions, and interface obligations survive the change of task or are repaired locally.

Options therefore sit between conventional RL and ORACLE. They show that temporally extended knowledge can be learned once and invoked in unrelated future objectives. ORACLE generalizes the idea beyond a fixed MDP doctrine: it asks how the option itself is identified, when two realizations express the same reusable skill, how option interfaces compose across modalities and world models, and when failed reuse demands accommodation of the skill’s declaration rather than another value update.

This perspective also locates the difference from classical RL. RL normally fixes the MDP doctrine and learns within it. General UOCL may accommodate by revising that doctrine: partial observability, nonstationarity, asynchronous information, non-Markovian composition, a failure of scalar additive reward, or the absence of the assumed Bellman fixed point may force the learner to leave \(\mathbf{MDP}_{\mathcal M}\) itself. Re-running Q-learning with more samples cannot repair a world that lies outside the doctrine that makes Q-learning meaningful.