ora-0083

5.5 Reinforcement learning inside the MDP–Bellman doctrine

Fixed doctrine.

Reinforcement learning does not begin merely with rewards and experience. It normally begins inside a doctrine declaring that the environment can be represented by a class \(\mathcal M\) of Markov decision processes

\[ M=(S,A,P,R,\gamma ) \]

with measurable or finite state and action objects, stochastic transition kernels, an additive reward interpretation, an objective such as discounted return, and an admissible policy class. Choosing morphisms such as MDP homomorphisms, simulations, bisimulations, or policy-preserving abstractions then forms a hypothesis category \(\mathbf{MDP}_{\mathcal M}\) [ Puterman , 1994 , Sutton and Barto , 2018 ] .

The decisive additional commitment is Bellman structure. For a discounted control problem, the doctrine must support an operator of the form

\begin{equation} (\mathcal BQ)(s,a) = R(s,a) +\gamma \int _S \sup _{a'\in A(s')}Q(s',a')\, P(ds'\mid s,a). \end{equation}

5.8

The reward must be integrable, the supremum must be meaningful in the chosen value object, and the relevant function space must support the required fixed point. In a finite discounted MDP with bounded rewards, \(\mathcal B\) is a contraction on bounded \(Q\)-functions in the supremum norm, so it has a unique fixed point. Other objectives require different existence and stability assumptions.

Definition 5.11 MDP–Bellman doctrine

An MDP–Bellman doctrine specifies the stochastic category of state transitions, reward and policy objects, the return aggregation rule, an ordered or normed value object, an admitted Bellman operator, and the equivalence under which its solutions represent the same decision behavior.

These are prerequisites of ordinary RL, not discoveries made by the learning algorithm. Q-learning estimates the fixed point only after the Bellman equation has been declared to characterize optimal behavior. Its classical convergence guarantees additionally require stationary transition and reward laws, sufficient visitation of state–action pairs, and step-size conditions. Those assumptions belong to the presentation and update portions of the doctrinal audit.

An interacting learner receives a prefix of state, action, reward, and successor observations. Transition and reward queries ask which MDPs remain consistent with that prefix; value and policy queries add decision semantics. Model-based reinforcement learning identifies \(P\) and \(R\), or a decision-sufficient quotient of them, and then invokes the internal planning construction supplied by the doctrine.

Model-based reinforcement learning is consequently a direct specialization: its UOCL component identifies an MDP model or an observational quotient, and its UODL component uses that model to select consequential actions. The environment’s response makes the presentation endogenous because an action changes which evidence arrives next.

Model-free reinforcement learning requires a more careful statement. A Q-learning agent need not identify \(P\) or \(R\) separately. It seeks a control-sufficient quotient represented by values or a policy. Calling this full identification of the MDP would be false. It is UOCL only relative to the smaller query doctrine whose answers are the value or action distinctions the algorithm preserves; as a decision process it lies naturally in UODL.

Doctrinal audit.

The structural doctrine is the MDP–Bellman package; trajectories provide the endogenous presentation; Bellman optimization or stochastic approximation is the internal solution construction; value, return, and policy form the privileged query doctrine; and bisimulation or policy equivalence determines the visible quotient. The doctrine has been learned only if Markovity, reward aggregation, Bellman adequacy, and the policy/value interfaces are themselves among the hypotheses that interaction can compare.