15.13.5 The concrete sequential worlds

15.13.5 The concrete sequential worlds

The exact URL–0 laboratory has two evaluator states \(x_0,x_1\) and two actions, stay and flip. In the direct world, stay preserves \(x_i\), flip maps it to \(x_{1-i}\), and the observation map distinguishes the two states. Duration is one and reward is zero. This is the smallest fully observed control.

Three modifications create the positive cases. First, the observation map may send both states to the same symbol. Aliasing alone is not enough to require a hidden-state constructor: in the quotient control, both states have identical rewards and induce the same distribution over next observations under every action. By contrast, the POMDP world gives \(x_1\) reward one under stay and \(x_0\) reward zero. The same visible symbol now conceals a behavioral distinction. Second, the pair \((x_1,\mathtt{flip})\) may have duration support \(\{ 2,3\} \) instead of \(\{ 1\} \), requiring the Time constructor. The composite world contains both defects.

URL–1 replaces these exact maps by stochastic trajectories. An episode starts in one of two hidden states and runs for 25 steps. In directly observed worlds, the visible symbol is \(o_i\); in aliased worlds both states emit \(o\). For the direct dynamics, action zero makes state one likely with probability 0.9 from \(x_1\) and 0.1 from \(x_0\), while action one reverses these tendencies. Aliased dynamics are persistent: the hidden state remains unchanged with probability 0.92 under action zero and 0.82 under action one. Rewards expose this memory only statistically—probability 0.88 in \(x_1\) and 0.12 in \(x_0\). Semi-Markov worlds independently replace unit duration by two or three with probability 0.25.

The learner never sees the hidden state in URL–1. It compares a predictor using only \((o_t,a_t)\) with one that also receives the previous observation, action, and reward. A significant held-out log-score improvement is evidence for an observation/history extension. A confidence interval for non-unit durations supplies the independent Time decision. Thus the “experiment” is concrete: act in a two-state stochastic system, collect episodes, and ask whether history or elapsed time improves its observable one-step account.

Boundary: A quotient control. Suppose two hidden names both emit \(o\), both return reward zero, and every action sends each name to the uniform distribution on the same two names. There are two simulator states but only one observable behavior. DIAL-URL must return Base. Treating hidden multiplicity itself as evidence for partial observability would be a false theory extension.