13.7 Why novelty and one-step guidance are insufficient

B1 compares the official TextDQN under extrinsic reward alone with the same learner receiving ALFWorld’s episodic-count and novel-object rewards. Both arms begin from byte-identical weights and share the same seed, replay budget, action channel, and evaluation games. Neither receives extrinsic training reward or solves an evaluation game. Yet the manipulation is active: the intrinsic arm receives dense shaped return, selects a different repeated action at evaluation, and ends far from the extrinsic-only checkpoint.

The failure is therefore not “nothing learned.” What was learned did not compose into goal-directed task structure. Novelty values an unfamiliar state or object; it does not declare why that encounter matters to a pickup-and-place factorization.

B2 sharpens the comparison. At episodes 12, 24, and 36 it evaluates three TextDQN arms: ordinary exploration with an epsilon-greedy rule; count/object novelty; and typed goal-relevant exploration. The third arm changes only exploratory proposals, favoring actions such as take, put, open, and task-matched navigation. None of the three solves a training or evaluation game. The structural arm briefly learns a more directed navigation and examination pattern at episode 12, then falls into the same repeated-action attractors as the other arms.

All checkpoints move substantially and diverge from one another. B2 therefore rules out an inactive optimizer and rules in a more specific diagnosis: isolated useful-looking actions do not constitute a temporally extended policy. The missing object is a composition.