4.7.4 A preregistered evaluation design

4.7.4 A preregistered evaluation design

A minimal evaluation trains one external discovery skill across registered synthetic worlds. Training worlds expose exact or simulator-checkable certificates; validation uses fresh worlds and seeds; admission withholds both the structural defect and its certificate until the skill is frozen. Transfer to categorical symbolic regression or simulator-grounded theory extension is a separate, stronger claim rather than part of the in-distribution result.

The minimum comparison includes:

1. the frozen agent without a skill document;

2. a manually written discovery skill;

3. ordinary SkillOpt using only terminal task score;

4. fixed-geometry LASKO skill optimization without accommodation;

5. DIAL execution with an untrained discovery skill;

6. DIAL-SkillOpt without the mixed witness; and

7. complete DIAL-SkillOpt.

Primary outcomes are locked-set admission rate, false-extension rate, correct localization, experimental cost, and abstention calibration. Secondary outcomes include time to a valid proposal, preservation of old results after transport, transfer to a new task family, and the fraction of admitted extensions that remain useful in later episodes. Shuffled-witness and equal-budget controls test whether improvement comes from the double geometry rather than extra information or computation.

. Train the skill that conducts discovery, not the admission rule that declares truth. Curve-object flows make local exploration reproducible; DIAL makes the two repair modes and their interaction observable; SkillOpt makes the external controller iteratively improvable. A finite theory extension remains a typed proposal whose scientific standing comes from evidence the optimizer could not edit.