6.14 A four-level symbolic-discovery challenge

The AI Feynman suite can anchor a more demanding four-level test without replacing its established task.

1. Equation recovery. Reproduce symbolic regression from numerical samples, recording predictive error, symbolic equivalence, expression complexity, and computation. This is an assimilatory task relative to the registered expression language.

2. Structural recovery. Return not only a formula but a typed presentation of its variables, units, symmetries, separability relations, and compositional factorization. The certificate compares the recovered structure with a hidden reference sketch. It remains assimilatory when that structural vocabulary is already supplied by the benchmark’s meta-language.

3. Representation repair. Begin with a deliberately inadequate grammar or ontology. Permit the system to introduce a latent quantity, new operation, quotient, or coordinate system when the obstruction cannot be resolved inside the original language. This tests a proto-accommodative proposal rather than admitted theory change.

4. Theory-package construction. Require the extension map, the transport of earlier fits, new predictions, countermodels, and an admission status. Evaluate the package on held-out regimes or simulator interventions that were unavailable during proposal. This tests whether a proposed accommodation survives equilibration with independent evidence.

The benchmark must also control memorization. Famous Feynman equations are likely to occur in frontier-model training data. We therefore need procedurally generated “Feynman-like” families, blinded renamings and coordinate changes, novel compositions of known mechanisms, and withheld intervention regimes. Historical equations remain useful calibration cases, but claims about creative mechanism should rest on worlds whose generating structure the model could not have retrieved verbatim.

Experiment: AI Feynman extension benchmark. Epistemic status: proposed benchmark design; this card specifies an evaluation target rather than reporting a completed experiment.
Input: numerical observations, declared units, and a versioned expression grammar; later levels withhold or corrupt parts of the grammar.
Proposal: equation, structural sketch, and any requested theory extension.
Admission: symbolic equivalence or predictive fit, recovery of hidden structure, extrapolation under interventions, and audit of transport.
Creativity label: equation recovery is Mathematics–E; a new primitive or doctrine is a candidate Mathematics–T result before admission and a benchmark-relative Mathematics–T result only after independent admission.
Boundary: recovery of a known formula is personal or system-relative creativity, not evidence of historical scientific discovery.