ora-0068

4.8.1 What would a Transformer have to demonstrate?

The comparison must hold training data and interaction fixed and vary the obligation being tested. A serious Transformer challenge should include at least the following contrasts.

Unseen composition: answer universal-property queries whose components appeared during learning but whose composite did not.
Re-presentation: preserve the answer after renaming, refactoring, changing viewpoint, or replacing a presentation by an equivalent one.
Compatible continuation: retain a settled query theory when new objects, arrows, or observations extend the world.
Localized contradiction: distinguish a faulty realization from a faulty overlap warrant and from evidence against the categorical doctrine.
Underdetermination: abstain, or return the residual hypothesis fiber, when the observations do not identify an answer.
Transport: reuse the discovered structure under a new task, observer, or modality without relearning the relevant compositions from scratch.

These tests differ from asking whether a model can solve a familiar crossword, write a function, or describe an object. They ask whether success persists under controlled changes to the presentation and evidence process. Ordinary benchmark scores may be poor witnesses for this purpose: recent audits have shown that apparently precise coding evaluations can contain underspecified prompts, overly restrictive tests, and incomplete behavioral coverage [ OpenAI , 2026b ] .

The challenge should not be framed as a return to a rigid symbolic doctrine. Neural networks trained with an appropriate compositional curriculum can exhibit human-like systematic generalization [ Lake and Baroni , 2023 ] ; conversely, strong interpolation can coexist with failure on controlled systematic splits [ Lake and Baroni , 2018 ] . This is evidence that architecture alone does not decide the issue. The presentation, training objective, observer, and admissible query transformations all matter.