18.7 Textual inversion as staged calibration

The Diffusers implementation of textual inversion provides a reproducible bridge from the preceding distinctions to an executable experiment [ Hugging Face , 2026 ] . Its training script adds one or more placeholder tokens, initializes their embeddings from a registered token, and updates only those embeddings against a small image collection. The tokenizer, text encoder apart from the new rows, variational autoencoder, denoiser, noise schedule, and base model can therefore be frozen explicitly. This is unusually valuable for ARTISTIC: the intervention is small enough to state exactly what representational structure the learner was permitted to change.

The initial experiments use textual inversion as a staged calibration, not merely as a competitor. Their identifiers use TI for textual-inversion calibration; the later natural-image studies use NI.

ARTISTIC-TI–0: concept naming.

Give the learner a few views of a novel procedural object or style and test whether a nonce token retrieves it under held-out backgrounds, viewpoints, colors, and prompts. This condition is designed as a positive control for lexical abbreviation and concept naming; the substantive result below shows that the intended control was not admitted.

ARTISTIC-TI–1: productive composition.

Hold out combinations of the learned concept with registered objects, attributes, counts, and spatial relations. Compare one learned vector with a matched multi-vector representation. Success requires preservation of both the new concept and every independently varied registered factor.

ARTISTIC-TI–2: relation naming.

Replace the object target by a nonce binary relation whose instances vary over object identity, order, viewpoint, and scene layout. The question is whether a token embedding can express a constructor rather than memorize a visual template. Counter-witness prompts reverse the arguments, preserve the objects while removing the relation, and instantiate the relation on unseen object pairs.

ARTISTIC-TI–3: candidate extension.

If TI-2 fails under adequate coverage, uncertainty, and optimization controls, freeze that failure as an obstruction. ARTISTIC may then propose a bounded typed extension—for example a relation-specific mediator, attention control, or adapter—and must demonstrate that the added structure, rather than extra training budget, supplies the missing productivity.

The comparison includes existing-token paraphrases, prompt search, a randomly initialized frozen token, textual inversion with matched compute, and a registered adapter oracle. Evaluation uses scene-graph truth in procedural worlds: concept fidelity, argument order, exact relation satisfaction, preservation of irrelevant factors, held-out composition, cross-seed transfer, and false activation. Every run records the base-model revision, Diffusers revision, initialization token, number of learned vectors, image and prompt splits, seeds, checkpoints, and compute budget.

Experiment: ARTISTIC-TI–0–3. Epistemic status: consolidated evaluation protocol distilled from the TI calibration ladder; the card defines the stronger language-extension test rather than claiming that it has been passed.
Question: when does adding a visual “word” cease to be embedding acquisition and require an extension of the visual language itself?
Positive control: few-shot naming of a novel object or style with a frozen text-to-image model.
Boundary challenge: a nonce relation applied to unseen ordered object pairs and scene configurations.
Accommodation trigger: a preregistered productivity defect that persists across seeds, vector counts, matched compute, and counter-witness acquisition.
Admission: the proposed extension passes held-out composition and counterfactual tests without degrading the frozen vocabulary.
Interpretive limit: a successful personalized token is evidence of concept acquisition; by itself it is not evidence that a new grammar or visual ontology has been invented.