lin-0034

1.3 Nonsmooth refinement: from a derivative to a response space

The Kock–Lawvere axiom makes the first-order response unique. This is exactly the property that a nonsmooth component may lose. Let

\[ \rho (x)=\max (0,x) \]

be the rectified linear unit. Away from the origin it has an ordinary derivative. At the origin its directional derivative is

\[ \rho '(0;v)=\max (0,v), \]

which is positively homogeneous but not linear in \(v\). There is therefore no single linear tangent map that describes its response in every direction. Relative to a sketch that declares unique first-order factorization, the kink is a compositional obstruction.

For a proper convex function \(f:\mathbb R^n\to \mathbb R\cup \{ +\infty \} \), convex analysis does not remove a kink by choosing an arbitrary derivative. It replaces the missing point-valued derivative at \(x\in \operatorname {dom}f\) by the subdifferential

\[ \partial f(x) = \left\{ g\in \mathbb R^n: f(y)\geq f(x)+\langle g,y-x\rangle \text{ for every }y \right\} . \]

For ReLU,

\[ \partial \rho (x) = \begin{cases} \{ 0\} , & x{\lt}0,\\ \mathopen {[}0,1\mathclose {]}, & x=0,\\ \{ 1\} , & x{\gt}0. \end{cases} \]

The response is a singleton on the smooth strata and a convex family at the kink. Subdifferential calculus studies how such generalized responses transform under composition and other operations [ Rockafellar , 1980 ] .

Definition 1.4 Registered generalized tangent response

For an admissible nonsmooth component \(f\) at \(x\), a generalized tangent response is registered data

\[ \mathcal R_f(x) \]

whose elements are the first-order responses permitted by a named semantics, such as a convex or Clarke subdifferential or an automatic-differentiation conservative gradient. A computational selection is additional data

\[ \sigma _f(x)\in \mathcal R_f(x). \]

The named semantics matters. A single convex ReLU has the convex subdifferential above, but the objective computed by an entire ReLU network is generally nonconvex. Clarke subdifferentials, limiting constructions, and conservative gradients need not agree, and their chain rules require different hypotheses. Modern nonsmooth automatic differentiation shows that backpropagation can efficiently transport conservative gradients through large classes of ReLU programs; it also shows that enumerating distinct Clarke subgradients can be computationally hard [ Bolte et al. , 2022 ] . Thus the value returned by a software derivative convention should not be silently identified with the whole generalized response object.

Design principle

At a nonsmooth point, distinguish three objects: the obstruction to a unique linear response, the registered space of admissible generalized responses, and the representative selected for computation. The second may serve as a repair language for the first; the third is an admission decision, not a mathematical inevitability.

This refinement provides the bridge to the next foundations. Tangent learning sketches in Chapter 4 need not assume that every component already has a unique classical derivative. They may instead declare which response object is admissible, test whether the selected responses compose, and retain the resulting failure as typed tangent information. DLINCS in Chapter 8 then becomes the computational layer that registers a selection policy, transports its choices through the model, and tests whether the resulting backward computation satisfies the declared tangent sketch.

A full categorical treatment would replace some point-valued tangent arrows by convex-valued correspondences, linear relations, or another suitable category of generalized responses. The present book does not assume that extension. It records the interface required of one: response semantics, selection, transport, and admission must remain explicit.