Why P-027 doesn’t use LayerNorm

The problem LayerNorm solves

Picture a network with 240 layers. Each layer takes a vector of numbers, transforms it, and hands it to the next layer. Suppose one layer produces $x = [2,\ 4,\ 6,\ 8]$. Nothing in the mathematics of a standard residual layer stops a later layer from producing $[200,\ 400,\ 600,\ 800]$ — the same pattern, one hundred times the magnitude. The shape of the information survived; its scale quietly drifted. Scale drift is not a cosmetic problem. Very large activations can saturate the arithmetic they live in, and the gradients that flow backward through the same layers can starve or explode as the drift compounds.

And it does compound. In our measurement harness, a randomly-initialized standard residual stream at depth 120 multiplied its state norms by $1.95 \times 10^{17}$ — and that is the stream we manually rescaled so the numbers would fit in the float format at all. Left alone, the control stream reaches $10^{43}$ by depth 300. Depth is rented, never owned: every layer pays rent in activation norm, and the standard architecture has no ledger, no budget, and no ceiling — only a corrective patch applied after the fact.

What LayerNorm does, exactly

LayerNorm is the patch, and a genuinely clever one. It takes each sample’s vector, subtracts its mean, and divides by its standard deviation — re-centering and re-scaling every layer’s output to a standard size before the next layer sees it. On our running example, with $x = [2, 4, 6, 8]$:

1. Mean: $\mu = 5$. Subtract it: $x - \mu = [-3,\ -1,\ 1,\ 3]$ — centered on zero.
2. Standard deviation: $\sigma \approx 2.236$. Divide by it: $\approx [-1.34,\ -0.45,\ 0.45,\ 1.34]$ — controlled scale.

In symbols, $\mathrm{LN}(x) = \gamma \odot \dfrac{x - \mu(x)}{\sqrt{\sigma^2(x) + \epsilon}} + \beta$, where $\gamma$ and $\beta$ are learned, and $\epsilon$ is a tiny guard against division by zero. The property that matters most is scale invariance: for any positive factor $c$, $\mathrm{LN}(cx) = \mathrm{LN}(x)$. Multiply the input by a hundred and the output does not move. You can check the arithmetic yourself:

The catch: it normalizes layers, not streams

Here is the distinction the whole paper turns on. LayerNorm’s guarantee is a statistic of each layer’s activations, per sample, after the fact. It is a real guarantee — but it is not a bound on the network. The next learned map after a LayerNorm has whatever spectral norm training gives it, the learned $(\gamma, \beta)$ can restore any scale and shift the normalization removed, and nothing in the construction prevents a trained network from growing its representation layer over layer, because scale is exactly the degree of freedom $\gamma$ holds. LayerNorm was never asked to bound the composed operator across 240 layers. Nothing else in the architecture was either.

Our training experiment (E6: 96 runs, four architectures, identical protocol) makes this concrete. The LayerNorm baseline — normalization placed on the branch’s pre-activations, the standard ResNet-with-LN configuration, with the residual trunk itself normalized nowhere — trains beautifully: 97.4% held-out at depth 240, parity with everything. And its measured trunk growth is $51.0\times \pm 7.4$ at that depth: statistically indistinguishable from the plain ResNet’s $48.2\times \pm 4.1$, and the highest of the four architectures. The normalizer did not reduce the trunk’s forward growth, because nothing in that architecture speaks to that quantity. LayerNorm manages weather; it does not issue a budget.

What P-027 does instead

The port-Hamiltonian stream replaces the residual update $x \to x + f(x)$ with a state update whose three channels never mix jobs: $$x_{t+1} \;=\; \Lambda_t\, C_t\, x_t \;+\; h\, B_t\, u_t.$$ Transport $C_t$ is a Cayley rotation — exactly orthogonal at any step size, so it moves information around the state without ever changing its norm: $\|C_t x\| = \|x\|$, by construction, for every $x$. Dissipation $\Lambda_t$ is a symmetric contraction with spectrum in $(-1, 1)$ at any learned parameters — the network can choose to dissipate less, but it cannot choose to amplify the state channel. The ceiling is not a training outcome; it is the parameterization. Ports $B_t u_t$ are the only door norm can enter through.

Summing the per-step inequality over depth gives the port budget, and this is the theorem LayerNorm has no counterpart of: $$\|x_L\| \;\le\; \|x_0\| \;+\; \sum_{t=0}^{L-1} h\,\|B_t u_t\|.$$ The bound is input-independent — it holds for every input simultaneously, including adversarial ones, because it is a property of the operator, not a statistic of the data. And it binds the backward pass identically: the gradients flow through the same $\Lambda_t C_t$ factors, transposed, so forward contraction and backward decay are one law. You can feel the difference in arithmetic:

The two guarantees, side by side

What the experiments say

E6 trained four architectures under one protocol — held-out splits, validation-selected learning rates, early stopping, three seeds, gradient and norm signatures measured on the trained models. At depth 240 on circles-4:

Read honestly, the table says three things. First: the architecture trains — accuracy at parity with every baseline at depth 240, with no normalizer anywhere in the stream, which is the claim the external audit demanded we test. Second: accuracy parity is the claim, not victory — 97.3 against 97.4 is a tie, and we report it as one. Third: the signatures differ where the theory says they must — the PH net’s growth is the lowest of the four at every tested depth, and unlike the baselines’ growth it is bounded by a quantity you can read off the parameters before training starts. Fourth, new since the E6b run: the trunk-LN row is the trade made concrete — the only placement that pins trunk scale (3.2×, flat across depth) is the one that collapses at depth 240 (40.9%) with the backward gradient max at 3×10⁸: statistical control of the median, no operator bound on the tail. On the harder spirals-2 task, no architecture solves the problem under this protocol (PH 57.6%, everyone else 55–67%, chance 50%) — we report that as found, because that is what it is.

Where this could be wrong

The paper keeps a running list of its own soft spots (Section 8), and the explainer should be at least as honest. Three objections carry real weight. Scale: the synthetic results were measured at width 32 on two toy tasks; the real-data round (E7a, sixty CIFAR-10 runs) is now in, and its verdict is mixed — stability and conservation transfer (zero PH failures against three trunk-LN collapses, growth bounded, no normalizer anywhere), but no forward-growth advantage at depth 48 and a 4× per-run compute bill. The paper reports exactly that, and still claims no competitiveness with tuned recipes. Placement: the LayerNorm baseline uses the standard in-branch placement with an un-normalized trunk; the trunk-normalized (post-LN) variant — the placement that would pin trunk scale — ran as E6b: it pins the trunk (3.2×, flat) and collapses at depth 240 with the backward maximum at 3×10⁸. The prediction was registered before the run; the branch that fired is the one the paper’s wager drew. Substitution: LayerNorm buys things at scale that an operator bound does not address — optimization conditioning, tolerance to initialization and width changes — and whether the budget substitutes for those at real scale is precisely the open research question. The question was never “is LayerNorm bad?” It is: how much of what we buy from normalization can be bought from architectural guarantees instead?