Neural Collapse Is Forbidden: Information Floors in Language Models

Within-class variance in language-model representations is commonly read as incomplete neural collapse. We argue it is allocated information storage, and that the allocation obeys a law. A one-line centering identity voids a family of simplex equiangular-tight-frame claims, including our own earlier ones; in dimensionless variance shares across 14 models, macro-category structure carries only 4-12% of representational variance and within-token context carries 79-91%, stable across a 100x parameter range. On the theory side, token-level weight decay penalizes a category in proportion to its type count, not its occurrence mass, reducing next-token prediction to an imbalanced K-class problem whose optimum orders category norms by type count. A converse floor, proved for binary categories, forces within-category dispersion to be at least proportional to the conditional mutual information I(token; context | category). The law holds: identity dispersion, not total variance, tracks this information across every tested model and partition, under a model-free estimate and even across models, where one model's information predicts another's dispersion; and over pretraining the category share overshoots, decays, and partially recovers, because the information it must carry never left.
Ancillary-file links:
Ancillary files (details):
- RELEASE-V2.md
- a2prime-attempt-2026-07-03.md
- a2prime-proof-2026-07-03.md
- coupling_estimate.json
- dynamics_shares.json
- floor_direct.json
- frame_mass.json
- freqnorm.json
- gate_a.json
- gate_a_gpt2.json
- gate_a_gpt2_train.json
- gate_a_pythia-1.4b.json
- gate_a_pythia-1.4b_train.json
- gate_a_pythia-410m.json
- gate_a_pythia-410m_train.json
- gate_a_qwen2.5-1.5b.json
- gate_a_qwen2.5-1.5b_train.json
- gate_a_train.json
- gaussian_control.json
- headrow_test.json
- law_final_layer.json
- mixedlm.json
- olmo2_dynamics.json
- olmo2_fingerprints.json
- olmo2_preregistration.md
- olmo2_tokmap.json
- pile_transfer.json
- pooled_partials.json
- pos_alpha0.json
- review_response.json
- shares.json
- signlemma-2026-07-04.md
- signlemma_stress.json
- twisted_seli_check.json
- twisted_seli_sweeps.json
- v2_coupling_estimate.py
- v2_figures.py
- v2_floor_direct.py
- v2_freqnorm.py
- v2_gate_a.py
- v2_headrow_test.py
- v2_mixedlm.py
- v2_olmo_dynamics.py
- v2_phase0.py
- v2_pile_transfer.py
- v2_pooled_partials.py
- v2_pythia28_redo.py
- v2_review_response.py
- v2_signlemma_stress.py
- v2_twisted_seli_check.py
- v2_twisted_seli_sweeps.py
- v2_wordnet_law.py
- validate.json
- wordnet_law.json