A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function

The Gaussian Error Linear Unit is usually motivated as the expected output of an input-dependent stochastic Bernoulli gate.

This work gives a complementary interpretation based on the Gaussian complementary first-order loss function: GELU is the signal-transmission term of the expected surplus of a hard linear gate with a Gaussian random threshold.

This view separates loss accounting from forward signal transmission and generalises to a threshold-transmission family that includes ReLU, GELU, SiLU/Swish, and hard swish as special cases.

The uniform-threshold case recovers a hard-swish-like compact piecewise-polynomial gate with an explicit threshold-width parameter, yielding fixed- and learned-width variants.

Controlled experiments on compact vision and language models show that calibrated or learned uniform-threshold gates are consistently competitive with GELU, ReLU, and SiLU/Swish, improve over them in most tested settings, and use the finite transition region nontrivially.