LayerNorm

LayerNorm is an operation that shifts and rescales a vector so that its mean is zero and its variance is 1.0, then re-scales this result by two learned parameters.

This scales the magnitudes of one vector (activations) to be scaled to match another vector before they are mixed.

When applied to a vector of length $d$, LayerNorm contributes $2d$ learned parameters to a model everywhere it is used.

Mathematically

Given an input vector $x$ of dimension $d$, LayerNorm is:

$\text{LayerNorm}(\mathbf{x})=\gamma \odot \frac{\mathbf{x}-\mu }{\sigma +ϵ}+\beta$ where

$\mu =\frac{1}{d}{\sum }_{i=1}^d{x}_i$ is the mean of the input vector

$\sigma =\sqrt{\frac{1}{d}{\sum }_{i=1}^d(x_i-\mu {)}^2}$ is the standard deviation of the input vector

$\gamma ,\beta \in {\mathbb{R}}^d$ is the learned per-dimension scale and bias

$ϵ=10^{-5}$ is a numerical stability constant

I don’t really know what this stuff means.