Normalization

BatchNorm

Normalizes data to a $\mu =0\sigma =1$ across a batch.

$${\hat{x}}_i=\frac{x_i-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}}$$ $$y_i=\gamma {\hat{x}}_i+\beta$$ $${\mu }_B=\frac{1}{m}\sum _{i=1}^m{x}_i\text{(batch mean)}$$ $${\sigma }_B^2=\frac{1}{m}\sum _{i=1}^m(x_i-{\mu }_B{)}^2\text{(batch variance)}$$

LayerNorm

Normalizes across all features in each sample independently.

$$\hat{x}=\frac{x-{\mu }_L}{\sqrt{{\sigma }_L^2+ϵ}}$$ $$y=\gamma \hat{x}+\beta$$ $${\mu }_L=\frac{1}{H}\sum _{i=1}^H{x}_i\text{(mean over features)}$$ $${\sigma }_L^2=\frac{1}{H}\sum _{i=1}^H(x_i-{\mu }_L{)}^2\text{(variance over features)}$$