Fourier Feature Networks

In an attempt to mitigate Spectral Bias, this method maps input x coordinates into a set of random Fourier Features.

Stemming from Fourier Series

We start with a Fourier Series

$$f(x)=\sum _{n=-\infty }^{\infty }\left[a_n\cos \left(\frac{2\pi nx}{L}\right)+b_n\sin \left(\frac{2\pi nx}{L}\right)\right]$$

Generalize to multiple dimensions. Given $x\in {\mathbb{R}}^d$

$$f(\mathbf{x})=\sum _{n=-\infty }^{\infty }\left[a_n\cos \left(\frac{2\pi n\cdot \mathbf{x}}{L}\right)+b_n\sin \left(\frac{2\pi n\cdot \mathbf{x}}{L}\right)\right]$$

Replace harmonic frequencies with arbitrary frequencies (because we don’t know what these frequencies are beforehand). Instead of $n=\cdots -1,0,1,2,\dots$ we use arbitrary frequency vectors ${\omega }_i\in {\mathbb{R}}^d$

$$f(\mathbf{x})=\sum _{i=1}^m\left[a_i\cos (2\pi {\omega }_i\cdot \mathbf{x})+b_i\sin (2\pi {\omega }_i\cdot \mathbf{x})\right]$$

Because we don’t know these harmonic frequencies, we randomly sample them. The current best choice to do so is sampling using a Gaussian Distribution.

$${\omega }_i\sim \mathcal{N}(0,{\sigma }^2\mathbf{I})$$ $$f(\mathbf{x})\approx \sum _{i=1}^m\left[a_i\cos (2\pi {\omega }_i\cdot \mathbf{x})+b_i\sin (2\pi {\omega }_i\cdot \mathbf{x})\right]$$

Meaning that we are making the assumption that we can represent our output as approximately a sum of sine and cosine functions at differing magnitudes at a set of enough randomly sampled frequencies.

Given that we have a bunch of $\omega$, we can finally stack our component form into a matrix representation of our mapping.

$$\gamma (\mathbf{x})=\left[\begin{array}{c}\cos (2\pi {\omega }_1\cdot \mathbf{x})\\ \sin (2\pi {\omega }_1\cdot \mathbf{x})\\ \cos (2\pi {\omega }_2\cdot \mathbf{x})\\ \sin (2\pi {\omega }_2\cdot \mathbf{x})\\ ⋮\\ \cos (2\pi {\omega }_m\cdot \mathbf{x})\\ \sin (2\pi {\omega }_m\cdot \mathbf{x})\end{array}\right]\in {\mathbb{R}}^{2m}$$

And we are trying to find

$$f(x)=w^T\gamma (x),\text{where}w=\left(\begin{array}{ccccccc}{a}_1& b_1& a_2& b_2& \dots & a_m& b_m\end{array}\right)$$

Which in a nice way looks like a Linear Layer! (see Layers)

Where we’re essentially learning the coefficients of a Fourier Series sampled at random frequencies $\omega$