Robust Loss Functions

From Maximum A Posteriori we have the loss function

$$J(\mathbf{x})=\frac{1}{2}\sum _{i=1}^N{\mathbf{e}}_i(\mathbf{x}{)}^T{\mathbf{W}}_i^{-1}({\mathbf{e}}_i(\mathbf{x}))$$

The gradient of which (which is used for Gauss-Newton Method is given by

$$\frac{\partial J(\mathbf{x})}{\partial \mathbf{x}}=\sum _{i=1}^N{\mathbf{e}}_i(\mathbf{x}{)}^T{\mathbf{W}}_i^{-1}\frac{\partial {\mathbf{e}}_i(\mathbf{x})}{\partial \mathbf{x}}$$

Because the cost function is quadratic, our cost explodes when outliers cause major errors. To handle this, there are a series of wrappers that we can use to limit the effects of robot loss functions.

$$J^{\prime }(\mathbf{x})=\sum _{i=1}^N{\alpha }_i\rho (u_i(\mathbf{x}))u_i(\mathbf{x})=\sqrt{{\mathbf{e}}_i(\mathbf{x}{)}^T{\mathbf{W}}_i^{-1}({\mathbf{e}}_i(\mathbf{x}))}$$

where $\alpha$ is a scalar weight you can define, $\rho$ is some non-linear cost function (the wrapper).

Some possible cost functions are

$$\underset{Cauchy}{\underbrace{\rho (u)=\frac{1}{2}\ln (1+u^2)}}\underset{Geman-McClure}{\underbrace{\rho (u)=\frac{1}{2}\frac{u^2}{1+u^2}}}$$ $$\underset{\text{Huber (Quad when close, Lin when far)}}{\underbrace{\rho (u)=\{\begin{array}{ll}\frac{1}{2}{u}^2& \text{if }|u|\le \delta \\ \delta |u|-\frac{1}{2}{\delta }^2& \text{if }|u|>\delta \end{array}}}$$

$\delta$ is a specifiable parameter.

These cost functions don’t explode as much as the squared loss, and thus are more robust towards outliers. The downside is that we end up with slower convergence (but keep in mind that’s only if our data has very few outliers)

In the case of nice data, you don't need Robust Loss Functions like these.