Stereo Camera

We express everything in a stereo camera with respect to the coordinate frame at the midpoint (known as the midpoint model).

$$\mathbf{\rho }={\mathbf{r}}_s^{ps}=\left[\begin{array}{c}x\\ y\\ z\end{array}\right]$$

The model for the left and right camera are as follows

$$\left[\begin{array}{c}{u}_l\\ v_l\end{array}\right]=\mathbf{PK}\frac{1}{z}\left[\begin{array}{c}x+\frac{b}{2}\\ y\\ z\end{array}\right]\left[\begin{array}{c}{u}_r\\ v_r\end{array}\right]=\mathbf{PK}\frac{1}{z}\left[\begin{array}{c}x-\frac{b}{2}\\ y\\ z\end{array}\right]$$

Assuming that the two cameras have the same intrinsic parameters

Stacking the two we get

$$\left[\begin{array}{c}{u}_l\\ v_l\\ u_r\\ v_r\end{array}\right]=\mathbf{s}(\mathbf{\rho })=\underset{\mathbf{M}}{\underbrace{\left[\begin{array}{cccc}{f}_u& 0& c_u& f_u\frac{b}{2}\\ 0& f_v& c_v& 0\\ f_u& 0& c_u& -f_u\frac{b}{2}\\ 0& f_v& c_v& 0\end{array}\right]}}\frac{1}{z}\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right]$$

You can also model the stereo camera with respect to the Left or right frames as well.

Left Model

The camera model becomes

$$\left[\begin{array}{c}{u}_l\\ v_l\\ u_r\\ v_r\end{array}\right]=\underset{\mathbf{M}}{\underbrace{\left[\begin{array}{cccc}{f}_u& 0& c_u& 0\\ 0& f_v& c_v& 0\\ f_u& 0& c_u& -f_{u}b\\ 0& f_v& c_v& 0\end{array}\right]}}\frac{1}{z}\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right]$$

The thing about stereo cameras is that we know the distance between the two cameras and the intrinsics of both. Because they lie in the same plane (only offset by b), we can formulate a relationship between a Point , $P$ z value and disparity

$$d=u_l-u_r=\frac{1}{z}{f}_{u}b$$

but keep in mind that we dont know z, and we have to usually guess using correspondance.

$$\left[\begin{array}{c}{u}_l\\ v_l\\ d\end{array}\right]=\mathbf{s}(\mathbf{\rho })=\underset{\mathbf{M}}{\underbrace{\left[\begin{array}{cccc}{f}_u& 0& c_u& 0\\ 0& f_v& c_v& 0\\ 0& 0& 0& f_{u}b\end{array}\right]}}\frac{1}{z}\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right]$$

This sensor model is just telling us how disparity relates to the position of the point. (left, disparity if we know the geometry we are looking at) (right, disparity that we guess from some form of correspondence algorithm)