Joint Gaussian PDFs

Following Multivariate Gaussian, we can also design a joint PDF of two multivariate gaussians.

$$p(\mathbf{x},\mathbf{y})=\mathcal{N}\left(\left[\begin{array}{c}{\mathbf{\mu }}_x\\ {\mathbf{\mu }}_y\end{array}\right],\left[\begin{array}{cc}{\mathbf{\Sigma }}_{xx}& {\mathbf{\Sigma }}_{xy}\\ {\mathbf{\Sigma }}_{yx}& {\mathbf{\Sigma }}_{yy}\end{array}\right]\right)$$ $${\mathbf{\Sigma }}_{xy}={\mathbf{\Sigma }}_{yx}^T$$

Which if expanded looks like

$$\frac{1}{(2\pi {)}^{d/2}{\left[\begin{array}{cc}{\mathbf{\Sigma }}_{xx}& {\mathbf{\Sigma }}_{xy}\\ {\mathbf{\Sigma }}_{yx}& {\mathbf{\Sigma }}_{yy}\end{array}\right]}^{1/2}}\exp \left(-\frac{1}{2}{\left(\left[\begin{array}{c}\mathbf{x}\\ \mathbf{y}\end{array}\right]-\left[\begin{array}{c}{\mathbf{\mu }}_x\\ {\mathbf{\mu }}_y\end{array}\right]\right)}^T{\left[\begin{array}{cc}{\mathbf{\Sigma }}_{xx}& {\mathbf{\Sigma }}_{xy}\\ {\mathbf{\Sigma }}_{yx}& {\mathbf{\Sigma }}_{yy}\end{array}\right]}^{-1}\left(\left[\begin{array}{c}\mathbf{x}\\ \mathbf{y}\end{array}\right]-\left[\begin{array}{c}{\mathbf{\mu }}_x\\ {\mathbf{\mu }}_y\end{array}\right]\right)\right)$$

We can always represent a joint probability as the product of two factors

$$p(\mathbf{x},\mathbf{y})=p(\mathbf{x}|\mathbf{y})p(\mathbf{y})$$

So you can definitely split a Joint Gaussian into something similar. To do so, we need to use something called the Schur Complement

$$\left[\begin{array}{cc}{\mathbf{\Sigma }}_{xx}& {\mathbf{\Sigma }}_{xy}\\ {\mathbf{\Sigma }}_{yx}& {\mathbf{\Sigma }}_{yy}\end{array}\right]=\left[\begin{array}{cc}1& {\mathbf{\Sigma }}_{xy}{\mathbf{\Sigma }}_{yy}^{-1}\\ 0& 1\end{array}\right]\left[\begin{array}{cc}{\mathbf{\Sigma }}_{xx}-{\mathbf{\Sigma }}_{xy}{\mathbf{\Sigma }}_{yy}^{-1}{\mathbf{\Sigma }}_{yx}& 0\\ 0& {\mathbf{\Sigma }}_{yy}\end{array}\right]\left[\begin{array}{cc}1& 0\\ {\mathbf{\Sigma }}_{yy}^{-1}{\mathbf{\Sigma }}_{yx}& 1\end{array}\right]$$

Inverting this we get

$${\left[\begin{array}{cc}{\mathbf{\Sigma }}_{xx}& {\mathbf{\Sigma }}_{xy}\\ {\mathbf{\Sigma }}_{yx}& {\mathbf{\Sigma }}_{yy}\end{array}\right]}^{-1}=\left[\begin{array}{cc}1& 0\\ -{\mathbf{\Sigma }}_{yy}^{-1}{\mathbf{\Sigma }}_{yx}& 1\end{array}\right]\left[\begin{array}{cc}({\mathbf{\Sigma }}_{xx}-{\mathbf{\Sigma }}_{xy}{\mathbf{\Sigma }}_{yy}^{-1}{\mathbf{\Sigma }}_{yx}{)}^{-1}& 0\\ 0& {\mathbf{\Sigma }}_{yy}^{-1}\end{array}\right]\left[\begin{array}{cc}1& -{\mathbf{\Sigma }}_{xy}{\mathbf{\Sigma }}_{yy}^{-1}\\ 0& 1\end{array}\right]$$

If we use this to analyze the quadratic part of the Gaussian PDF…

$${\left(\left[\begin{array}{c}\mathbf{x}\\ \mathbf{y}\end{array}\right]-\left[\begin{array}{c}{\mathbf{\mu }}_x\\ {\mathbf{\mu }}_y\end{array}\right]\right)}^T{\left[\begin{array}{cc}{\mathbf{\Sigma }}_{xx}& {\mathbf{\Sigma }}_{xy}\\ {\mathbf{\Sigma }}_{yx}& {\mathbf{\Sigma }}_{yy}\end{array}\right]}^{-1}\left(\left[\begin{array}{c}\mathbf{x}\\ \mathbf{y}\end{array}\right]-\left[\begin{array}{c}{\mathbf{\mu }}_x\\ {\mathbf{\mu }}_y\end{array}\right]\right)$$ $$={\left(\left[\begin{array}{c}\mathbf{x}\\ \mathbf{y}\end{array}\right]-\left[\begin{array}{c}{\mathbf{\mu }}_x\\ {\mathbf{\mu }}_y\end{array}\right]\right)}^T\left[\begin{array}{cc}1& 0\\ -{\mathbf{\Sigma }}_{yy}^{-1}{\mathbf{\Sigma }}_{yx}& 1\end{array}\right]\left[\begin{array}{cc}({\mathbf{\Sigma }}_{xx}-{\mathbf{\Sigma }}_{xy}{\mathbf{\Sigma }}_{yy}^{-1}{\mathbf{\Sigma }}_{yx}{)}^{-1}& 0\\ 0& {\mathbf{\Sigma }}_{yy}^{-1}\end{array}\right]\left[\begin{array}{cc}1& -{\mathbf{\Sigma }}_{xy}{\mathbf{\Sigma }}_{yy}^{-1}\\ 0& 1\end{array}\right]\left(\left[\begin{array}{c}\mathbf{x}\\ \mathbf{y}\end{array}\right]-\left[\begin{array}{c}{\mathbf{\mu }}_x\\ {\mathbf{\mu }}_y\end{array}\right]\right)$$

$=\underset{p(\mathbf{x}|\mathbf{y})}{\underbrace{(\mathbf{x}-{\mathbf{\mu }}_{\mathbf{x}}-{\mathbf{\Sigma }}_{\mathbf{xy}}{\mathbf{\Sigma }}_{\mathbf{yy}}^{-1}(\mathbf{y}-{\mathbf{\mu }}_{\mathbf{y}}){)}^T({\mathbf{\Sigma }}_{\mathbf{xx}}-{\mathbf{\Sigma }}_{\mathbf{xy}}{\mathbf{\Sigma }}_{\mathbf{yy}}^{-1}{\mathbf{\Sigma }}_{\mathbf{yx}}{)}^{-1}(\mathbf{x}-{\mathbf{\mu }}_{\mathbf{x}}-{\mathbf{\Sigma }}_{\mathbf{xy}}{\mathbf{\Sigma }}_{\mathbf{yy}}^{-1}(\mathbf{y}-{\mathbf{\mu }}_{\mathbf{y}}))}}+\underset{p(\mathbf{y})}{\underbrace{(\mathbf{y}-{\mathbf{\mu }}_{\mathbf{y}}{)}^T{\mathbf{\Sigma }}_{\mathbf{yy}}^{-1}(\mathbf{y}-{\mathbf{\mu }}_{\mathbf{y}})}}$ ^^ Because we are looking at the quadratic part of the Joint Gaussian and addition means multiplication!!

This gets us the following breakdown of the Joint Gaussian Distribution

$$p(\mathbf{x},\mathbf{y})=p(\mathbf{x}|\mathbf{y})p(\mathbf{y})$$ $$p(\mathbf{x}|\mathbf{y})=\mathcal{N}({\mathbf{\mu }}_{\mathbf{x}}-{\mathbf{\Sigma }}_{\mathbf{xy}}{\mathbf{\Sigma }}_{\mathbf{yy}}^{-1}(\mathbf{y}-{\mathbf{\mu }}_{\mathbf{y}}),{\mathbf{\Sigma }}_{\mathbf{xx}}-{\mathbf{\Sigma }}_{\mathbf{xy}}{\mathbf{\Sigma }}_{\mathbf{yy}}^{-1}{\mathbf{\Sigma }}_{\mathbf{yx}})$$ $$p(\mathbf{y})=\mathcal{N}({\mathbf{\mu }}_y,{\mathbf{\Sigma }}_{yy})$$