Matrix and Component Forms

This document outlines some basic translations between matrix and component form to allow me to derive things properly.

Basic Matrix-Vector Product

Matrix Form

$$y=Wx$$

Explicit Representation

$$W=\left[\begin{array}{ccc}{W}_{11}& W_{12}& W_{13}\\ W_{21}& W_{22}& W_{23}\\ W_{31}& W_{32}& W_{33}\end{array}\right],\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]$$

Computing Each Element

$$y_1=W_{11}{x}_1+W_{12}{x}_2+W_{13}{x}_3$$ $$y_2=W_{21}{x}_1+W_{22}{x}_2+W_{23}{x}_3$$ $$y_3=W_{31}{x}_1+W_{32}{x}_2+W_{33}{x}_3$$

Component Form

$$y_i=\sum _j{W}_{ij}{x}_j$$

• The index j gets summed out
• The hanging index, i, becomes the index of the output

Matrix-Matrix Product

Matrix Form

$$C=AB$$

Explicit Representation

$$C=\left[\begin{array}{cc}{A}_{11}& A_{12}\\ A_{21}& A_{22}\\ A_{31}& A_{32}\end{array}\right]\left[\begin{array}{ccc}{B}_{11}& B_{12}& B_{13}\\ B_{21}& B_{22}& B_{23}\end{array}\right]$$

Computing Each Element

$$C_{11}=A_{11}{B}_{11}+A_{12}{B}_{21}$$ $$C_{12}=A_{11}{B}_{12}+A_{12}{B}_{22}$$ $$C_{13}=A_{11}{B}_{13}+A_{12}{B}_{23}$$ $$C_{21}=A_{21}{B}_{11}+A_{22}{B}_{21}$$ $$C_{22}=A_{21}{B}_{12}+A_{22}{B}_{22}$$ $$C_{23}=A_{21}{B}_{13}+A_{22}{B}_{23}$$ $$C_{31}=A_{31}{B}_{11}+A_{32}{B}_{21}$$ $$C_{32}=A_{31}{B}_{12}+A_{32}{B}_{22}$$ $$C_{33}=A_{31}{B}_{13}+A_{32}{B}_{23}$$

Component Form

$$C_{i,j}=\sum _k{A}_{i,k}{B}_{k,j}$$

• k gets summed out!!

Inner and Outer Product

$$\text{inner product }c=a^{T}b$$ $$\text{outer product }C=a{b}^T$$

In code, these both return a dot product (same value). You have to explicitly try to turn it into an outer product. a.unsqueeze(1)@b.unsqueeze(0) or torch.outer(a,b)

Component Form

$$\text{inner product (dot product) }c=\sum _i{a}_i{b}_i$$ $$C_{ij}=a_i{b}_j$$

Transpose

Matrix Form

$$B=A^T$$

Explicit

$$A=\left[\begin{array}{ccc}{A}_{11}& A_{12}& A_{13}\\ A_{21}& A_{22}& A_{23}\end{array}\right]$$

Transpose

$$B=A^T=\left[\begin{array}{cc}{A}_{11}& A_{21}\\ A_{12}& A_{22}\\ A_{13}& A_{23}\end{array}\right]$$

Component Form

$$B_{i,j}=A_{j,i}$$

• indexes get flipped

Complex Translations

Matrix Form

$$y=Wx+b$$

Explicit

$$\left(\begin{array}{c}{y}_1\\ y_2\\ \dots \\ y_n\end{array}\right)=\left(\begin{array}{cccc}{W}_{11}& W_{12}& \dots & W_{1m}\\ W_{21}& W_{22}& \dots & W_{2m}\\ W_{31}& W_{32}& \dots & W_{3m}\\ \dots & & & \\ W_{n1}& W_{n2}& \dots & W_{nm}\end{array}\right)\cdot \left(\begin{array}{c}{x}_1\\ x_2\\ x_3\\ \dots \\ x_m\end{array}\right)+\left(\begin{array}{c}{b}_1\\ b_2\\ b_3\\ \dots \\ b_n\end{array}\right)$$

Component Form

$$y_i=\sum W_{i,j}{x}_j+b_i$$

• Notice how j is summed out (and so is x for that manner)

given that X is of shape [b, m] coming in. As a result, we shuffle the linear layer equation a bit to make things smoother.

$$Y=X{W}^T+b$$

This is because $[b,n]=[b,m]@[m,n]+[n]$ and so less code to write.

$$Y_{ij}=\sum _k{X}_{ik}{W}_{kj}+b_j$$

Note that b, the bias is broadcasted.

Quadratic Form (Matrix and Vector)

Matrix Form

$$z=x^{T}Ax$$

Explicit

$$z={\left(\begin{array}{c}{x}_1\\ x_2\\ x_3\\ \dots \\ x_n\end{array}\right)}^T\left(\begin{array}{cccc}{W}_{11}& W_{12}& \dots & W_{1n}\\ W_{21}& W_{22}& \dots & W_{2n}\\ W_{31}& W_{32}& \dots & W_{3n}\\ \dots & & & \\ W_{n1}& W_{n2}& \dots & W_{nn}\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\\ x_3\\ \dots \\ x_n\end{array}\right)$$

Component Form This collapses to a single value.

$$z=\sum _i^n\sum _j^n{x}_i{A}_{i,j}{x}_j$$

Quadratic Form (Matrix and Matrix)

Matrix Form

$$Z=X^{T}AX$$

Explicit

$$\left(\begin{array}{ccc}{Z}_{11}& \dots & Z_{1n}\\ Z_{21}& \dots & Z_{2n}\\ \dots & & \\ Z_{n1}& \dots & Z_{nn}\end{array}\right)={\left(\begin{array}{cccc}{X}_{11}& X_{12}& \dots & X_{1m}\\ X_{21}& X_{22}& \dots & X_{2m}\\ \dots & & & \\ X_{n1}& X_{n2}& \dots & X_{nm}\end{array}\right)}^T\left(\begin{array}{cccc}{W}_{11}& W_{12}& \dots & W_{1m}\\ W_{21}& W_{22}& \dots & W_{2m}\\ W_{31}& W_{32}& \dots & W_{3m}\\ \dots & & & \\ W_{m1}& W_{m2}& \dots & W_{mm}\end{array}\right)\left(\begin{array}{cccc}{X}_{11}& X_{12}& \dots & X_{1m}\\ X_{21}& X_{22}& \dots & X_{2m}\\ \dots & & & \\ X_{n1}& X_{n2}& \dots & X_{nm}\end{array}\right)$$

Component Form Its just the smaller form but its done on every element of Z

$$Z_{pq}=\sum _i^n\sum _j^n{x}_{i,p}{A}_{i,j}{x}_{j,q}$$

• note how i and j get summed out.

Trace

Matrix Form AKA the SUM of the diagonal

$$z=tr(A)$$

Component Form

$$z=\sum _i{A}_{ii}$$

Trace of Product

Matrix Form

$$z=tr(AB)$$

Explicit

$$z=tr\left(\sum _k{A}_{i,k}{B}_{k,j}\right)$$

Two things to note, A and B must both be the same dimensions (one needs to be the transpose of the other.). So A in NxM and B is MxN. I only care about

$$z=tr\left(C_{ii}=\sum _j{A}_{ij}{B}_{ji}\right)$$

I only care about the diagonal so… Component Form

$$z=\sum _i\sum _j{A}_{ij}{B}_{ji}$$

Element-Wise Operations

Matrix Form

$$C=A\odot B$$

Component Form

$$C_{ij}=A_{ij}\cdot B_{ij}$$

• A and B gotta be the same size in theoretical land, in coding, broadcasting can be done

Summary

Operation	Component Form	Explicit Example
$\mathbf{y}=W\mathbf{x}$	$y_i={\sum }_j{W}_{ij}{x}_j$	$y_1=W_{11}{x}_1+W_{12}{x}_2$
$C=AB$	$C_{ik}={\sum }_j{A}_{ij}{B}_{jk}$	$C_{11}=A_{11}{B}_{11}+A_{12}{B}_{21}$
$B=A^T$	$B_{ij}=A_{ji}$	$B_{12}=A_{21}$
$C={\mathbf{ab}}^T$	$C_{ij}=a_i{b}_j$	$C_{12}=a_1{b}_2$
$z={\mathbf{a}}^T\mathbf{b}$	$z={\sum }_i{a}_i{b}_i$	$z=a_1{b}_1+a_2{b}_2$
$C=A\odot B$	$C_{ij}=A_{ij}{B}_{ij}$	$C_{11}=A_{11}{B}_{11}$
$z=x^{T}Ax$	$z={\sum }_i^n{\sum }_j^n{x}_i{A}_{i,j}{x}_j$	$z=x_1{A}_{1,2}{x}_2+x_2{A}_{2,2}{x}_2$
$Z=X^{T}AX$	$z_{p,q}={\sum }_i^n{\sum }_j^n{x}_{i,p}{A}_{i,j}{x}_{j,q}$	$z_{4,3}=x_{1,4}{A}_{1,2}{x}_{2,3}+x_{2,4}{A}_{2,2}{x}_{2,3}$