Projection

qIf Ax = b has no solution, we can solve the closest problem $A\hat{x}=p$ by projecting $b$ onto $C(A)$. The Projection matrix saves this process in memory $Pb=p$. The projection guide vector $\vec{e}$ from the tip of $b$ to $p$, is the “error” of the projection $e=b-p$ and always orthogonal to $A$, $A^T\underbrace{(b-A\hat{x})}\_p=A\hat{x}=0$, therefore lies in $N(A^T)$ and $\perp C(A)$. The General Projection Formula summarizes the central equation $A^{T}A=A^{T}b$

Projection Workflow

1. Find $\hat{x}$
2. Find $p$
3. Find $P$ $$\begin{array}{c}(A^{T}A)\hat{x}=A^{T}b\\ p=A\hat{x}\\ P=A(A^{T}A{)}^{-1}{A}^T\end{array}$$

Projections with Orthonormal Matrices Q and Gram-Schmidt

Orthonormal matrices $Q$ simplify the projection significantly due to the property $Q^{T}Q=I$. The formulas become:

$$\begin{array}{ccc}(Q^{T}Q)\hat{x}=Q^{T}b& \to & \hat{x}=Q^{T}b\\ p=Q\hat{x}& \to & p=Q\hat{x}\\ P=A(Q^{T}Q{)}^{-1}{Q}^T& \to & P=Q{Q}^T\end{array}$$

Other

The error of rewriting the inverse

Only for square, invertible matrices the part $(A^{T}A{)}^{-1}$ can be rewritten, cancelling with the rest such that it yields $P=I$. Because such matrix $m\times n\text{in}{R}^n$ is the whole $R^n$ space, the projection of any $b$ in it is just $b$ itself, thus $P=I$.

Properties of the Projection Matrix

• $P=P^2$ Projecting a vector by the same projection matrix is idempotent
• $P^T=P$ Symmetry comes from $A^{T}A$ terms.

1D-Projection

For one dimensional projection the same formulas apply but with vectors. This is a simplification

$$\hat{x}=\frac{a^{T}b}{a^{T}a}$$ $$proj\_a(b)=a\hat{x}$$ $$P=\frac{a{a}^T}{a^{T}a}$$