Lagmental Vicfred

Least Squares Projects b onto the Column Space by Vicfred

Last updated: Sun 02 February 2025

tags: mathematics, matrix factorizations, least squares, positive matrices

A least-squares solution makes the residual orthogonal to every column of A. The formulas are more useful when each symbol has a job rather than merely decorating the theorem.

The data

Matrix factorizations expose different geometry: \(A=QR\) separates an orthonormal frame, while \(A=U\Sigma V^\ast\) separates rotations from axis scaling.

$$ x^\ast=\arg\min_x\|Ax-b\|_2^2 $$

The definition determines which expressions are legal; only then does the identity become meaningful. An equality in \(\mathcal A\) may change ambient meaning, so I keep \(\mathsf D\) separate from \(\mathsf C\).

$$ A^\ast(Ax^\ast-b)=0 $$

Derivation

Now evaluate one representative case. The result should agree with the structural law above, but it is obtained without assuming the conclusion.

$$ A=QR,\quad Q^\ast Q=I\Longrightarrow\|Ax-b\|_2^2=\|Rx-Q^\ast b\|_2^2+\|(I-QQ^\ast)b\|_2^2 $$

Invariant content

What survives the example is not its particular numbers but the relation encoded by the two rows below. That relation is the part worth transporting to a new setting.

$$ \begin{aligned} \mathsf{D}\;&:\quad x^\ast=\arg\min_x\|Ax-b\|_2^2,\\[5pt] \mathsf{C}\;&:\quad A^\ast(Ax^\ast-b)=0. \end{aligned} $$

Scope

Conditioning matters numerically. An exact algebraic identity can be a poor computational method when it squares the condition number.

$$ \boxed{\begin{gathered} \text{compact conclusion}\\[-2pt] A^\ast(Ax^\ast-b)=0 \end{gathered}} $$

This is enough machinery for one note: an exact object, a worked case, a structural law, and a clearly marked boundary. Each layer can now be tested independently.

This article was posted on Sat 19 November 2016. Facts and circumstances may have changed since publication.
Please contact me before jumping to conclusions if something seems wrong or unclear.