A theory for optimal regularization in the finite dimensional case

Consider the matrix problem $\text{Ax = y + ε =}\text{y}\text{?}$ in the case where A is known precisely, the problem is ill conditioned, and ε is a random noise vector. Compute regularized “ridge” estimates,$\text{x}\text{?}{}_{\mathrm{\lambda }}\text{= (A}{}^1\text{A + λI)}{}^{-1}\text{A}{}^1\text{y}\text{?}$,where ¹ denotes matrix transpose. Of great concern is the determination of the value of λ for which x?_λ “best” approximates $\text{x}{}_{\mathrm{<mtext>0</mtext>}}\text{= A}{}^{\mathrm{+}}\text{y}$. Let $\text{Q = 6}\text{x}\text{?}{}_{\mathrm{\lambda }}\text{? x}{}_{\mathrm{<mtext>0</mtext>}}\text{6}{}^2$,and define λ₀ to be the value of λ for which Q is a minimum. We look for λ₀ among solutions of dQ/dλ = 0. Though Q is not computable (since ε is unknown), we can use this approach to study the behavior of λ₀ as a function of y and ε. Theorems involving “noise to signal ratios” determine when λ₀ exists and define the cases λ₀ > 0 and λ₀ = ∞. Estimates for λ₀ and the minimum square error Q₀ = Q(λ₀) are derived.