Inner products

Properties

Obeys Conjugate symmetry, bilinearity, homogeniety, non negativity, positive definiteness.

Bilinearity: linear in a and b separately: \((\ga a)^{}(\beta b) = \conj{\ga}\beta a^{}b\). (Range of \(\dprod{)\) need not be $\mathrm{\Re }$.}

$⟨Ax+x,y⟩=⟨x,A^{\ast }y+y⟩$.

Orthogonality

If $⟨x,y⟩=0$, $x$ orthogonal to y.

Associated norm

Defines norm ${\parallel x\parallel }^2=⟨x,x⟩$.

$\mathrm{△}$ ineq holds: Take ${\parallel x-y\parallel }^2=⟨x-\text{)}y\text{(},x-y⟩$, expand it, use cauchy schwartz.

2-norm Bound on size

(Cauchy, Schwarz). $|⟨a,b⟩|\le \parallel a\parallel \parallel b\parallel$.

Proof

$0\le f(d)={\parallel u+dv\parallel }^2=⟨u,u⟩-2d⟨u,v⟩+⟨v,v⟩$.

Minimize f(d) wrt $d$ to get: $d=⟨u,v⟩⟨v,v{⟩}^{-1}$.\ So, $0\le ⟨u,u⟩-|⟨u,v⟩|⟨v,v{⟩}^{-1}$.

Tightness

$|⟨a,b⟩|=\parallel a\parallel \parallel b\parallel$ when $⟨a,b⟩=0$.

General norm-bound on size

(Aka Holder’s inequality) For $p,q\ge 1$, $p^{-1}+q^{-1}=1$: $p,q$ are Holder conjugates; then \(|\dprod{a,b}| \leq \norm{a}{p}\norm{b}{q}\) : a tight bound.

Proof

For $p,q>1$, By Young’s ineq, $|a_i{b}_i|\le \frac{|a_i{|}^p}{p}+\frac{|b_i{|}^q}{q}$; \(\frac{1}{\norm{a}{p}\norm{b}{q}}|\dprod{a,b}| \leq p^{-1} + q^{-1} =1\).

Taking the limiting case as $p\to 1$, we also have the $p=1,q=\mathrm{\infty }$ case.

Standard inner product

$⟨a,b⟩=b^{T}a$. Can be generalized to $a,b\in C^m:b^{\ast }a$.

Geometric interpretation

$⟨a,b⟩=b^{T}a=\parallel a\parallel \parallel b\parallel \cos \theta$.

So, orthogonality = perpendicularity.

Proof

Prove for 2 dimensions by seeing: $\left[\begin{array}{c}{a}_1{\text{a}}_2\end{array}\right]=\left[\begin{array}{c}\parallel a\parallel \cos \theta \parallel a\parallel \sin \theta \end{array}\right]$, using $cos(A-B)=\cos A\cos B+\sin A\sin B$.

Consider plane formed by a, b. Get new orthonormal basis Q [\(QQ^{} = I\)], so that $q_1,q_2$ span this plane; so $Q{e}_i=q_i$. The representations of the points a, b wrt this new basis is Qa, Qb; their norm remains same. By the 2 dimensional case, $⟨Qa,Qb⟩=\parallel a\parallel \parallel b\parallel \cos \theta$. But, $⟨Qa,Qb⟩=⟨a,b⟩$ as \(QQ^{}=I\)!

In function spaces

Consider functions with domain $X=[a,b]$, and let $p$ be a probability measure on $X$: ${\int }_X\overline{f(x)}g(x)dx={\int }_a^b\overline{f(x)}g(x)p(x)dx$ for complex valued $f(x)$; can include weight function $W$ too. This defines norm too. Often scaled to make length of certain basis function vectors to be 1.

Orthogonality

Orthogonality of k vectors $⟹$ mutual independence - else contradiction. Orthogonal vector spaces. Orthogonality among bases $⟹$ orthogonality of vector spaces.

Weighted Inner product

Invertible matrix W, \(A=W^{}W\), A +ve semidefinite. Skew vectors before dot product: \(\dprod{a,b}_{W} = \dprod{Wa, Wb} = b^{T}W^{}Wa = b^{T}Aa\). Sometimes writ as \(\dprod{a,b}{A}\). a and b are A conjugate if \(\dprod{a,b}{A}=0\).

Specify inner product using Gram matrix G

Aka Gramian matrix. Take symmetric +ve semidefinite G. $G_{i,j}=⟨x_i,x_j⟩$ for some $\left\{x_i\right\}$; so $G=X^{T}X$ for $X=(x_i)$. Then, for any $u=\sum c_i{x}_i,v=\sum b_i{x}_i$, can rewrite in $\left\{x_i\right\}$ basis as $c$, b and find $⟨.,.⟩$ using $⟨c,b⟩=c^{T}Gb$. As G +ve semi-def, $c^{T}Gc\ge 0$; $c^{T}Gc=0$ iff $c^T{X}^{T}Xc=0$ or Xc=0: meaning of normness preserved.

G often normalized to make $G_{i,i}=1$.

Extension to $\mathrm{\infty }$ dimensions: Mercer’s theorem.