🧚🏽‍♀️Abstract Linear Algebra I Unit 8 – Inner Products and Orthogonality

Key Concepts and Definitions

• Inner product is a generalization of the dot product that allows us to define angles and lengths in abstract vector spaces
• Orthogonality refers to two vectors being perpendicular or at right angles to each other
  • Orthogonal vectors have an inner product of zero
• Norm of a vector is a measure of its length or magnitude in a vector space
  • Induced by the inner product as $\sqrt{\langle v, v \rangle}$
• Orthonormal set is a collection of vectors that are both orthogonal to each other and have unit norm (length 1)
• Orthogonal projection is the process of finding the closest point in a subspace to a given vector
  • Useful for approximating solutions and minimizing errors
• Gram-Schmidt process is an algorithm for constructing an orthonormal basis from a linearly independent set of vectors
• Cauchy-Schwarz inequality states that the absolute value of the inner product of two vectors is less than or equal to the product of their norms
  • $|\langle u, v \rangle| \leq \|u\| \|v\|$

Inner Product Spaces

• An inner product space is a vector space equipped with an inner product operation
• Inner product is a function that takes two vectors and returns a scalar value
  • Denoted as $\langle u, v \rangle$ for vectors $u$ and $v$
• Inner product spaces allow us to define geometric concepts like angles, lengths, and orthogonality in abstract vector spaces
• Examples of inner product spaces include:
  • Euclidean space $\mathbb{R}^n$ with the dot product
  • Space of continuous functions $C[a, b]$ with the integral inner product $\langle f, g \rangle = \int_a^b f(x)g(x) dx$
• Inner product spaces have a rich structure and satisfy several important properties
  • Symmetry, linearity, and positive-definiteness
• Many concepts from Euclidean geometry can be generalized to inner product spaces
  • Orthogonal projections, Gram-Schmidt process, and least-squares approximations

Properties of Inner Products

• Symmetry: $\langle u, v \rangle = \langle v, u \rangle$ for all vectors $u$ and $v$
• Linearity in the first argument:
  • $\langle au, v \rangle = a\langle u, v \rangle$ for any scalar $a$
  • $\langle u_1 + u_2, v \rangle = \langle u_1, v \rangle + \langle u_2, v \rangle$ for any vectors $u_1$, $u_2$, and $v$
• Positive-definiteness: $\langle v, v \rangle \geq 0$ for all vectors $v$, with equality if and only if $v = 0$
• Cauchy-Schwarz inequality: $|\langle u, v \rangle| \leq \|u\| \|v\|$ for all vectors $u$ and $v$
  • Equality holds if and only if $u$ and $v$ are linearly dependent
• Norm induced by the inner product: $\|v\| = \sqrt{\langle v, v \rangle}$
  • Satisfies the properties of a norm (non-negativity, homogeneity, and triangle inequality)
• Parallelogram law: $\|u + v\|^2 + \|u - v\|^2 = 2(\|u\|^2 + \|v\|^2)$ for all vectors $u$ and $v$

Orthogonality and Orthonormal Sets

• Two vectors $u$ and $v$ are orthogonal if their inner product is zero: $\langle u, v \rangle = 0$
  • Orthogonal vectors are perpendicular or at right angles to each other
• An orthogonal set is a collection of non-zero vectors that are pairwise orthogonal
  • $\langle u_i, u_j \rangle = 0$ for all $i \neq j$
• Orthonormal set is an orthogonal set where each vector has unit norm (length 1)
  • $\|u_i\| = 1$ for all vectors $u_i$ in the set
• Orthonormal sets are particularly useful as bases for inner product spaces
  • Coefficients of a vector with respect to an orthonormal basis are easily computed using inner products
• Orthonormal bases simplify many computations and have desirable numerical properties
  • Minimize roundoff errors and provide a natural coordinate system
• Examples of orthonormal sets include:
  • Standard basis vectors $\{e_1, e_2, \ldots, e_n\}$ in $\mathbb{R}^n$
  • Trigonometric functions $\{\frac{1}{\sqrt{2\pi}}, \frac{1}{\sqrt{\pi}}\cos(nx), \frac{1}{\sqrt{\pi}}\sin(nx)\}$ in $L^2[-\pi, \pi]$

Gram-Schmidt Process

• Gram-Schmidt process is an algorithm for constructing an orthonormal basis from a linearly independent set of vectors
• Takes a linearly independent set $\{v_1, v_2, \ldots, v_n\}$ and produces an orthonormal set $\{u_1, u_2, \ldots, u_n\}$
• The process works by iteratively orthogonalizing and normalizing the vectors
  1. Set $u_1 = \frac{v_1}{\|v_1\|}$
  2. For $i = 2, \ldots, n$:
     • Compute the projection of $v_i$ onto the subspace spanned by $\{u_1, \ldots, u_{i-1}\}$:
       • $proj_i = \sum_{j=1}^{i-1} \langle v_i, u_j \rangle u_j$
     • Subtract the projection from $v_i$ to obtain the orthogonal component:
       • $u_i' = v_i - proj_i$
     • Normalize $u_i'$ to obtain the orthonormal vector:
       • $u_i = \frac{u_i'}{\|u_i'\|}$
• The resulting set $\{u_1, u_2, \ldots, u_n\}$ is an orthonormal basis for the subspace spanned by the original vectors
• Gram-Schmidt process is widely used in numerical linear algebra and has applications in:
  • Least-squares approximations
  • QR factorization
  • Solving systems of linear equations

Orthogonal Projections

• Orthogonal projection is the process of finding the closest point in a subspace to a given vector
• Given a subspace $W$ and a vector $v$, the orthogonal projection of $v$ onto $W$ is the unique vector $proj_W(v)$ in $W$ that minimizes the distance to $v$
  • $proj_W(v) = \arg\min_{w \in W} \|v - w\|$
• Orthogonal projection can be computed using an orthonormal basis $\{u_1, \ldots, u_k\}$ for the subspace $W$:
  • $proj_W(v) = \sum_{i=1}^k \langle v, u_i \rangle u_i$
• Properties of orthogonal projections:
  • $proj_W(v)$ is the unique vector in $W$ such that $v - proj_W(v)$ is orthogonal to every vector in $W$
  • $proj_W$ is a linear transformation
  • $proj_W(v) = v$ if and only if $v \in W$
  • $\|v - proj_W(v)\| \leq \|v - w\|$ for all $w \in W$
• Orthogonal projections have numerous applications, including:
  • Least-squares approximations and regression analysis
  • Signal and image processing (denoising, compression)
  • Solving systems of linear equations and optimization problems

Applications in Linear Algebra

• Inner products and orthogonality have a wide range of applications in linear algebra and related fields
• Least-squares approximations:
  • Finding the best approximation of a vector in a subspace
  • Minimizing the sum of squared errors between data points and a model
• Orthogonal diagonalization of symmetric matrices:
  • Eigenvectors of a symmetric matrix form an orthonormal basis
  • Allows for efficient computation of matrix powers and exponentials
• Principal component analysis (PCA):
  • Identifying the directions of maximum variance in a dataset
  • Useful for dimensionality reduction and data visualization
• Quantum mechanics:
  • State vectors in a Hilbert space (an infinite-dimensional inner product space)
  • Observables represented by Hermitian operators with orthogonal eigenvectors
• Fourier analysis and signal processing:
  • Representing functions as linear combinations of orthogonal basis functions (e.g., trigonometric functions, wavelets)
  • Analyzing and filtering signals in the frequency domain

Practice Problems and Examples

1. Verify that the following functions define an inner product on the vector space of continuous functions $C[0, 1]$:
   • $\langle f, g \rangle = \int_0^1 f(x)g(x) dx$
   • $\langle f, g \rangle = f(0)g(0) + \int_0^1 f'(x)g'(x) dx$
2. Compute the orthogonal projection of the vector $v = (1, 2, 3)$ onto the subspace $W = \text{span}\{(1, 1, 1), (1, 0, -1)\}$ in $\mathbb{R}^3$.
3. Apply the Gram-Schmidt process to the following set of vectors in $\mathbb{R}^4$:
   • $v_1 = (1, 0, 0, 0)$, $v_2 = (1, 1, 0, 0)$, $v_3 = (1, 1, 1, 0)$, $v_4 = (1, 1, 1, 1)$
4. Prove that if $\{u_1, \ldots, u_n\}$ is an orthonormal set in an inner product space $V$, then for any vector $v \in V$:
   • $\|v\|^2 = \sum_{i=1}^n |\langle v, u_i \rangle|^2 + \|v - \sum_{i=1}^n \langle v, u_i \rangle u_i\|^2$
5. Find the closest point in the plane $x + y + z = 1$ to the point $(2, 3, 4)$ in $\mathbb{R}^3$.
6. Determine whether the following sets of vectors are orthogonal, orthonormal, or neither:
   • $\{(1, 1, 0), (1, -1, 0), (0, 0, 1)\}$ in $\mathbb{R}^3$
   • $\{(1, 0, 1), (0, 1, 1), (-1, 1, 0)\}$ in $\mathbb{R}^3$
   • $\{\sin x, \cos x\}$ in $L^2[-\pi, \pi]$
7. Compute the Fourier coefficients of the function $f(x) = x^2$ on the interval $[-\pi, \pi]$ with respect to the orthonormal basis $\{\frac{1}{\sqrt{2\pi}}, \frac{1}{\sqrt{\pi}}\cos(nx), \frac{1}{\sqrt{\pi}}\sin(nx)\}$.