Visualisation II: Further Dimensionality Reduction

Part 1: Linear PCA

Principal Component Analysis

• PCA (Hotelling (1933)) is a linear embedding
• Today its presented as:
  • Rotate to find ‘directions’ in data with maximal variance
  • How do we find these directions?

$$\mathbf{S}=\frac{1}{n}\sum_{i=1}^n\left(\mathbf{ y}_{i, :}-\boldsymbol{ \mu}\right)\left(\mathbf{ y}_{i, :} - \boldsymbol{ \mu}\right)^\top$$

Principal Component Analysis

• Find directions in the data, $\mathbf{ z}= \mathbf{U}\mathbf{ y}$, for which variance is maximized.

Lagrangian

• Solution is found via constrained optimisation (which uses Lagrange multipliers): $$L\left(\mathbf{u}_{1},\lambda_{1}\right)=\mathbf{u}_{1}^{\top}\mathbf{S}\mathbf{u}_{1}+\lambda_{1}\left(1-\mathbf{u}_{1}^{\top}\mathbf{u}_{1}\right)$$

• Gradient with respect to $\mathbf{u}_{1}$ $$\frac{\text{d}L\left(\mathbf{u}_{1},\lambda_{1}\right)}{\text{d}\mathbf{u}_{1}}=2\mathbf{S}\mathbf{u}_{1}-2\lambda_{1}\mathbf{u}_{1}$$ rearrange to form $$\mathbf{S}\mathbf{u}_{1}=\lambda_{1}\mathbf{u}_{1}.$$ Which is known as an eigenvalue problem.

• Further directions that are orthogonal to the first can also be shown to be eigenvectors of the covariance.

PCA directions

❮ ❯

Maximum variance directions are eigenvectors of the covariance matrix

Probabilistic PCA

• Represent data, $\mathbf{Y}$, with a lower dimensional set of latent variables $\mathbf{Z}$.

• Assume a linear relationship of the form $$\mathbf{ y}_{i,:}=\mathbf{W}\mathbf{ z}_{i,:}+\boldsymbol{ \epsilon}_{i,:},$$ where $$\boldsymbol{ \epsilon}_{i,:} \sim \mathcal{N}\left(\mathbf{0},\sigma^2\mathbf{I}\right)$$

• PPCA defines a probabilistic model where:

  • Data is generated from latent variables through linear transformation
  • Gaussian noise is added to the transformed variables
  • Latent variables have standard Gaussian prior

• Maximum likelihood recovers classical PCA

Graphical model representing probabilistic PCA

Probabilistic PCA

$$\boldsymbol{\Sigma} = \sigma^2 \mathbf{I}.$$

$$p(\mathbf{Y}|\mathbf{W}, \sigma^2) = \prod_{i=1}^n\mathcal{N}\left(\mathbf{ y}_{i, :}|\mathbf{0},\mathbf{W}\mathbf{W}^\top + \sigma^2 \mathbf{I}\right)$$

$$\mathbf{W}= \mathbf{U}\mathbf{L} \mathbf{R}^\top$$

$$\mathbf{S} = \sum_{i=1}^n(\mathbf{ y}_{i, :} - \boldsymbol{ \mu})(\mathbf{ y}_{i,:} - \boldsymbol{ \mu})^\top,$$

$$\ell_i = \sqrt{\lambda_i - \sigma^2}$$

PPCA as Manifold Learning

Posterior for Principal Component Analysis

$$p(\mathbf{ z}_{i, :} | \mathbf{ y}_{i, :})$$

$$p(\mathbf{ z}_{i, :} | \mathbf{ y}_{i, :}) \propto p(\mathbf{ y}_{i, :}|\mathbf{W}, \mathbf{ z}_{i, :}, \sigma^2) p(\mathbf{ z}_{i, :})$$

$$\log p(\mathbf{ z}_{i, :} | \mathbf{ y}_{i, :}) = \log p(\mathbf{ y}_{i, :}|\mathbf{W}, \mathbf{ z}_{i, :}, \sigma^2) + \log p(\mathbf{ z}_{i, :}) + \text{const}$$

$$\log p(\mathbf{ z}_{i, :} | \mathbf{ y}_{i, :}) = -\frac{1}{2\sigma^2} (\mathbf{ y}_{i, :} - \mathbf{W}\mathbf{ z}_{i, :})^\top(\mathbf{ y}_{i, :} - \mathbf{W}\mathbf{ z}_{i, :}) - \frac{1}{2} \mathbf{ z}_{i, :}^\top \mathbf{ z}_{i, :} + \text{const}$$

Scikit-learn implementation PCA

Oil Flow Data

mu_x, C_x = posterior(Y, W, sigma2)

where mu_x and C_x are the posterior mean and posterior covariance for the given $\mathbf{Y}$.

Don’t forget to subtract the mean of the data Y inside your function before computing the posterior: remember we assumed at the beginning of our analysis that the data had been centred (i.e. the mean was removed).}{# Answer Code # Write code for you answer to this exercise in this box # Do not delete these comments, otherwise you will get zero for this answer. # Make sure your code has run and the answer is correct before submitting your notebook for marking. import numpy as np import scipy as sp def posterior(Y, W, sigma2): Y_cent = Y - Y.mean(0) # Compute posterior over X C_x = mu_x = return mu_x, C_x}{20}}