Visualisation II: Further Dimensionality Reduction

Part 1: Linear PCA

Principal Component Analysis

• PCA (Hotelling (1933)) is a linear embedding
• Today its presented as:
  • Rotate to find ‘directions’ in data with maximal variance
  • How do we find these directions?

\[ \mathbf{S}=\frac{1}{n}\sum_{i=1}^n\left(\mathbf{ y}_{i, :}-\boldsymbol{ \mu}\right)\left(\mathbf{ y}_{i, :} - \boldsymbol{ \mu}\right)^\top \]

Principal Component Analysis

• Find directions in the data, \(\mathbf{ z}= \mathbf{U}\mathbf{ y}\), for which variance is maximized.

Lagrangian

• Solution is found via constrained optimisation (which uses Lagrange multipliers): \[ L\left(\mathbf{u}_{1},\lambda_{1}\right)=\mathbf{u}_{1}^{\top}\mathbf{S}\mathbf{u}_{1}+\lambda_{1}\left(1-\mathbf{u}_{1}^{\top}\mathbf{u}_{1}\right) \]

• Gradient with respect to \(\mathbf{u}_{1}\) \[\frac{\text{d}L\left(\mathbf{u}_{1},\lambda_{1}\right)}{\text{d}\mathbf{u}_{1}}=2\mathbf{S}\mathbf{u}_{1}-2\lambda_{1}\mathbf{u}_{1}\]

Lagrangian

• Rearrange to form \[\mathbf{S}\mathbf{u}_{1}=\lambda_{1}\mathbf{u}_{1}.\] Which is known as an eigenvalue problem.
• Further directions that are orthogonal to the first can also be shown to be eigenvectors of the covariance.

PCA directions

Maximum variance directions are eigenvectors of the covariance matrix

Probabilistic PCA

• Represent data, \(\mathbf{Y}\), with a lower dimensional set of latent variables \(\mathbf{Z}\).
• Assume a linear relationship of the form \[ \mathbf{ y}_{i,:}=\mathbf{W}\mathbf{ z}_{i,:}+\boldsymbol{ \epsilon}_{i,:}, \] where \[ \boldsymbol{ \epsilon}_{i,:} \sim \mathscr{N}\left(\mathbf{0},\sigma^2\mathbf{I}\right) \]

Probabilistic PCA

• PPCA defines a probabilistic model where:
  • Data is generated from latent variables through linear transformation
  • Gaussian noise is added to the transformed variables
  • Latent variables have standard Gaussian prior
• Maximum likelihood recovers classical PCA

Graphical model representing probabilistic PCA

Posterior for Principal Component Analysis

\[ p(\mathbf{ z}_{i, :} | \mathbf{ y}_{i, :}) \]

\[ p(\mathbf{ z}_{i, :} | \mathbf{ y}_{i, :}) \propto p(\mathbf{ y}_{i, :}|\mathbf{W}, \mathbf{ z}_{i, :}, \sigma^2) p(\mathbf{ z}_{i, :}) \]

\[ \log p(\mathbf{ z}_{i, :} | \mathbf{ y}_{i, :}) = \log p(\mathbf{ y}_{i, :}|\mathbf{W}, \mathbf{ z}_{i, :}, \sigma^2) + \log p(\mathbf{ z}_{i, :}) + \text{const} \]

\[ \log p(\mathbf{ z}_{i, :} | \mathbf{ y}_{i, :}) = -\frac{1}{2\sigma^2} (\mathbf{ y}_{i, :} - \mathbf{W}\mathbf{ z}_{i, :})^\top(\mathbf{ y}_{i, :} - \mathbf{W}\mathbf{ z}_{i, :}) - \frac{1}{2} \mathbf{ z}_{i, :}^\top \mathbf{ z}_{i, :} + \text{const} \]

Scikit-learn implementation PCA

Oil Flow Data

Examples: Motion Capture Data

OSU Motion Capture Data: Run 1

Part 2: Beyond Linear Methods

When Dimensionality Reduction Fails

• Dimensionality reduction can fail in several key scenarios:
  • Truly high dimensional data with no simpler structure
  • Highly nonlinear relationships between dimensions
  • Varying intrinsic dimensionality across data

The Swiss Roll Example

• Classic example of nonlinear structure
• 2D manifold embedded in 3D
• Linear methods like PCA fail
• Requires nonlinear methods (t-SNE, UMAP)

Common Failure Modes

• Key failure scenarios:
  • Linear methods on nonlinear manifolds
  • Assuming global structure when only local exists
  • Not accounting for noise

Part 3: Graph Based Methods

t-SNE

, \[ where $s_{i,j} = 1$ if two points are linked in the latent space and $q_{i,j}$ is the probability of that link. The latent distances are given by \] {i,j} = ({i, :} - {j, :})^{i, :} - _{j, :}). $$}

= i {j} p_{i,k} $$}

• Converts distances to probabilities
• Uses t-distribution in low-dimensional space
• Excellent for visualization
• Computationally intensive
• Results depend on perplexity parameter

Oil Flow Data

t-SNE Embedding of Oil Flow Data

Part 4: Practical Implementation

Practical Tips for Dimensionality Reduction

Always Start with PCA

• “Do PCA, always”
• Quick sanity check
• Provides baseline
• Helps understand data

Understanding Matrix Structure

• Learn to read Gram matrices
• Check distance matrices
• Look for block structure
• Identify outliers

Understanding Matrix Structure

Oil Flow Data

Gram Matrix of Oil Flow Data

• Learn to read Gram matrices
• Check distance matrices
• Look for block structure
• Identify outliers

Examining Projections

• Look at multiple 2D views
• Check consecutive dimensions
• Examine suspicious patterns
• Plot residuals

Evaluating Neighborhoods

• Check \(k\)-nearest neighbors
• Verify local distances
• Look for disconnected regions
• Examine boundary points

Common Pitfalls

• Not scaling features appropriately
• Ignoring outliers
• Using wrong distance metric
• Trusting single visualization
• Assuming linearity

Validation Strategies

• Check reconstruction error
• Verify known relationships
• Test with held-out data
• Compare multiple methods

Implementation Guidelines

• Start simple
• Scale appropriately
• Check intermediate results
• Validate assumptions
• Document choices