Logistic Regression and GLMs

Review

• Last week: Specified Class Conditional Distributions, \(p(\mathbf{ x}_i|y_i, \boldsymbol{ \theta})\).
• Used Bayes Classifier + naive Bayes model to specify joint distribution.
• Used Bayes rule to compute posterior probability of class membership.
• This week:
  • direct estimation of probability of class membership.
  • introduction of generalised linear models.

Logistic Regression

• Model the log odds with a linear model.

Logistic Regression

• In regression we model probability of \(y_i |\mathbf{ x}_i\) directly
• Allows less flexibility in questions, but more flexibility in model assumptions

Log Odds

• Model the log-odds with basis functions
• Odds are ratio of probability of positive vs negative outcome

Log Odds

• Probability is between zero and one
• Odds are: \[ \frac{\pi}{1-\pi} \]

Log Odds

• Odds are between \(0\) and \(\infty\)
• Logarithm of odds maps them to \(-\infty\) to \(\infty\)

Logit Link Function - 1

• The Logit function is our link function
• \[g(\pi_i) = \log\frac{\pi_i}{1-\pi_i}\]

Logit Link Function - 2

• For standard regression:
• \[f(\mathbf{ x}_i) = \mathbf{ w}^\top \boldsymbol{ \phi}(\mathbf{ x}_i)\]

Logistic function - 3

• Logistic (or sigmoid) squashes real line to between 0 & 1. Sometimes also called a ‘squashing function.’

Logistic funciton - 4

• For classification (logistic regression): \[\log \frac{\pi_i}{1-\pi_i} = \mathbf{ w}^\top \boldsymbol{ \phi}(\mathbf{ x}_i)\]

Soft Arg Max (Softmax) Function

• Softmax turns unnormalised scores into probabilities
• Invariant to adding a constant to all scores

Prediction Function

Prediction Function

• Can write \(\pi\) as a function of input, parameters, \[\pi(\mathbf{ x},\mathbf{ w}) = \frac{1}{1+ \exp\left(-\mathbf{ w}^\top \boldsymbol{ \phi}(\mathbf{ x})\right)}.\]
• Compute the output \(\mathbf{ w}^\top \boldsymbol{ \phi}(\mathbf{ x})\)
• Apply inverse link function, \(h(\mathbf{ w}^\top \boldsymbol{ \phi}(\mathbf{ x}))\).
• Use this value in a Bernoulli distribution to form the likelihood.

Bernoulli Reminder

• From last time \[P(y_i|\mathbf{ w}, \mathbf{ x}) = \pi_i^{y_i} (1-\pi_i)^{1-y_i}\]

• Trick for switching betwen probabilities

def bernoulli(y, pi):
    if y == 1:
        return pi
    else:
        return 1-pi

Maximum Likelihood

• Conditional independence of data: \[P(\mathbf{ y}|\mathbf{ w}, \mathbf{X}) = \prod_{i=1}^nP(y_i|\mathbf{ w}, \mathbf{ x}_i). \]

Negative Log Likelihood

Objective Function

• Probability of positive outcome for the \(i\)th data point \[\pi_i = h\left(\mathbf{ w}^\top \boldsymbol{ \phi}(\mathbf{ x}_i)\right),\] where \(h(\cdot)\) is the inverse link function
• Objective function of the form \[\begin{align*} E(\mathbf{ w}) = & - \sum_{i=1}^ny_i \log h\left(\mathbf{ w}^\top \boldsymbol{ \phi}(\mathbf{ x}_i)\right) \\& - \sum_{i=1}^n(1-y_i)\log \left(1-h\left(\mathbf{ w}^\top \boldsymbol{ \phi}(\mathbf{ x}_i)\right)\right). \end{align*}\]

Minimise Objective

• Grdient wrt \(\pi(\mathbf{ x};\mathbf{ w})\) \[\begin{aligned} \frac{\text{d}E(\mathbf{ w})}{\text{d}\mathbf{ w}} = & -\sum_{i=1}^n\frac{y_i}{h\left(\mathbf{ w}^\top \boldsymbol{ \phi}(\mathbf{ x})\right)}\frac{\text{d}h(f_i)}{\text{d}f_i} \boldsymbol{ \phi}(\mathbf{ x}_i) \\ & + \sum_{i=1}^n \frac{1-y_i}{1-h\left(\mathbf{ w}^\top \boldsymbol{ \phi}(\mathbf{ x})\right)}\frac{\text{d}h(f_i)}{\text{d}f_i} \boldsymbol{ \phi}(\mathbf{ x}_i) \end{aligned}\]

Link Function Gradient

• Also need gradient of inverse link function wrt parameters. \[\begin{aligned} h(f_i) &= \frac{1}{1+\exp(-f_i)}\\ &=(1+\exp(-f_i))^{-1} \end{aligned}\]

Link Function Gradient

Objective Gradient

Objective Gradient

Multiplying everything out leads to \[ \frac{\text{d}E(\mathbf{ w})}{\text{d}\mathbf{ w}} = -\sum_{i=1}^n\left(y_i - h\left(\mathbf{ w}^\top \boldsymbol{ \phi}(\mathbf{ x}_i)\right)\right)\boldsymbol{ \phi}(\mathbf{ x}_i). \]

Optimisation of the Function

• Can’t find a stationary point of the objective function analytically.
• Optimisation has to proceed by numerical methods.
  • Newton’s method or
  • gradient based optimisation methods
  • Iterative Reweighted Least Squares
• Similarly to matrix factorisation, for large data stochastic gradient descent (Robbins Munro (Robbins and Monro, 1951) optimisation procedure) works well.

Ad Matching for Facebook

• This approach used in many internet companies.
• Example: ad matching for Facebook.
  • Millions of advertisers
  • Billions of users
  • How do you choose who to show what?
• Logistic regression used in combination with decision trees
• Paper available here

Going Further: Optimization

Other optimization techniques for generalized linear models include Newton’s method, it requires you to compute the Hessian, or second derivative of the objective function.

Methods that are based on gradients only include L-BFGS and conjugate gradients. Can you find these in python? Are they suitable for very large data sets? }

Bayesian Approaches