Objective Functions: A Simple Example with Matrix Factorisation

Objective Function

• Last week we motivated the importance of probability.
• This week we motivate the idea of the ‘objective function.’

The Perceptron

• Developed in 1957 by Rosenblatt.
• Take a data point at, \(\mathbf{ x}_i\).
• Predict class, \(y_i=1\) if \(\sum_jw_{j} \mathbf{ x}_{i, j} + b > 0\)
• Otherwise predict \(y_i=-1\).

Mathematical Drawing of Decision Boundary

• Decision boundary defined by hyperplane
• Classification: \(\text{sign}(\mathbf{ x}^\top \mathbf{ w})\)
• Two features: \(w_1x_{i,1} + w_2x_{i,2} + b\)
• Boundary where prediction switches from -1 to +1

Reminder: Equation of Plane

• Plane equation: \(w_1 x_{i, 1} + w_2 x_{i, 2} + b = 0\)
• Decision boundary where \(\text{sign}(\cdot)\) argument equals zero

Reminder: Equation of Plane

• Rearrange to plot: \(x_2 = -\frac{(b+x_1w_1)}{w_2}\)
• Separating hyperplane divides feature space

Perceptron Algorithm: Initialisation Maths

• Set \(\mathbf{ w}\) with random selected point \(i\) \[ \mathbf{ w}= y_i \mathbf{ x}_i. \]
• Why? Consider \[ \text{sign}(\mathbf{ w}^\top\mathbf{ x}_i) \]

Perceptron Algorithm: Initialisation Maths

• Setting \(\mathbf{ w}\) to \(y_i\mathbf{ x}_i\) implies \[ \text{sign}(\mathbf{ w}^\top\mathbf{ x}_i) = \text{sign}(y_i\mathbf{ x}_i^\top \mathbf{ x}_i) = y_i \]

Drawing Decision Boundary

• Plot \[ x_2 = -\frac{(b+x_1 w_1)}{w_2} \]

or specify \(x_2\) and compute \(x_1\) given the values for \(x_2\), \[ x_1 = -\frac{b + x_2w_2}{w_1} \]

Switching Formulae

• Use of first or second formula
• Depends on how hyperplane leaves plot

Code for Perceptron

Perceptron Reflection

• Algorithm updates are intuitive and interpretable
• What happens when classes aren’t linearly separable?

Perceptron Reflections

• Non-convergence indicates inseparable data
• Possible fix: anneal learning rate
• Non-linear extensions: basis functions, kernel methods, multi-layer networks

The Objective Function

• Perceptron algorithm lacks explicit objective function
• Updates provide intuition but not clear optimization target
• Objective functions (loss/error/cost) often easier starting point
• Connection between algorithm and objective not immediately obvious

Objective Functions and Regression

• Classification: map feature to class label.
• Regression: map feature to real value

Regression

• Our prediction function is \[f(x_i) = mx_i + c\]

• Need an algorithm to fit it.

Least Squares

• Least squares: minimise an error.

\[E(m, c) = \sum_{i=1}^n(y_i - f(x_i))^2\]

Regression

• Create an artifical data set.

• true value for \(m\) m_true = 1.4

• true value for \(c\) c_true = -3.1

We can use these values to create our artificial data. The formula \[y_i = mx_i + c\] is translated to code as follows:

y = m_true*x+c_true

Plot of Data

We can now plot the artifical data we’ve created.

Plot of data

• Points lie exactly on a straight line
• Not very realistic
• Corrupt them with Gaussian ‘noise.’

Noise Corrupted Plot

Contour Plot of Error Function

Steepest Descent

• Minimise the sum of squares error function.
• E.g. gradient descent.
• Initialise with a guess for \(m\) and \(c\)
• update that guess by subtracting a portion of the gradient from the guess.
• Like walking down a hill in the steepest direction of the hill to get to the bottom.

Algorithm

• We start with a guess for \(m\) and \(c\).

m_star = 0.0
c_star = -5.0

Offset Gradient

• Gradient of the error wrt \(c\), \[ \frac{\text{d}E(m, c)}{\text{d} c} = -2\sum_{i=1}^n(y_i - mx_i - c) \]

Compute as

c_grad = -2*(y-m_star*x - c_star).sum()

Slope Gradient

Gradient wrt \(m\) is similar \[ \frac{\text{d}E(m, c)}{\text{d} m} = -2\sum_{i=1}^nx_i(y_i - mx_i - c) \]

Compute as

which can be implemented in python (numpy) as

m_grad = -2*(x*(y-m_star*x - c_star)).sum()

Update Equations

• Gradients with respect to \(m\) and \(c\).
• Can update our inital guesses for \(m\) and \(c\) using the gradient.

Update Equations

• We don’t want to just subtract the gradient from \(m\) and \(c\),
• We need to take a small step in the gradient direction.
• Otherwise we might overshoot the minimum.

Update Equations

• We want to follow the gradient to get to the minimum, the gradient changes all the time.

Move in Direction of Gradient

Update Equations

• The step size has already been introduced
• it’s known as the learning rate and is denoted by \(\eta\). \[ c_\text{new}\leftarrow c_{\text{old}} - \eta\frac{\text{d}E(m, c)}{\text{d}c} \]

Step Size

• gives us an update for our estimate of \(c\) and \[ m_\text{new} \leftarrow m_{\text{old}} - \eta\frac{\text{d}E(m, c)}{\text{d}m} \]
• Giving us an update for \(m\).

Update Code

• These updates can be coded as

learn_rate = 0.01
c_star = c_star - learn_rate*c_grad
m_star = m_star - learn_rate*m_grad

Iterating Updates

• Fit model by descending gradient.

Gradient Descent Algorithm

Stochastic Gradient Descent

• If \(n\) is small, gradient descent is fine.
• But sometimes (e.g. on the internet) \(n\) could be a billion.
• Stochastic gradient descent is more similar to perceptron.
• Look at gradient of one data point at a time rather than summing across all data points)
• This gives a stochastic estimate of gradient.

Stochastic Gradient Descent

• The real gradient with respect to \(m\) is given by \[ \frac{\text{d}E(m, c)}{\text{d} m} = -2\sum_{i=1}^nx_i(y_i - mx_i - c) \]

Decompose the Sum

This could be split up into lots of individual updates \[m_1 \leftarrow m_\text{old} + 2\eta\left[x_1 (y_1 - m_\text{old}x_1 - c_\text{old})\right]\] \[m_2 \leftarrow m_1 + 2\eta\left[x_2 (y_2 - m_\text{old}x_2 - c_\text{old})\right]\] \[m_3 \leftarrow m_2 + 2\eta \left[\dots\right]\] \[m_n \leftarrow m_{n-1} + 2\eta\left[x_n (y_n - m_\text{old}x_n - c_\text{old})\right]\]

which would lead to the same final update.

Updating \(c\) and \(m\)

• In the sum we don’t change \(m\) and \(c\) we use for computing the gradient term at each update.
• In stochastic gradient descent we do change them.
• This means it’s not quite the same as steepest descent.

• But we can present each data point in a random order, like we did for the perceptron.
• This makes the algorithm suitable for large-scale data use

Stochastic Gradient Descent

• Since the data is presented in a random order we write \[ m_\text{new} = m_\text{old} + 2\eta\left[x_i (y_i - m_\text{old}x_i - c_\text{old})\right] \]

SGD for Linear Regression

Putting it all together in an algorithm, we can do stochastic gradient descent for our regression data.

Mini-Batch

• In practice use mini-batches.
  • Instead of computing gradient of one example.
  • Use a small batch of examples.

Reflection on Linear Regression and Supervised Learning

• Learning rate size
• Smaller data sets and variation in results

Lab Class

• You will take the ideas you have learnt.
• You will apply them in the domain of matrix factorisation.
• Matrix factorization presents a different error function.

for loss functions.

Movie Body Count Data Example

Movie Violence Data

• Data containing movie information (year, length, rating, genre, IMDB Rating).
• Try and predict what factors influence a movie’s violence

Plotting the Data

Recommender Systems

Inner Products for Representing Similarity

The Library on an Infinite Plane

\[ \mathbf{v}_j = \begin{bmatrix} v_{j,1} \\ v_{j,2}\end{bmatrix}, \]

\[ \mathbf{u}_i = \begin{bmatrix} u_{i,1} \\ u_{i,2}\end{bmatrix}. \]

Collecting the Data

Processing the Data

Measuring Similarity

\[ \mathbf{a}^\top\mathbf{b} = \sum_{i} a_i b_i \]

\[ \mathbf{a}^\top\mathbf{b} = |\mathbf{a}||\mathbf{b}| \cos(\theta) \]

Objective Function

The error function (or objective function, or cost function) we will choose is known as ‘sum of squares,’ we will aim to minimize the sum of squared squared error between the inner product of \(\mathbf{u}_i\) and \(\mathbf{v}_i\) and the observed score for the user/item pairing, given by \(y_{i, j}\).

The total objective function can be written as \[ E(\mathbf{U}, \mathbf{V}) = \sum_{i,j} s_{i,j} (y_{i,j} - \mathbf{u}_i^\top \mathbf{v}_j)^2 \] where \(s_{i,j}\) is an indicator variable that is 1 if user \(i\) has rated item \(j\) and is zero otherwise. Here \(\mathbf{U}\) is the matrix made up of all the vectors \(\mathbf{u}\), \[ \mathbf{U} = \begin{bmatrix} \mathbf{u}_1 \dots \mathbf{u}_n\end{bmatrix}^\top \] where we note that \(i\)th row of \(\mathbf{U}\) contains the vector associated with the \(i\)th user and \(n\) is the total number of users. This form of matrix is known as a design matrix. Similarly, we define the matrix \[ \mathbf{V} = \begin{bmatrix} \mathbf{v}_1 \dots \mathbf{v}_m\end{bmatrix}^\top \] where again the \(j\)th row of \(\mathbf{V}\) contains the vector associated with the \(j\)th item and \(m\) is the total number of items in the data set.

Objective Optimization

The idea is to mimimize this objective. A standard, simple, technique for minimizing an objective is gradient descent or steepest descent. In gradient descent we simply choose to update each parameter in the model by subtracting a multiple of the objective function’s gradient with respect to the parameters. So for a parameter \(u_{i,j}\) from the matrix \(\mathbf{U}\) we would have an update as follows: \[ u_{k,\ell} \leftarrow u_{k,\ell} - \eta \frac{\text{d} E(\mathbf{U}, \mathbf{V})}{\text{d}u_{k,\ell}} \] where \(\eta\) (which is pronounced eta in English) is a Greek letter representing the learning rate.

We can compute the gradient of the objective function with respect to \(u_{k,\ell}\) as \[ \frac{\text{d}E(\mathbf{U}, \mathbf{V})}{\text{d}u_{k,\ell}} = -2 \sum_j s_{k,j}v_{j,\ell}(y_{k, j} - \mathbf{u}_k^\top\mathbf{v}_{j}). \] Similarly each parameter \(v_{i,j}\) needs to be updated according to its gradient.

Steepest Descent Algorithm

Initialisation

Stochastic Gradient Descent or Robbins Monroe Algorithm

{In we saw how stochastic gradient descent (Robbins and Monro, 1951) involves updating separating each gradient update according to each separate observation, rather than summing over them all. It is an approximate optimization method, but it has proven convergence under certain conditions and can be much faster in practice. It is used widely by internet companies for doing machine learning in practice. For example, Facebook’s ad ranking algorithm uses stochastic gradient descent.

Making Predictions

Predictions can be made from the model of the appropriate rating for a given user, \(i\), for a given film, \(j\), by simply taking the inner product between their vectors \(\mathbf{u}_i\) and \(\mathbf{v}_j\).

Is Our Map Enough? Are Our Data Enough?

Is two dimensions really enough to capture the complexity of humans and their artforms? Perhaps we need even more dimensions to capture that complexity. Extending our books analogy further, consider how we should place books that have a historical timeframe as well as some geographical location. Do we really want books from the 2nd World War to sit alongside books from the Roman Empire? Books on the American invasion of Sicily in 1943 are perhaps less related to books about Carthage than those that study the Jewish Revolt from 66-70 (in the Roman Province of Judaea). So books that relate to subjects which are closer in time should be stored together. However, a student of rebellion against empire may also be interested in the relationship between the Jewish Revolt of 66-70 and the Indian Rebellion of 1857, nearly 1800 years later. Whilst the technologies are different, the psychology of the people is shared: a rebellious nation angainst their imperial masters, triggered by misrule with a religious and cultural background. To capture such complexities we would need further dimensions in our latent representation. But are further dimensions justified by the amount of data we have? Can we really understand the facets of a film that only has at most three or four ratings?

Going Further

If you want to take this model further then you’ll need more data. One possible source of data is the movielens data set. They have data sets containing up to ten million movie ratings. The few ratings we were able to collect in the class are not enough to capture the rich structure underlying these films. Imagine if we assume that the ratings are uniformly distributed between 1 and 5. If you know something about information theory then you could use that to work out the maximum number of bits of information we could gain per rating.

Now we’ll download the movielens 100k data and see if we can extract information about these movies.