ML paradigm

ML paradigm

ML paradigm (not covered in this unit)

• Semi-supervised learning is for data with a mix of labelled and unlabelled data
  • classify text documents
  • fraud detection

Supervised learning

We cover the following methods:

Unsupervised learning

We cover the following methods:

• Dimension reduction:
  • multi-dimensional scaling (MDS)
  • principle component analysis (PCA)

Simple linear regression

Pricing a second-hand 2004 Toyota Yaris car

Regression trees

Pricing a second-hand 2004 Toyota Yaris car

For now don’t worry how this is calculated - we will learn this later.

$k$-nearest neighbour

Pricing a second-hand 2004 Toyota Yaris car

Neural network

Pricing a second-hand 2004 Toyota Yaris car

Comparing models for regression problem

Pricing a second-hand 2004 Toyota Yaris car

• Which model to use? How to choose parameters?
  We’ll come back to these questions later.

Breast cancer diagnosis

• We use the Wisconsin breast cancer data set to build a model to predict if the breast mass sample is malignant (M) or benign (B).
• Here the response is categorical with two classes (M and B).

Features from the digitized image from FNA

• Before fitting any model, we can already see that the average radius and concave portions in the digitized image separate out the two classes.

Logistic regression

Again don’t worry how this is calculated - we will cover this in Week 4.

$k$-nearest neighbour

Neural network

Dimension reduction and clustering

• Data with too many variables is hard to understand or process.
• We can reduce dimension in a meaningful way.
• Or cluster similar groups together to find inherent grouping structure.

We need maths

Assumption for supervised learning

• We assume that there is some fixed, but typically unknown, function $f$ that maps the predictors to response:

$$y = f(x_1, x_2, ..., x_p) + e.$$

More notational matters

• Predictions for
  • $y_i$ are denoted as $\hat{y}_i = f(x_{i1}, x_{i2}, ..., x_{ip})$.
  • $e_i$ are denoted as $\hat{e}_i = y_i - \hat{y}_i$ and referred to as the residual for the $i$-th observation.
• Estimations of model parameters are denoted with $\hat{ }$,
  • e.g., $\hat{\beta}_0$ is an estimate of the intercept in the previous multiplie linear regression model.

Notations for multiple linear regression

Consider a multiple linear regression:

$$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + ... + \beta_p x_{ip} + e_i,\quad\text{for }i = 1, ..., n$$

• $y_i$ is the $i$-th observation of variable $y$,
• $x_{ij}$ is the $i$-th observation of the $j$-th predictor,
• $\beta_0$ is the intercept and $\beta_j$ is the slope or coefficient of $x_j$, and
• $e_i$ is the error term for the $i$-th observation and is often assumed $e_i \sim N(0, \sigma^2)$.

Notice we do not use $i$ when we refer to a generic variable, use $i$ for observations and $j$ for features.

Alternative notations for multiple linear regression

• We can use a summation notation where $x_{i0} = 1$:

$$y_i = \beta_0 + \sum_{j=1}^p\beta_jx_{ij} + e_i = \sum_{j=0}^p\beta_jx_{ij} + e_i,\quad\text{for }i = 1, ..., n$$

Properties of multiple linear regression

Matrix

• A matrix is a rectangular array of scalars (i.e. numbers).

$$\begin{bmatrix}2 & 3 \\4 & 3 \\ 1 & 1\end{bmatrix}\qquad\begin{bmatrix}-3 & 3 \\2 & 9 \end{bmatrix}\qquad\begin{bmatrix}1 & 3 & -1 \\ -4 & 0 & 0\end{bmatrix}$$

• In this unit, a matrix is denoted as a bold capital letter, e.g. $\mathbf{A}$, $\mathbf{X}$ and $\mathbf{B}$.

Dimensions of a matrix

• Occasionally the dimension of the matrix is denoted in the subscript:

$$\mathbf{X}_{n\times p} = \begin{bmatrix}x_{11} & x_{12} & \cdots & x_{1p}\\ x_{21} & x_{22} & \cdots & x_{2p}\\ \vdots & \vdots & \ddots & \vdots\\ x_{n1} & x_{n2} & \cdots & x_{np}\end{bmatrix}$$

Diagonal matrix

• A diagonal matrix is a matrix where off-diagonal entries are zero.

$$\mathbf{D} = \text{diag}(d_{1}, d_{2}, d_{3}, d_4) = \begin{bmatrix} d_{1} & 0 & 0 & 0\\0 & d_{2} & 0& 0\\ 0 & 0 & d_3 & 0 \\ 0 & 0 & 0 & d_4 \end{bmatrix}\qquad \mathbf{I}_3 = \begin{bmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$$

• An identity matrix of dimension $n$ is a $n\times n$ diagonal matrix denoted as $\mathbf{I}_n$ where all the diagonal entries is 1.

Vectors

• A $m$-vector is a $m\times 1$ matrix:

$$\boldsymbol{y} = \begin{bmatrix}y_1 \\ y_2\\ \vdots \\ y_m\end{bmatrix}$$

• In this unit, a vector is denoted with bold italic lower case letter,
  e.g. $\boldsymbol{y}$, $\boldsymbol{x}$, and $\boldsymbol{\beta}$.

Transpose operator

• A transposed matrix or vector is denoted with $\top$

$$\mathbf{X} = \begin{bmatrix}x_{11} & x_{12} & x_{13}\\ x_{21} & x_{22} & x_{23}\end{bmatrix}\qquad \mathbf{X}^\top = \begin{bmatrix}x_{11} & x_{21} \\ x_{12} & x_{22} \\ x_{13} & x_{23} \end{bmatrix}$$

$$\boldsymbol{y} = \begin{bmatrix}y_1\\y_2 \\ \vdots\\ y_n\\\end{bmatrix}\qquad\boldsymbol{y}^\top = \begin{bmatrix}y_1 & y_2 & \cdots & y_n \end{bmatrix}$$

Resources for linear algebra

• See any first year linear algebra textbook.
• Or this linear algebra cheat sheet.

Goodness of fit

Mean square error

$$\begin{align*} MSE = E\left[y-\hat{y}\right]^2 &= E\left[f(x_1, ..., x_p) + e - \hat{f}(x_1, ..., x_p)\right]^2\\ &= \underbrace{E\left[f(x_1, ..., x_p) - \hat{f}(x_1, ..., x_p)\right]^2}_{\text{reducible}} + \underbrace{\text{var}(e)}_{\text{irreducible}} \\ &= \text{bias}[\hat{f}(x)]^2 + \text{var}[\hat{f}(x)] + \text{var}(e) \end{align*}$$

Model bias and variance

• If a model has:
  • a high bias, the average response is far from the true value – referred to as underfitting
  • a high variance, then the fitted model doesn’t generalise well beyond the (training) data – this is referred to as overfitting

Used Toyota car listing

• Let’s have a look at the Toyota car listing.
• Note this data contains other Toyota models and not just Yaris.

library(tidyverse)
toyota <- read_csv("https://emitanaka.org/iml/data/toyota.csv") 
glimpse(toyota)

Rows: 6,738
Columns: 9
$ model        <chr> "GT86", "GT86", "GT86", "GT86", "GT86", "GT86", "GT86", "…
$ year         <dbl> 2016, 2017, 2015, 2017, 2017, 2017, 2017, 2017, 2020, 201…
$ price        <dbl> 16000, 15995, 13998, 18998, 17498, 15998, 18522, 18995, 2…
$ transmission <chr> "Manual", "Manual", "Manual", "Manual", "Manual", "Manual…
$ mileage      <dbl> 24089, 18615, 27469, 14736, 36284, 26919, 10456, 12340, 5…
$ fuelType     <chr> "Petrol", "Petrol", "Petrol", "Petrol", "Petrol", "Petrol…
$ tax          <dbl> 265, 145, 265, 150, 145, 260, 145, 145, 150, 265, 265, 14…
$ mpg          <dbl> 36.2, 36.2, 36.2, 36.2, 36.2, 36.2, 36.2, 36.2, 33.2, 36.…
$ engineSize   <dbl> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, …

A linear regression

• A proposed model is $\log_{10}(\texttt{price}_i) = \beta_0 + \beta_1 \texttt{year}_i + e_i$

ggplot(toyota, aes(year, price)) +
  geom_point() +
  geom_point(data = ~filter(., model == "Yaris"),
             aes(color = "Yaris")) +
  scale_y_log10() +
  labs(color = "")

A linear regression with R

$$\log_{10}(\texttt{price}_i) = \beta_0 + \beta_1 \texttt{year}_i + \epsilon_i$$

• This model is fitted using lm() in R as

fit <- lm(log10(price) ~ year, data = toyota)

10^predict(fit, data.frame(year = 2004))

       1 
2505.726 

Extracting model parameters in R

• You can use the broom package to get the estimate of model parameters for many models

slopes <- broom::tidy(fit)
slopes

Extracting model summaries in R

broom::glance(fit)

Extractng model values in R

broom::augment(fit)

Predicting from model fit with R

• The predict is a generic (S3) function that works for many kind of model objects

10^(slopes$estimate[1] + slopes$estimate[2] * 2004)

[1] 2505.726

10^predict(fit, data.frame(year = 2004))

       1 
2505.726 

Splitting data into testing and training data

• Let’s use rsample to split the data into the training and testing data:

library(rsample)
set.seed(1) # to replicate the result
toyota_split <- initial_split(toyota, prop = 0.75)
toyota_split

<Training/Testing/Total>
<5053/1685/6738>

toyota_train <- training(toyota_split)
toyota_test <- testing(toyota_split)

• This is randomly splitting 75% of data into training data and the remaining 25% into testing data.

Train the ML models

• Now we train three models:
  • reg - a simple linear regression model,
  • tree - a regression tree, and
  • knn - a $k$-nearest neighbour

library(rpart) # for regression tree
library(kknn) # for k-nearest neighbour
model_fits <- list(
  "reg" = lm(log10(price) ~ year,
             data = toyota_train),
  "tree" = rpart(log10(price) ~ year,
                 data = toyota_train,
                 method = "anova"),
  "knn" = train.kknn(log10(price) ~ year,
                     data = toyota_train)
)

Predict response on testing data

• And predict the responses on the testing data:

results_test <- imap_dfr(model_fits, ~{
    toyota_test %>% 
      select(year, price) %>% 
      mutate(.model = .y,
             .pred = 10^predict(.x, .))
  })

head(results_test)results_test <- imap_dfr(model_fits, ~{
    toyota_test %>% 
      select(year, price) %>% 
      mutate(.model = .y,
             .pred = 10^predict(.x, .))
  })

head(results_test)results_test <- imap_dfr(model_fits, ~{
    toyota_test %>% 
      select(year, price) %>% 
      mutate(.model = .y,
             .pred = 10^predict(.x, .))
  })

head(results_test)

Visualising the predictions

results_pred <- imap_dfr(model_fits, ~{
    tibble(year = seq(min(toyota$year), max(toyota$year))) %>% 
      mutate(.pred = 10^predict(.x, .),
             .model = .y)
  })

gres <- ggplot(toyota_train, aes(year, price)) +
  geom_point(alpha = 0.5) + 
  geom_line(data = results_pred, aes(y = .pred, color = .model)) + 
  scale_color_manual(values = c("#C8008F", "#006DAE", "#008A25")) +
  scale_y_log10(label = scales::dollar_format(prefix = "€", accuracy = 1)) +
  theme(legend.position = "bottom") +
  labs(title = "Training data", y = "", color = "")  +
  guides(color = "none")
    

gres

gres %+% toyota_test +
  labs(title = "Testing data")

Predictive accuracies with R

library(yardstick)
results_test %>% 
  group_by(.model) %>% 
  metric_set(rmse, mae, mape, mpe, rsq)(., price, .pred) %>% 
  pivot_wider(.model, names_from = .metric, values_from = .estimate)

Select a model

What is the aim?

It’s not necessary causation

Beware of missing or inappropriate data

Preprocessing the data