Toy breast cancer diagnosis data

Infinite hyperplanes

Distance to hyperplane

Margin for hyperplane (a)

Margin for hyperplane (b)

Margin for hyperplane (c)

Maximal margin classifier

Support vectors

• There will always be at least two equidistant vectors vectors to the maximal margin hyperplane.
• These are called support vectors – there are three in this example.

Maximal margin hyperplane

Non-separable case

Support vector classifier

Optimisation

• For support vector classification, we solve the optimisation problem: $$\hat{\boldsymbol{\beta}} = \arg\max_{\boldsymbol{\beta}\in\mathbb{R}^{p+1}}M$$ subject to $\sum_{j=1}^p \beta_j^2 = 1$ and $y_i \times g(\boldsymbol{x}_i) \geq M(1 - \varepsilon_i)$.

Tuning parameter $C$

Nonlinear boundaries

• The support vector classifier is a linear classifier.
• It doesn’t work well for nonlinear boundaries.

Enlarging the feature space

• We can make support vector classifier more flexible by adding higher order polynomial terms, e.g. $x_1^2$ and $x_2^3$.
• We treat these new terms as predictors (referred to as enlarging the feature space).

Support vector machines

The inner product

• Consider two $p$-vectors $$\begin{align*} \boldsymbol{x}_i & = (x_{i1}, x_{i2}, \dots, x_{ip}) \in \mathbb{R}^p \\ \text{and} \quad \boldsymbol{x}_k & = (x_{k1}, x_{k2}, \dots, x_{kp}) \in \mathbb{R}^p. \end{align*}$$
• The inner product is defined as

$$\langle \boldsymbol{x}_i, \boldsymbol{x}_k\rangle = \boldsymbol{x}_i^\top \boldsymbol{x}_k = x_{i1}x_{k1} + x_{i2}x_{k2} + \dots + x_{ip}x_{kp} = \sum_{j=1}^{p} x_{ij}x_{kj}.$$

Dual representation

• In support vector machines, we write $$\beta_j = \sum_{k=1}^n\alpha_ky_kx_{kj}.$$

Dual representation

• We can generalise the expression by replacing the inner product with a kernel function: $$g(\boldsymbol{x}_i) = \beta_0 + \sum_{j=1}^p\beta_jx_{ij} = \beta_0 + \sum_{k=1}^n\alpha_ky_k\color{#006DAE}{\mathcal{K}(\boldsymbol{x}_i, \boldsymbol{x}_k)}.$$
• Under this representation, we don’t need to manually enlarge the feature space.
• Instead we choose a kernel function.

Kernel functions

• A kernel function is an inner product of vectors mapped to a (higher dimensional) feature space $$\mathcal{K}(\boldsymbol{x}_i, \boldsymbol{x}_k) = \langle \psi(\boldsymbol{x}_i), \psi(\boldsymbol{x}_j) \rangle$$ $$\psi: \mathbb{R}^p \rightarrow \mathbb{R}^d$$ where $d > p$.

Examples of kernels

• Standard kernels include: $$\begin{align*} \text{Linear} \quad \mathcal{K}(\boldsymbol{x}_i, \boldsymbol{x}_k) & = \langle\boldsymbol{x}_i, \boldsymbol{x}_k \rangle \\ \text{Polynomial} \quad \mathcal{K}(\boldsymbol{x}, \boldsymbol{y}) & = (\langle\boldsymbol{x}_i, \boldsymbol{x}_k \rangle + 1)^d \\ \text{Radial} \quad \mathcal{K}(\boldsymbol{x}_i, \boldsymbol{x}_k) & = \exp(-\gamma\lvert\lvert\boldsymbol{x}_i-\boldsymbol{x}_k\lvert\lvert^2) \end{align*}$$

Breast cancer diagnosis

library(tidyverse)
cancer <- read_csv("https://emitanaka.org/iml/data/cancer.csv") %>% 
  mutate(diagnosis_malignant = ifelse(diagnosis=="M", 1, 0),
         diagnosis = factor(diagnosis, levels = c("B", "M"))) %>% 
  janitor::clean_names()

cancer %>% 
  ggplot(aes(radius_mean, concave_points_mean)) +
  geom_point(alpha = 0.25, size = 2, aes(color = diagnosis)) +
  scale_color_manual(values = c("forestgreen", "red2")) +
  labs(x = "Average radius",
       y = str_wrap("Average concave portions of the contours", 20),
       color = "Diagnosis")

Support vector machine with R

library(e1071)
svmfitl <- svm(diagnosis ~ radius_mean + concave_points_mean,
              data = cancer,
              kernel = "linear",
              cost = 1,
              scale = FALSE)

• cost is related to $C$
• Large cost tolerates less classification error.
• Small cost tolerates more classification error.

Main output of svm

• svmfitl$index: The indexes of the support vectors.
• svmfitl$coefs: The $y_k\alpha_k$ from the dual representation.
• svmfitl$SV: The support vectors.
• svmfitl$b: Negative intercept $-\hat{\beta}_0$.
• For linear kernels, $\hat{\beta}_k$ can be calculated as t(svmfitl$coefs) %*% svmfitl$SV.

Classification plot for linear kernel

cancer_grid <- expand_grid(radius_mean = seq(min(cancer$radius_mean),
                                             max(cancer$radius_mean),
                                             length.out = 50),
                           concave_points_mean = seq(min(cancer$concave_points_mean),
                                                     max(cancer$concave_points_mean),
                                                     length.out = 50)) %>% 
  mutate(predl = predict(svmfitl, .))

cancer %>% 
  mutate(support_vector = 1:n() %in% svmfitl$index) %>% 
  ggplot(aes(radius_mean, concave_points_mean)) +
  geom_tile(data = cancer_grid, aes(fill = predl), alpha = 0.5) +
  geom_point(alpha = 0.5, size = 3, aes(color = diagnosis, shape = support_vector)) +
  scale_color_manual(values = c("forestgreen", "red2")) +
  labs(x = "Average radius",
       y = str_wrap("Average concave portions of the contours", 20),
       color = "Diagnosis",
       shape = "Support vector",
       fill = "Prediction",
       title = "Linear kernel")

Support vector machine with other kernels

• Polynomial kernel

svmfitp <- svm(diagnosis ~ radius_mean + concave_points_mean,
               data = cancer,
               kernel = "polynomial",
               cost = 1,
               scale = FALSE)

• Radial kernel:

svmfitr <- svm(diagnosis ~ radius_mean + concave_points_mean,
               data = cancer,
               kernel = "radial",
               cost = 1,
               scale = FALSE)

Classification plots for polynomial & radial kernels

library(patchwork)
cancer_grid <- cancer_grid %>% 
  mutate(predp = predict(svmfitp, .),
         predr = predict(svmfitr, .))

p1 <- cancer %>% 
  mutate(support_vector = 1:n() %in% svmfitp$index) %>% 
  ggplot(aes(radius_mean, concave_points_mean)) +
  geom_tile(data = cancer_grid, aes(fill = predp), alpha = 0.5) +
  geom_point(alpha = 0.5, size = 3, aes(color = diagnosis, shape = support_vector)) +
  scale_color_manual(values = c("forestgreen", "red2")) +
  labs(x = "Average radius",
       y = str_wrap("Average concave portions of the contours", 20),
       color = "Diagnosis",
       shape = "Support vector",
       fill = "Prediction",
       title = "Polynomial kernel")

p2 <- cancer %>% 
  mutate(support_vector = 1:n() %in% svmfitr$index) %>% 
  ggplot(aes(radius_mean, concave_points_mean)) +
  geom_tile(data = cancer_grid, aes(fill = predr), alpha = 0.5) +
  geom_point(alpha = 0.5, size = 3, aes(color = diagnosis, shape = support_vector)) +
  scale_color_manual(values = c("forestgreen", "red2")) +
  labs(x = "Average radius",
       y = "",
       color = "Diagnosis",
       shape = "Support vector",
       fill = "Prediction",
       title = "Radial kernel")


p1 + p2 + plot_layout(guides = "collect")

Limitations of support vector machine methods