Neural network II - table of contents


  • ETC3250/5250
    • Neural network II
  • Acknowledgement
  • Wide and deep neural networks
    • Neural network layers
    • Wide neural network
    • Deep neural network
  • Feed forward neural networks
    • Mathematical notation
    • Forward evaluation
    • Forward evaluation
    • Forward evaluation
    • Forward evaluation
    • Forward evaluation
    • Forward evaluation
    • Forward evaluation
    • Forward evaluation
    • Forward evaluation
    • Forward evaluation
    • Forward evaluation
    • General feed forward neural network
  • Calibration
    • Calibration
    • Regression loss
    • Binary classification loss
    • Multi-class classification loss
    • Optimisation
    • Illustrative example
    • Finding the zero slope
    • Optimising a multi-variable function
    • Searching for the optimal value
    • Illustrative example: Step 0
    • Illustrative example: Step 1
    • Illustrative example: Step 2
    • Illustrative example: Step 3
    • Illustrative example: Step 4
    • Illustrative example: Step 20
    • Gradient descent
    • Limitations of gradient descent
  • Stochastic gradient descent
    • Stochastic gradient descent
    • Batch illustration
    • Epoch
    • Epoch
    • Epoch
    • Epoch
    • Epoch
    • Epoch
    • Epoch
    • Backpropagation
    • Vanishing gradient problem
  • An application to fashion MNIST data
    • Fashion MNIST data
    • Neural network in R 1 prepare the data
    • Neural network in R 2 Define architechture
    • Neural network in R 3 Choose loss function
    • Neural network in R 4 Fit model
    • Model diagnostic
    • Predicting from neural network
    • Visually checking the prediction: observation 1
    • Visually checking the prediction: observation 2
    • Visually checking the prediction: observation 3
    • Visually checking the prediction: observation 4
    • Visually checking the prediction: observation 5
  • Limitations
    • Cons of neural network
  • Takeaways


ETC3250/5250

Introduction to Machine Learning

Neural network II

Lecturer: Emi Tanaka

Department of Econometrics and Business Statistics

Acknowledgement

This lecture benefited from the lecture notes by Dr. Ruben Loaiza-Maya.

Wide and deep neural networks

Neural network layers

  • The neural network we saw last week had 3 layers:
    • Input layer (2 nodes)
    • Middle or hidden layer (3 nodes)
    • Output layer (3 nodes)
  • The number of nodes in the input and output layer are constrained by the input data and desired output (regression or classification).
gr1 cluster0 Input layer cluster1 Hidden layer cluster2 Output layer A1 1 B1 ReLU A1->B1 B2 ReLU A1->B2 B3 ReLU A1->B3 A2 Income A2->B1 A2->B2 A2->B3 A3 A3->B3 C1 Softmax (0 = cheap) B1->C1 C2 Softmax (1 = average) B1->C2 C3 Softmax (2 = expensive) B1->C3 B2->C1 B2->C2 B2->C3 B3->C1 B3->C2 B3->C3

Wide neural network

  • We refer to neural network with large number of neurons (nodes) in the hidden layer as a wide neural network.
  • The number of parameters in the neural network model generally increases exponentially as the number of neurons increase.
  • Wide neural networks are however harder to calibrate as the number of neurons increase.
gr2 cluster0 Input layer cluster1 Hidden layer cluster2 Output layer A1 1 B1 ReLU A1->B1 B2 ReLU A1->B2 B3 ReLU A1->B3 B4 ReLU A1->B4 B5 ReLU A1->B5 A2 x₁ A2->B1 A2->B2 A2->B3 A2->B4 A2->B5 A3 A3->B3 A4 A4->B4 A5 A5->B5 C1 Softmax B1->C1 C2 Softmax B1->C2 C3 Softmax B1->C3 B2->C1 B2->C2 B2->C3 B3->C1 B3->C2 B3->C3 B3->C3 B4->C1 B4->C2 B4->C3 C4 B4->C4 B5->C1 B5->C2 B5->C3 C5 B5->C5

Here, the addition of 2 neurons increases the number of parameters by 10.

Deep neural network

  • Deep neural networks, also called feed forward neural networks, add more layers.
  • The number of parameters also can increase exponentially.
  • But it can achieve flexibility with less neuron and it can be faster to calibrate.
gr3 cluster0 Input layer cluster1 Hidden layer 1 cluster2 Hidden layer 2 cluster3 Output layer A1 1 B1 ReLU A1->B1 B2 ReLU A1->B2 B3 ReLU A1->B3 A2 x₁ A2->B1 A2->B2 A2->B3 nodeAI1 C1 Tanh B1->C1 C2 Tanh B1->C2 B2->C1 B2->C2 B3->C1 B3->C2 nodeCI0 D1 Softmax C1->D1 D2 Softmax C1->D2 D3 Softmax C1->D3 C2->D1 C2->D2 C2->D3 nodeCI1

Feed forward neural networks

Mathematical notation

  • We now use a new mathematical notation for neural networks:
    • LLL denotes the total number of layers,
    • ak(l)a^{(l)}_kak(l)​ denotes the value for layer lll for kkk-th node,
    • KlK_lKl​ denotes the total number of nodes in layer lll,
    • hlh_lhl​ denotes the activation function in layer lll,
    • bk(l)b_k^{(l)}bk(l)​ denotes the bias in node kkk of layer lll,
    • wk(l)\boldsymbol{w}_k^{(l)}wk(l)​ denotes the weights for kkk-th node of layer lll.

Forward evaluation

gr4 cluster0 Input layer cluster1 ReLU layer cluster2 Tanh layer cluster3 Output layer A1 1 B1 A1->B1 B2 A1->B2 B3 A1->B3 A2 x₁ A2->B1 β₁ A2->B2 A2->B3 A3 A3->B1 A3->B2 A3->B3 C1 B1->C1 C2 B1->C2 C3 B1->C3 B2->C1 w₁⁽³⁾ B2->C2 B2->C3 B3->C1 B3->C2 B3->C3 D1 C1->D1 D2 C1->D2 D3 C1->D3 C2->D1 C2->D2 w₁⁽³⁾ C2->D3 C3->D3

Input layer

x=(1,x1)⊤\boldsymbol{x} = (1, x_1)^\topx=(1,x1​)⊤

Forward evaluation

gr5 cluster0 Input layer cluster1 ReLU layer cluster2 Tanh layer cluster3 Output layer A1 1 B1 a₁⁽²⁾ A1->B1 B2 A1->B2 B3 A1->B3 A2 x₁ A2->B1 β₁ A2->B2 A2->B3 A3 A3->B1 A3->B2 A3->B3 C1 B1->C1 C2 B1->C2 C3 B1->C3 B2->C1 w₁⁽³⁾ B2->C2 B2->C3 B3->C1 B3->C2 B3->C3 D1 C1->D1 D2 C1->D2 D3 C1->D3 C2->D1 w₁⁽³⁾ C2->D2 C2->D3 C3->D3

Input layer

x=(1,x1)⊤\boldsymbol{x} = (1, x_1)^\topx=(1,x1​)⊤

Layer 2

  • The activation function h2h_2h2​ is ReLU.
  • a1(2)=h2(β1⊤x)a^{(2)}_1 = h_2(\boldsymbol{\beta}_1^\top\boldsymbol{x})a1(2)​=h2​(β1⊤​x)

Forward evaluation

gr6 cluster0 Input layer cluster1 ReLU layer cluster2 Tanh layer cluster3 Output layer A1 1 B1 a₁⁽²⁾ A1->B1 B2 a₂⁽²⁾ A1->B2 B3 a₃⁽²⁾ A1->B3 A2 x₁ A2->B1 β₁ A2->B2 β₂ A2->B2 A2->B3 A3 A3->B1 A3->B2 A3->B3 C1 B1->C1 C2 B1->C2 C3 B1->C3 B2->C1 w₁⁽³⁾ B2->C2 B2->C3 B3->C1 B3->C2 B3->C3 D1 C1->D1 D2 C1->D2 D3 C1->D3 C2->D1 w₁⁽³⁾ C2->D2 C2->D3 C3->D3

Input layer

x=(1,x1)⊤\boldsymbol{x} = (1, x_1)^\topx=(1,x1​)⊤

Layer 2

  • The activation function h2h_2h2​ is ReLU.
  • a1(2)=h2(β1⊤x)a^{(2)}_1 = h_2(\boldsymbol{\beta}_1^\top\boldsymbol{x})a1(2)​=h2​(β1⊤​x), a2(2)=h2(β2⊤x)a^{(2)}_2 = h_2(\boldsymbol{\beta}_2^\top\boldsymbol{x})a2(2)​=h2​(β2⊤​x)

Forward evaluation

gr7 cluster0 Input layer cluster1 ReLU layer cluster2 Tanh layer cluster3 Output layer A1 1 B1 a₁⁽²⁾ A1->B1 B2 a₂⁽²⁾ A1->B2 B3 a₃⁽²⁾ A1->B3 A2 x₁ A2->B1 β₁ A2->B2 β₂ A2->B2 A2->B3 β₃ A3 A3->B1 A3->B2 A3->B3 C1 B1->C1 C2 B1->C2 C3 B1->C3 B2->C1 w₁⁽³⁾ B2->C2 B2->C3 B3->C1 B3->C2 B3->C3 D1 C1->D1 D2 C1->D2 D3 C1->D3 C2->D1 w₁⁽³⁾ C2->D2 C2->D3 C3->D3

Input layer

x=(1,x1)⊤\boldsymbol{x} = (1, x_1)^\topx=(1,x1​)⊤

Layer 2

  • The activation function h2h_2h2​ is ReLU.
  • a1(2)=h2(β1⊤x)a^{(2)}_1 = h_2(\boldsymbol{\beta}_1^\top\boldsymbol{x})a1(2)​=h2​(β1⊤​x), a2(2)=h2(β2⊤x)a^{(2)}_2 = h_2(\boldsymbol{\beta}_2^\top\boldsymbol{x})a2(2)​=h2​(β2⊤​x), a3(2)=h2(β3⊤x)a^{(2)}_3 = h_2(\boldsymbol{\beta}_3^\top\boldsymbol{x})a3(2)​=h2​(β3⊤​x)

Forward evaluation

gr8 cluster0 Input layer cluster1 ReLU layer cluster2 Tanh layer cluster3 Output layer A1 1 B1 a₁⁽²⁾ A1->B1 B2 a₂⁽²⁾ A1->B2 B3 a₃⁽²⁾ A1->B3 A2 x₁ A2->B1 β₁ A2->B2 β₂ A2->B2 A2->B3 β₃ A3 A3->B1 A3->B2 A3->B3 C1 B1->C1 C2 B1->C2 C3 B1->C3 B2->C1 w₁⁽³⁾ B2->C2 B2->C3 B3->C1 B3->C2 B3->C3 D1 C1->D1 D2 C1->D2 D3 C1->D3 C2->D1 w₁⁽³⁾ C2->D2 C2->D3 C3->D3

Input layer

x=(1,x1)⊤\boldsymbol{x} = (1, x_1)^\topx=(1,x1​)⊤

Layer 2

  • The activation function h2h_2h2​ is ReLU.
  • a1(2)=h2(β1⊤x)a^{(2)}_1 = h_2(\boldsymbol{\beta}_1^\top\boldsymbol{x})a1(2)​=h2​(β1⊤​x), a2(2)=h2(β2⊤x)a^{(2)}_2 = h_2(\boldsymbol{\beta}_2^\top\boldsymbol{x})a2(2)​=h2​(β2⊤​x), a3(2)=h2(β3⊤x)a^{(2)}_3 = h_2(\boldsymbol{\beta}_3^\top\boldsymbol{x})a3(2)​=h2​(β3⊤​x)
  • a(2)=(a1(2),a2(2),a3(2))⊤\boldsymbol{a}^{(2)} = (a^{(2)}_1, a^{(2)}_2, a^{(2)}_3)^\topa(2)=(a1(2)​,a2(2)​,a3(2)​)⊤

Forward evaluation

gr9 cluster0 Input layer cluster1 ReLU layer cluster2 Tanh layer cluster3 Output layer A1 1 B1 a₁⁽²⁾ A1->B1 B2 a₂⁽²⁾ A1->B2 B3 a₃⁽²⁾ A1->B3 A2 x₁ A2->B1 β₁ A2->B2 β₂ A2->B2 A2->B3 β₃ A3 A3->B1 A3->B2 A3->B3 C1 a₁⁽³⁾ b₁⁽³⁾ B1->C1 C2 B1->C2 C3 B1->C3 B2->C1 w₁⁽³⁾ B2->C2 B2->C3 B3->C1 B3->C2 B3->C3 D1 C1->D1 D2 C1->D2 D3 C1->D3 C2->D1 w₁⁽³⁾ C2->D2 C2->D3 C3->D3

Input layer

x=(1,x1)⊤\boldsymbol{x} = (1, x_1)^\topx=(1,x1​)⊤

Layer 2

  • The activation function h2h_2h2​ is ReLU.
  • a1(2)=h2(β1⊤x)a^{(2)}_1 = h_2(\boldsymbol{\beta}_1^\top\boldsymbol{x})a1(2)​=h2​(β1⊤​x), a2(2)=h2(β2⊤x)a^{(2)}_2 = h_2(\boldsymbol{\beta}_2^\top\boldsymbol{x})a2(2)​=h2​(β2⊤​x), a3(2)=h2(β3⊤x)a^{(2)}_3 = h_2(\boldsymbol{\beta}_3^\top\boldsymbol{x})a3(2)​=h2​(β3⊤​x)
  • a(2)=(a1(2),a2(2),a3(2))⊤\boldsymbol{a}^{(2)} = (a^{(2)}_1, a^{(2)}_2, a^{(2)}_3)^\topa(2)=(a1(2)​,a2(2)​,a3(2)​)⊤

Layer 3

  • The activation function h3h_3h3​ is Tanh.
  • a1(3)=h2(b1(3)+w1(3)⊤a(2))a^{(3)}_1 = h_2(b_1^{(3)}+\boldsymbol{w}_1^{(3)\top}\boldsymbol{a}^{(2)})a1(3)​=h2​(b1(3)​+w1(3)⊤​a(2))

Forward evaluation

gr10 cluster0 Input layer cluster1 ReLU layer cluster2 Tanh layer cluster3 Output layer A1 1 B1 a₁⁽²⁾ A1->B1 B2 a₂⁽²⁾ A1->B2 B3 a₃⁽²⁾ A1->B3 A2 x₁ A2->B1 β₁ A2->B2 β₂ A2->B2 A2->B3 β₃ A3 A3->B1 A3->B2 A3->B3 C1 a₁⁽³⁾ b₁⁽³⁾ B1->C1 C2 a₂⁽³⁾ b₂⁽³⁾ B1->C2 C3 B1->C3 B2->C1 w₁⁽³⁾ B2->C2 w₂⁽³⁾ B2->C3 B3->C1 B3->C2 B3->C3 D1 C1->D1 D2 C1->D2 D3 C1->D3 C2->D1 w₁⁽³⁾ C2->D2 C2->D3 C3->D3

Input layer

x=(1,x1)⊤\boldsymbol{x} = (1, x_1)^\topx=(1,x1​)⊤

Layer 2

  • The activation function h2h_2h2​ is ReLU.
  • a1(2)=h2(β1⊤x)a^{(2)}_1 = h_2(\boldsymbol{\beta}_1^\top\boldsymbol{x})a1(2)​=h2​(β1⊤​x), a2(2)=h2(β2⊤x)a^{(2)}_2 = h_2(\boldsymbol{\beta}_2^\top\boldsymbol{x})a2(2)​=h2​(β2⊤​x), a3(2)=h2(β3⊤x)a^{(2)}_3 = h_2(\boldsymbol{\beta}_3^\top\boldsymbol{x})a3(2)​=h2​(β3⊤​x)
  • a(2)=(a1(2),a2(2),a3(2))⊤\boldsymbol{a}^{(2)} = (a^{(2)}_1, a^{(2)}_2, a^{(2)}_3)^\topa(2)=(a1(2)​,a2(2)​,a3(2)​)⊤

Layer 3

  • The activation function h3h_3h3​ is Tanh.
  • a1(3)=h3(b1(3)+w1(3)⊤a(2))a^{(3)}_1 = h_3(b_1^{(3)}+\boldsymbol{w}_1^{(3)\top}\boldsymbol{a}^{(2)})a1(3)​=h3​(b1(3)​+w1(3)⊤​a(2)), a2(3)=h3(b2(3)+w2(3)⊤a(2))a^{(3)}_2 = h_3(b_2^{(3)}+\boldsymbol{w}_2^{(3)\top}\boldsymbol{a}^{(2)})a2(3)​=h3​(b2(3)​+w2(3)⊤​a(2))

Forward evaluation

gr11 cluster0 Input layer cluster1 ReLU layer cluster2 Tanh layer cluster3 Output layer A1 1 B1 a₁⁽²⁾ A1->B1 B2 a₂⁽²⁾ A1->B2 B3 a₃⁽²⁾ A1->B3 A2 x₁ A2->B1 β₁ A2->B2 β₂ A2->B2 A2->B3 β₃ A3 A3->B1 A3->B2 A3->B3 C1 a₁⁽³⁾ b₁⁽³⁾ B1->C1 C2 a₂⁽³⁾ b₂⁽³⁾ B1->C2 C3 B1->C3 B2->C1 w₁⁽³⁾ B2->C2 w₂⁽³⁾ B2->C3 B3->C1 B3->C2 B3->C3 D1 C1->D1 D2 C1->D2 D3 C1->D3 C2->D1 w₁⁽³⁾ C2->D2 C2->D3 C3->D3

Input layer

x=(1,x1)⊤\boldsymbol{x} = (1, x_1)^\topx=(1,x1​)⊤

Layer 2

  • The activation function h2h_2h2​ is ReLU.
  • a1(2)=h2(β1⊤x)a^{(2)}_1 = h_2(\boldsymbol{\beta}_1^\top\boldsymbol{x})a1(2)​=h2​(β1⊤​x), a2(2)=h2(β2⊤x)a^{(2)}_2 = h_2(\boldsymbol{\beta}_2^\top\boldsymbol{x})a2(2)​=h2​(β2⊤​x), a3(2)=h2(β3⊤x)a^{(2)}_3 = h_2(\boldsymbol{\beta}_3^\top\boldsymbol{x})a3(2)​=h2​(β3⊤​x)
  • a(2)=(a1(2),a2(2),a3(2))⊤\boldsymbol{a}^{(2)} = (a^{(2)}_1, a^{(2)}_2, a^{(2)}_3)^\topa(2)=(a1(2)​,a2(2)​,a3(2)​)⊤

Layer 3

  • The activation function h3h_3h3​ is Tanh.
  • a1(3)=h3(b1(3)+w1(3)⊤a(2))a^{(3)}_1 = h_3(b_1^{(3)}+\boldsymbol{w}_1^{(3)\top}\boldsymbol{a}^{(2)})a1(3)​=h3​(b1(3)​+w1(3)⊤​a(2)), a2(3)=h3(b2(3)+w2(3)⊤a(2))a^{(3)}_2 = h_3(b_2^{(3)}+\boldsymbol{w}_2^{(3)\top}\boldsymbol{a}^{(2)})a2(3)​=h3​(b2(3)​+w2(3)⊤​a(2))
  • a(3)=(a1(3),a2(3))⊤\boldsymbol{a}^{(3)} = (a^{(3)}_1, a^{(3)}_2)^\topa(3)=(a1(3)​,a2(3)​)⊤

Forward evaluation

gr12 cluster0 Input layer cluster1 ReLU layer cluster2 Tanh layer cluster3 Output layer A1 1 B1 a₁⁽²⁾ A1->B1 B2 a₂⁽²⁾ A1->B2 B3 a₃⁽²⁾ A1->B3 A2 x₁ A2->B1 β₁ A2->B2 β₂ A2->B2 A2->B3 β₃ A3 A3->B1 A3->B2 A3->B3 C1 a₁⁽³⁾ b₁⁽³⁾ B1->C1 C2 a₂⁽³⁾ b₂⁽³⁾ B1->C2 C3 B1->C3 B2->C1 w₁⁽³⁾ B2->C2 w₂⁽³⁾ B2->C3 B3->C1 B3->C2 B3->C3 D1 f₁(x) b₁⁽⁴⁾ C1->D1 D2 C1->D2 D3 C1->D3 C2->D1 w₁⁽⁴⁾ C2->D2 C2->D3 C3->D3

Input layer

x=(1,x1)⊤\boldsymbol{x} = (1, x_1)^\topx=(1,x1​)⊤

Layer 2

  • The activation function h2h_2h2​ is ReLU.
  • a1(2)=h2(β1⊤x)a^{(2)}_1 = h_2(\boldsymbol{\beta}_1^\top\boldsymbol{x})a1(2)​=h2​(β1⊤​x), a2(2)=h2(β2⊤x)a^{(2)}_2 = h_2(\boldsymbol{\beta}_2^\top\boldsymbol{x})a2(2)​=h2​(β2⊤​x), a3(2)=h2(β3⊤x)a^{(2)}_3 = h_2(\boldsymbol{\beta}_3^\top\boldsymbol{x})a3(2)​=h2​(β3⊤​x)
  • a(2)=(a1(2),a2(2),a3(2))⊤\boldsymbol{a}^{(2)} = (a^{(2)}_1, a^{(2)}_2, a^{(2)}_3)^\topa(2)=(a1(2)​,a2(2)​,a3(2)​)⊤

Layer 3

  • The activation function h3h_3h3​ is Tanh.
  • a1(3)=h3(b1(3)+w1(3)⊤a(2))a^{(3)}_1 = h_3(b_1^{(3)}+\boldsymbol{w}_1^{(3)\top}\boldsymbol{a}^{(2)})a1(3)​=h3​(b1(3)​+w1(3)⊤​a(2)), a2(3)=h3(b2(3)+w2(3)⊤a(2))a^{(3)}_2 = h_3(b_2^{(3)}+\boldsymbol{w}_2^{(3)\top}\boldsymbol{a}^{(2)})a2(3)​=h3​(b2(3)​+w2(3)⊤​a(2))
  • a(3)=(a1(3),a2(3))⊤\boldsymbol{a}^{(3)} = (a^{(3)}_1, a^{(3)}_2)^\topa(3)=(a1(3)​,a2(3)​)⊤

Output layer

  • The activation function h4h_4h4​ is Softmax.
  • f1(x)=h4(b1(4)+w1(4)⊤a(3))f_1(\boldsymbol{x}) = h_4(b_1^{(4)}+\boldsymbol{w}_1^{(4)\top}\boldsymbol{a}^{(3)})f1​(x)=h4​(b1(4)​+w1(4)⊤​a(3))

Forward evaluation

gr13 cluster0 Input layer cluster1 ReLU layer cluster2 Tanh layer cluster3 Output layer A1 1 B1 a₁⁽²⁾ A1->B1 B2 a₂⁽²⁾ A1->B2 B3 a₃⁽²⁾ A1->B3 A2 x₁ A2->B1 β₁ A2->B2 β₂ A2->B2 A2->B3 β₃ A3 A3->B1 A3->B2 A3->B3 C1 a₁⁽³⁾ b₁⁽³⁾ B1->C1 C2 a₂⁽³⁾ b₂⁽³⁾ B1->C2 C3 B1->C3 B2->C1 w₁⁽³⁾ B2->C2 w₂⁽³⁾ B2->C3 B3->C1 B3->C2 B3->C3 D1 f₁(x) b₁⁽⁴⁾ C1->D1 D2 f₂(x) b₁⁽⁴⁾ C1->D2 D3 C1->D3 C2->D1 w₁⁽⁴⁾ C2->D2 w₂⁽⁴⁾ C2->D3 C3->D3

Input layer

x=(1,x1)⊤\boldsymbol{x} = (1, x_1)^\topx=(1,x1​)⊤

Layer 2

  • The activation function h2h_2h2​ is ReLU.
  • a1(2)=h2(β1⊤x)a^{(2)}_1 = h_2(\boldsymbol{\beta}_1^\top\boldsymbol{x})a1(2)​=h2​(β1⊤​x), a2(2)=h2(β2⊤x)a^{(2)}_2 = h_2(\boldsymbol{\beta}_2^\top\boldsymbol{x})a2(2)​=h2​(β2⊤​x), a3(2)=h2(β3⊤x)a^{(2)}_3 = h_2(\boldsymbol{\beta}_3^\top\boldsymbol{x})a3(2)​=h2​(β3⊤​x)
  • a(2)=(a1(2),a2(2),a3(2))⊤\boldsymbol{a}^{(2)} = (a^{(2)}_1, a^{(2)}_2, a^{(2)}_3)^\topa(2)=(a1(2)​,a2(2)​,a3(2)​)⊤

Layer 3

  • The activation function h3h_3h3​ is Tanh.
  • a1(3)=h3(b1(3)+w1(3)⊤a(2))a^{(3)}_1 = h_3(b_1^{(3)}+\boldsymbol{w}_1^{(3)\top}\boldsymbol{a}^{(2)})a1(3)​=h3​(b1(3)​+w1(3)⊤​a(2)), a2(3)=h3(b2(3)+w2(3)⊤a(2))a^{(3)}_2 = h_3(b_2^{(3)}+\boldsymbol{w}_2^{(3)\top}\boldsymbol{a}^{(2)})a2(3)​=h3​(b2(3)​+w2(3)⊤​a(2))
  • a(3)=(a1(3),a2(3))⊤\boldsymbol{a}^{(3)} = (a^{(3)}_1, a^{(3)}_2)^\topa(3)=(a1(3)​,a2(3)​)⊤

Output layer

  • The activation function h4h_4h4​ is Softmax.
  • f1(x)=h4(b1(4)+w1(4)⊤a(3))f_1(\boldsymbol{x}) = h_4(b_1^{(4)}+\boldsymbol{w}_1^{(4)\top}\boldsymbol{a}^{(3)})f1​(x)=h4​(b1(4)​+w1(4)⊤​a(3)), f2(x)=h4(b2(4)+w2(4)⊤a(3))f_2(\boldsymbol{x}) = h_4(b_2^{(4)}+\boldsymbol{w}_2^{(4)\top}\boldsymbol{a}^{(3)})f2​(x)=h4​(b2(4)​+w2(4)⊤​a(3))

Forward evaluation

gr14 cluster0 Input layer cluster1 ReLU layer cluster2 Tanh layer cluster3 Output layer A1 1 B1 a₁⁽²⁾ A1->B1 B2 a₂⁽²⁾ A1->B2 B3 a₃⁽²⁾ A1->B3 A2 x₁ A2->B1 β₁ A2->B2 β₂ A2->B2 A2->B3 β₃ A3 A3->B1 A3->B2 A3->B3 C1 a₁⁽³⁾ b₁⁽³⁾ B1->C1 C2 a₂⁽³⁾ b₂⁽³⁾ B1->C2 C3 B1->C3 B2->C1 w₁⁽³⁾ B2->C2 w₂⁽³⁾ B2->C3 B3->C1 B3->C2 B3->C3 D1 f₁(x) b₁⁽⁴⁾ C1->D1 D2 f₂(x) b₁⁽⁴⁾ C1->D2 D3 f₃(x) b₁⁽⁴⁾ C1->D3 C2->D1 w₁⁽⁴⁾ C2->D2 w₂⁽⁴⁾ C2->D3 w₃⁽⁴⁾ C3->D3

Input layer

x=(1,x1)⊤\boldsymbol{x} = (1, x_1)^\topx=(1,x1​)⊤

Layer 2

  • The activation function h2h_2h2​ is ReLU.
  • a1(2)=h2(β1⊤x)a^{(2)}_1 = h_2(\boldsymbol{\beta}_1^\top\boldsymbol{x})a1(2)​=h2​(β1⊤​x), a2(2)=h2(β2⊤x)a^{(2)}_2 = h_2(\boldsymbol{\beta}_2^\top\boldsymbol{x})a2(2)​=h2​(β2⊤​x), a3(2)=h2(β3⊤x)a^{(2)}_3 = h_2(\boldsymbol{\beta}_3^\top\boldsymbol{x})a3(2)​=h2​(β3⊤​x)
  • a(2)=(a1(2),a2(2),a3(2))⊤\boldsymbol{a}^{(2)} = (a^{(2)}_1, a^{(2)}_2, a^{(2)}_3)^\topa(2)=(a1(2)​,a2(2)​,a3(2)​)⊤

Layer 3

  • The activation function h3h_3h3​ is Tanh.
  • a1(3)=h3(b1(3)+w1(3)⊤a(2))a^{(3)}_1 = h_3(b_1^{(3)}+\boldsymbol{w}_1^{(3)\top}\boldsymbol{a}^{(2)})a1(3)​=h3​(b1(3)​+w1(3)⊤​a(2)), a2(3)=h3(b2(3)+w2(3)⊤a(2))a^{(3)}_2 = h_3(b_2^{(3)}+\boldsymbol{w}_2^{(3)\top}\boldsymbol{a}^{(2)})a2(3)​=h3​(b2(3)​+w2(3)⊤​a(2))
  • a(3)=(a1(3),a2(3))⊤\boldsymbol{a}^{(3)} = (a^{(3)}_1, a^{(3)}_2)^\topa(3)=(a1(3)​,a2(3)​)⊤

Output layer

  • The activation function h4h_4h4​ is Softmax.
  • f1(x)=h4(b1(4)+w1(4)⊤a(3))f_1(\boldsymbol{x}) = h_4(b_1^{(4)}+\boldsymbol{w}_1^{(4)\top}\boldsymbol{a}^{(3)})f1​(x)=h4​(b1(4)​+w1(4)⊤​a(3)), f2(x)=h4(b2(4)+w2(4)⊤a(3))f_2(\boldsymbol{x}) = h_4(b_2^{(4)}+\boldsymbol{w}_2^{(4)\top}\boldsymbol{a}^{(3)})f2​(x)=h4​(b2(4)​+w2(4)⊤​a(3)), f3(x)=h4(b3(4)+w3(4)⊤a(3))f_3(\boldsymbol{x}) = h_4(b_3^{(4)}+\boldsymbol{w}_3^{(4)\top}\boldsymbol{a}^{(3)})f3​(x)=h4​(b3(4)​+w3(4)⊤​a(3))

General feed forward neural network

    • Input layer: x=(1,x1,…,xp)⊤\boldsymbol{x} = (1, x_1, \dots, x_p)^\topx=(1,x1​,…,xp​)⊤
    • Layer 2: a(2)=(h2(β1⊤x),…,h2(βK2⊤x))⊤\boldsymbol{a}^{(2)} = (h_2(\boldsymbol{\beta}^\top_1\boldsymbol{x}), \dots, h_2(\boldsymbol{\beta}^\top_{K_2}\boldsymbol{x}))^\topa(2)=(h2​(β1⊤​x),…,h2​(βK2​⊤​x))⊤
    • Layer 3: a(3)=(h3(b1(3)+w1(3)⊤a(2)),…,h3(bK3(3)+wK3(3)⊤a(2)))⊤\boldsymbol{a}^{(3)} = (h_3(b_1^{(3)}+\boldsymbol{w}^{(3)\top}_1\boldsymbol{a}^{(2)}), \dots, h_3(b_{K_3}^{(3)}+\boldsymbol{w}^{(3)\top}_{K_3}\boldsymbol{a}^{(2)}))^\topa(3)=(h3​(b1(3)​+w1(3)⊤​a(2)),…,h3​(bK3​(3)​+wK3​(3)⊤​a(2)))⊤
    • ⋯\cdots⋯
    • Layer lll:a(l)=(hl(b1(l)+w1(l)⊤a(l−1)),…,hl(bKl(l)+wKl(l)⊤a(l−1)))⊤\boldsymbol{a}^{(l)} = (h_l(b_1^{(l)}+\boldsymbol{w}^{(l)\top}_1\boldsymbol{a}^{(l - 1)}), \dots, h_l(b_{K_l}^{(l)}+\boldsymbol{w}^{(l)\top}_{K_l}\boldsymbol{a}^{(l-1)}))^\topa(l)=(hl​(b1(l)​+w1(l)⊤​a(l−1)),…,hl​(bKl​(l)​+wKl​(l)⊤​a(l−1)))⊤
    • ⋯\cdots⋯
    • Layer L−1L -1L−1: a(L−1)=(hL−1(b1(L−1)+w1(L−1)⊤a(L−2)),…,hL−1(bKL−1(L−1)+wKL−1(L−1)⊤a(L−2)))⊤\boldsymbol{a}^{(L-1)} = (h_{L-1}(b_1^{(L-1)}+\boldsymbol{w}^{(L-1)\top}_1\boldsymbol{a}^{(L-2)}), \dots, h_{L-1}(b_{K_{L-1}}^{({L-1})}+\boldsymbol{w}^{({L-1})\top}_{K_{L-1}}\boldsymbol{a}^{(L-2)}))^\topa(L−1)=(hL−1​(b1(L−1)​+w1(L−1)⊤​a(L−2)),…,hL−1​(bKL−1​(L−1)​+wKL−1​(L−1)⊤​a(L−2)))⊤
    • Output layer: fk(x)=hL(b(L)+wk(L)⊤a(L−1))f_k(\boldsymbol{x}) = h_L(b^{(L)}+\boldsymbol{w}_k^{(L)\top}\boldsymbol{a}^{(L-1)})fk​(x)=hL​(b(L)+wk(L)⊤​a(L−1)).
  • The output actually depends on the parameters: θ=(β1,…,βK2⏟coefficients,b1(3),…,bKL−1(L−1)⏟biases,w1(3)⊤,…,wk(L)⊤⏟weights)⊤.\boldsymbol{\theta} = (\underbrace{\beta_1, \dots, \beta_{K_2}}_{\text{coefficients}}, \underbrace{b_1^{(3)}, \dots, b_{K_{L-1}}^{(L-1)}}_{\text{biases}},\underbrace{\boldsymbol{w}_1^{(3)\top}, \dots, \boldsymbol{w}_k^{(L)\top}}_{\text{weights}})^\top.θ=(coefficientsβ1​,…,βK2​​​​,biasesb1(3)​,…,bKL−1​(L−1)​​​,weightsw1(3)⊤​,…,wk(L)⊤​​​)⊤.
  • We can write this dependency more explicitly as fk(x ∣ θ)f_k(\boldsymbol{x}~|~\boldsymbol{\theta})fk​(x ∣ θ).
  • How do we calibrate (or train) these parameters θ\boldsymbol{\theta}θ?

Calibration

Calibration

  • As like other models, we:
    • define a loss function, and
    • find parameters that minimise this loss function.
  • What loss function we use depends on the problem.

Regression loss

  • Find θ\boldsymbol{\theta}θ that minimises the mean squared error:

MSE(θ)=1n∑i=1n(yi−f(xi ∣ θ))2\text{MSE}(\boldsymbol{\theta}) = \frac{1}{n}\sum_{i=1}^n\left(y_i - f(\boldsymbol{x}_i~|~\boldsymbol{\theta})\right)^2MSE(θ)=n1​i=1∑n​(yi​−f(xi​ ∣ θ))2

Binary classification loss

  • Find θ\boldsymbol{\theta}θ that minimises the binary cross-entropy (BCE):

BCE(θ)=−1n∑i=1n{yilog⁡(P(yi=1 ∣ xi,θ))+(1−yi)log⁡(1−P(yi=1 ∣ xi,θ))}\text{BCE}(\boldsymbol{\theta}) = -\frac{1}{n}\sum_{i=1}^n\left\{y_i\log(P(y_i=1~|~\boldsymbol{x}_i,\boldsymbol{\theta})) + (1-y_i)\log(1 - P(y_i=1~|~\boldsymbol{x}_i,\boldsymbol{\theta}))\right\}BCE(θ)=−n1​i=1∑n​{yi​log(P(yi​=1 ∣ xi​,θ))+(1−yi​)log(1−P(yi​=1 ∣ xi​,θ))}

Multi-class classification loss

  • Find θ\boldsymbol{\theta}θ that minimises the cross-entropy (CE):

CE(θ)=−1n∑i=1n∑j=1myijlog⁡(P(yij=1 ∣ xi,θ))\text{CE}(\boldsymbol{\theta}) = -\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^my_{ij}\log(P(y_{ij} = 1~|~\boldsymbol{x}_i,\boldsymbol{\theta}))CE(θ)=−n1​i=1∑n​j=1∑m​yij​log(P(yij​=1 ∣ xi​,θ))

Optimisation

  • Regardless of whether it is regression or classification, we must find hyperparameters that optimise a loss function.
  • Finding these hyperparameters is often hard with no closed-form solution.
  • There are various optimisation methods to find these hyperparameters – we discuss:
    • gradient descent, and
    • stochastic gradient descent.

Illustrative example

  • Suppose that we have a regression problem that require callibration of one hyperparameter with MSE loss function.
  • Our goal is to find θ\thetaθ that corresponds to the bottom of the curve.

Finding the zero slope

  • Suppose that we calculate MSE(θ)\text{MSE}(\theta)MSE(θ) for three values of θ\thetaθ.
  • For this function, the optimal θ\thetaθ is where the individual with the red hat is.
  • This is also where the slope (or derivative) is zero: ∂MSE(θ)∂θ=0.\frac{\partial \text{MSE}(\theta)}{\partial \theta} = 0.∂θ∂MSE(θ)​=0.

Optimising a multi-variable function

  • In practice, θ\boldsymbol{\theta}θ is typically a vector of length ddd with d>1d > 1d>1.
  • The optimal value of θ\boldsymbol{\theta}θ is then found by solving for θ\boldsymbol{\theta}θ such that ∇MSE(θ)=0\nabla \text{MSE}(\boldsymbol{\theta}) = 0∇MSE(θ)=0 where ∇MSE(θ)=(∂MSE(θ)∂θ1,…,∂MSE(θ)∂θd)⊤\nabla \text{MSE}(\boldsymbol{\theta}) = \left(\frac{\partial\text{MSE}(\boldsymbol{\theta})}{\partial\theta_1}, \dots, \frac{\partial\text{MSE}(\boldsymbol{\theta})}{\partial\theta_d}\right)^\top∇MSE(θ)=(∂θ1​∂MSE(θ)​,…,∂θd​∂MSE(θ)​)⊤ is the vector of slopes for all parameters (gradient).
  • Finding θ\boldsymbol{\theta}θ such that ∇MSE(θ)=0\nabla \text{MSE}(\boldsymbol{\theta}) = 0∇MSE(θ)=0 is hard!
  • We use gradient descent to find this.

Searching for the optimal value

  • To search for the optimal value of θ\thetaθ, we choose some starting points.
  • We can calculate the slope at the starting points giving us a guide where to search next: θ[s+1]=θ[s]−r∂MSE(θ)∂θ∣θ=θ[s],\theta^{[s+1]} = \theta^{[s]} - r \left.\frac{\partial\text{MSE}(\theta)}{\partial\theta}\right\vert_{\theta=\theta^{[s]}},θ[s+1]=θ[s]−r∂θ∂MSE(θ)​∣∣​θ=θ[s]​, where r>0r > 0r>0 is the length of the step.

Illustrative example: Step 0

θ[s+1]=θ[s]−r∂MSE(θ)∂θ∣θ=θ[s]\theta^{[s+1]} = \theta^{[s]} - r \left.\frac{\partial\text{MSE}(\theta)}{\partial\theta}\right\vert_{\theta=\theta^{[s]}}θ[s+1]=θ[s]−r∂θ∂MSE(θ)​∣∣​θ=θ[s]​

  • Suppose
    • ∂MSE(θ)∂θ=4θ\frac{\partial\text{MSE}(\theta)}{\partial\theta} = 4\theta∂θ∂MSE(θ)​=4θ,
    • r=0.1r = 0.1r=0.1, and
    • the starting point θ[0]=−1.8.\theta^{[0]} = -1.8.θ[0]=−1.8.
  • θ[1]=−1.8−0.1×4×(−1.8)=−1.08.\theta^{[1]} = -1.8 - 0.1 \times 4 \times (-1.8) = -1.08.θ[1]=−1.8−0.1×4×(−1.8)=−1.08.

Illustrative example: Step 1

θ[s+1]=θ[s]−r∂MSE(θ)∂θ∣θ=θ[s]\theta^{[s+1]} = \theta^{[s]} - r \left.\frac{\partial\text{MSE}(\theta)}{\partial\theta}\right\vert_{\theta=\theta^{[s]}}θ[s+1]=θ[s]−r∂θ∂MSE(θ)​∣∣​θ=θ[s]​

  • θ[1]=−1.08\theta^{[1]} = -1.08θ[1]=−1.08
  • θ[2]=−1.08−0.1×4×(−1.08)=−0.648.\theta^{[2]} = -1.08 - 0.1 \times 4 \times (-1.08) = -0.648.θ[2]=−1.08−0.1×4×(−1.08)=−0.648.

Illustrative example: Step 2

θ[s+1]=θ[s]−r∂MSE(θ)∂θ∣θ=θ[s]\theta^{[s+1]} = \theta^{[s]} - r \left.\frac{\partial\text{MSE}(\theta)}{\partial\theta}\right\vert_{\theta=\theta^{[s]}}θ[s+1]=θ[s]−r∂θ∂MSE(θ)​∣∣​θ=θ[s]​

  • θ[2]=−0.648\theta^{[2]} = -0.648θ[2]=−0.648
  • θ[3]=−0.648−0.1×4×(−0.648)=−0.3888.\theta^{[3]} = -0.648 - 0.1 \times 4 \times (-0.648) = -0.3888.θ[3]=−0.648−0.1×4×(−0.648)=−0.3888.

Illustrative example: Step 3

θ[s+1]=θ[s]−r∂MSE(θ)∂θ∣θ=θ[s]\theta^{[s+1]} = \theta^{[s]} - r \left.\frac{\partial\text{MSE}(\theta)}{\partial\theta}\right\vert_{\theta=\theta^{[s]}}θ[s+1]=θ[s]−r∂θ∂MSE(θ)​∣∣​θ=θ[s]​

  • θ[3]=−0.3888\theta^{[3]} = -0.3888θ[3]=−0.3888
  • θ[4]=−0.3888−0.1×4×(−0.3888)=−0.23328\theta^{[4]} = -0.3888 - 0.1 \times 4 \times (-0.3888) = -0.23328θ[4]=−0.3888−0.1×4×(−0.3888)=−0.23328

Illustrative example: Step 4

θ[s+1]=θ[s]−r∂MSE(θ)∂θ∣θ=θ[s]\theta^{[s+1]} = \theta^{[s]} - r \left.\frac{\partial\text{MSE}(\theta)}{\partial\theta}\right\vert_{\theta=\theta^{[s]}}θ[s+1]=θ[s]−r∂θ∂MSE(θ)​∣∣​θ=θ[s]​

  • θ[4]=−0.23328\theta^{[4]} = -0.23328θ[4]=−0.23328
  • θ[5]=−0.23328−0.1×4×(−0.23328)=−0.139968\theta^{[5]} = -0.23328 - 0.1 \times 4 \times (-0.23328) = -0.139968θ[5]=−0.23328−0.1×4×(−0.23328)=−0.139968

⋯\cdots⋯

Illustrative example: Step 20

θ[s+1]=θ[s]−r∂MSE(θ)∂θ∣θ=θ[s]\theta^{[s+1]} = \theta^{[s]} - r \left.\frac{\partial\text{MSE}(\theta)}{\partial\theta}\right\vert_{\theta=\theta^{[s]}}θ[s+1]=θ[s]−r∂θ∂MSE(θ)​∣∣​θ=θ[s]​

  • θ[20]=−0.00006581085\theta^{[20]} = -0.00006581085θ[20]=−0.00006581085
  • What if there are more than one parameter?

Gradient descent

  • In the multi-variable version: θ[s+1]=θ[s]−r∇MSE(θ)∣θ=θ[s].\boldsymbol{\theta}^{[s+1]} = \left.\boldsymbol{\theta}^{[s]} - r\nabla \text{MSE}(\boldsymbol{\theta})\right\vert_{\theta=\theta^{[s]}}.θ[s+1]=θ[s]−r∇MSE(θ)∣∣​θ=θ[s]​. where r>0r>0r>0 is the learning rate.
  • You need a starting value θ[0]\boldsymbol{\theta}^{[0]}θ[0] and take a number of steps.
  • This is known as the gradient descent method.

Limitations of gradient descent

  • In practice, the loss function generally involves valuation for each observation, e.g. for MSE: ∇MSE(θ)=1n∑i=1n∇(yi−f(xi ∣ θ))2.\nabla \text{MSE}(\boldsymbol{\theta}) = \frac{1}{n}\sum_{i=1}^n\nabla\left(y_i - f(\boldsymbol{x}_i~|~\boldsymbol{\theta})\right)^2.∇MSE(θ)=n1​i=1∑n​∇(yi​−f(xi​ ∣ θ))2.
  • This can be computationally expensive if nnn is large.
  • Gradient descent optimisation can get stuck in local optima.

Stochastic gradient descent

Stochastic gradient descent

  • In a stochastic gradient descent (SGD) considers at each step θ[s+1]=θ[s]−r∇MSE^(θ)∣θ=θ[s]\boldsymbol{\theta}^{[s+1]} = \boldsymbol{\theta}^{[s]} - r \left.\nabla \hat{\text{MSE}}(\boldsymbol{\theta})\right\vert_{\boldsymbol{\theta}=\boldsymbol{\theta}^{[s]}}θ[s+1]=θ[s]−r∇MSE^(θ)∣∣​θ=θ[s]​ where ∇MSE^(θ)∣θ=θ[s]=1M∑i∈B[s]∇(yi−f(xi ∣ θ[s]))2.\left.\nabla \hat{\text{MSE}}(\boldsymbol{\theta})\right\vert_{\boldsymbol{\theta}=\boldsymbol{\theta}^{[s]}}=\frac{1}{M}\sum_{i\in B^{[s]}}\nabla\left(y_i - f(\boldsymbol{x}_i~|~\boldsymbol{\theta}^{[s]})\right)^2.∇MSE^(θ)∣∣​θ=θ[s]​=M1​i∈B[s]∑​∇(yi​−f(xi​ ∣ θ[s]))2. where B[s]B^{[s]}B[s] denotes a set of random batch of M≪nM \ll nM≪n observations at iteration sss.

Batch illustration

  • Suppose we have n=1000n = 1000n=1000.
  • We draw a (random without replacement) batch of size 3, say resulting in B[1]={133,606,851}B^{[1]} = \{133, 606, 851\}B[1]={133,606,851}.
  • With gradient descent, ∇MSE(θ)=11000(∇(y1−f(x1 ∣ θ)2+∇(y2−f(x2 ∣ θ)2+⋯+∇(y1000−f(x1000 ∣ θ)2).\nabla\text{MSE}(\boldsymbol{\theta}) = \frac{1}{1000}\left(\nabla(y_1 - f(\boldsymbol{x}_1~|~\boldsymbol{\theta})^2 + \nabla(y_2 - f(\boldsymbol{x}_2~|~\boldsymbol{\theta})^2 + \cdots + \nabla(y_{1000} - f(\boldsymbol{x}_{1000}~|~\boldsymbol{\theta})^2\right).∇MSE(θ)=10001​(∇(y1​−f(x1​ ∣ θ)2+∇(y2​−f(x2​ ∣ θ)2+⋯+∇(y1000​−f(x1000​ ∣ θ)2).
  • But for stochastic gradient descent, we replace this with: ∇MSE^(θ)=13(∇(y133−f(x133 ∣ θ)2+∇(y606−f(x606 ∣ θ)2+∇(y851−f(x851 ∣ θ)2).\nabla\hat{\text{MSE}}(\boldsymbol{\theta}) = \frac{1}{3}\left(\nabla(y_{133} - f(\boldsymbol{x}_{133}~|~\boldsymbol{\theta})^2 + \nabla(y_{606} - f(\boldsymbol{x}_{606}~|~\boldsymbol{\theta})^2 + \nabla(y_{851} - f(\boldsymbol{x}_{851}~|~\boldsymbol{\theta})^2\right).∇MSE^(θ)=31​(∇(y133​−f(x133​ ∣ θ)2+∇(y606​−f(x606​ ∣ θ)2+∇(y851​−f(x851​ ∣ θ)2).
  • The latter is faster to calculate and can escape from local optima.

Epoch

  • The number of epochs is the number of times that the algorithm has used all observations in the data.
  • Illustrative data: y1y_1y1​, y2y_2y2​, y3y_3y3​, y4y_4y4​, y5y_5y5​, y6y_6y6​, y7y_7y7​, y8y_8y8​, y9y_9y9​, y10y_{10}y10​, y11y_{11}y11​, y12y_{12}y12​

Epoch

  • The number of epochs is the number of times that the algorithm has used all observations in the data.
  • Illustrative data: y1y_1y1​, y2y_2y2​, y3y_3y3​, y4y_4y4​, y5y_5y5​, y6y_6y6​, y7y_7y7​, y8y_8y8​, y9y_9y9​, y10y_{10}y10​, y11y_{11}y11​, y12y_{12}y12​
  1. s=1s = 1s=1, B[1]={4,8,10}B^{[1]} = \color{red}{\{4, 8, 10\}}B[1]={4,8,10}

Epoch

  • The number of epochs is the number of times that the algorithm has used all observations in the data.
  • Illustrative data: y1y_1y1​, y2y_2y2​, y3y_3y3​, y4y_4y4​, y5y_5y5​, y6y_6y6​, y7y_7y7​, y8y_8y8​, y9y_9y9​, y10y_{10}y10​, y11y_{11}y11​, y12y_{12}y12​
  1. s=1s = 1s=1, B[1]={4,8,10}B^{[1]} = \{4, 8, 10\}B[1]={4,8,10}
  2. s=2s = 2s=2, B[2]={1,5,12}B^{[2]} = \color{red}{\{1, 5, 12\}}B[2]={1,5,12}

Epoch

  • The number of epochs is the number of times that the algorithm has used all observations in the data.
  • Illustrative data: y1y_1y1​, y2y_2y2​, y3y_3y3​, y4y_4y4​, y5y_5y5​, y6y_6y6​, y7y_7y7​, y8y_8y8​, y9y_9y9​, y10y_{10}y10​, y11y_{11}y11​, y12y_{12}y12​
  1. s=1s = 1s=1, B[1]={4,8,10}B^{[1]} = \{4, 8, 10\}B[1]={4,8,10}
  2. s=2s = 2s=2, B[2]={1,5,12}B^{[2]} = \{1, 5, 12\}B[2]={1,5,12}
  3. s=3s = 3s=3, B[3]={2,3,6}B^{[3]} = \color{red}{\{2, 3, 6\}}B[3]={2,3,6}

Epoch

  • The number of epochs is the number of times that the algorithm has used all observations in the data.
  • Illustrative data: y1y_1y1​, y2y_2y2​, y3y_3y3​, y4y_4y4​, y5y_5y5​, y6y_6y6​, y7y_7y7​, y8y_8y8​, y9y_9y9​, y10y_{10}y10​, y11y_{11}y11​ , y12y_{12}y12​
  1. s=1s = 1s=1, B[1]={4,8,10}B^{[1]} = \{4, 8, 10\}B[1]={4,8,10}
  2. s=2s = 2s=2, B[2]={1,5,12}B^{[2]} = \{1, 5, 12\}B[2]={1,5,12}
  3. s=3s = 3s=3, B[3]={2,3,6}B^{[3]} = \{2, 3, 6\}B[3]={2,3,6}
  4. s=4s = 4s=4, B[4]={7,9,11}B^{[4]} = \color{red}{\{7, 9, 11\}}B[4]={7,9,11}

Epoch

  • The number of epochs is the number of times that the algorithm has used all observations in the data.
  • Illustrative data: y1y_1y1​, y2y_2y2​, y3y_3y3​, y4y_4y4​, y5y_5y5​, y6y_6y6​, y7y_7y7​, y8y_8y8​, y9y_9y9​, y10y_{10}y10​, y11y_{11}y11​ , y12y_{12}y12​
  1. s=1s = 1s=1, B[1]={4,8,10}B^{[1]} = \{4, 8, 10\}B[1]={4,8,10}
  2. s=2s = 2s=2, B[2]={1,5,12}B^{[2]} = \{1, 5, 12\}B[2]={1,5,12}
  3. s=3s = 3s=3, B[3]={2,3,6}B^{[3]} = \{2, 3, 6\}B[3]={2,3,6}
  4. s=4s = 4s=4, B[4]={7,9,11}B^{[4]} = \{7, 9, 11\}B[4]={7,9,11}

Epoch 1

Epoch

  • The number of epochs is the number of times that the algorithm has used all observations in the data.
  • Illustrative data: y1y_1y1​, y2y_2y2​, y3y_3y3​, y4y_4y4​, y5y_5y5​, y6y_6y6​, y7y_7y7​, y8y_8y8​, y9y_9y9​, y10y_{10}y10​, y11y_{11}y11​ , y12y_{12}y12​
  1. s=1s = 1s=1, B[1]={4,8,10}B^{[1]} = \{4, 8, 10\}B[1]={4,8,10}
  2. s=2s = 2s=2, B[2]={1,5,12}B^{[2]} = \{1, 5, 12\}B[2]={1,5,12}
  3. s=3s = 3s=3, B[3]={2,3,6}B^{[3]} = \{2, 3, 6\}B[3]={2,3,6}
  4. s=4s = 4s=4, B[4]={7,9,11}B^{[4]} = \{7, 9, 11\}B[4]={7,9,11}

Epoch 1

  1. s=5s = 5s=5, B[5]={3,4,11}B^{[5]} = \{3, 4, 11\}B[5]={3,4,11}
  2. s=6s = 6s=6, B[6]={2,7,9}B^{[6]} = \{2, 7, 9\}B[6]={2,7,9}
  3. s=7s = 7s=7, B[7]={6,10,12}B^{[7]} = \{6, 10, 12\}B[7]={6,10,12}
  4. s=8s = 8s=8, B[8]={1,5,8}B^{[8]} = \{1, 5, 8\}B[8]={1,5,8}

Epoch 2

Backpropagation

  • In evaluating ∇MSE^(θ)\nabla \hat{\text{MSE}}(\boldsymbol{\theta})∇MSE^(θ), we must evaluate ∇(yi−f(xi ∣ θ))2\nabla (y_i - f(\boldsymbol{x}_i~|~\boldsymbol{\theta}))^2∇(yi​−f(xi​ ∣ θ))2.

  • Now we have: ∇(yi−f(xi ∣ θ))2=−2(yi−f(xi ∣ θ))×∂f(xi∣θ)∂ai(L−1)⏟Layer L−1×∂ai(L−1)∂ai(L−2)⏟Layer L−2×∂ai(L−2)∂ai(L−3)⏟Layer L−3×⋯×∂ai(2)∂θ⏟Layer 1.\nabla (y_i - f(\boldsymbol{x}_i~|~\boldsymbol{\theta}))^2 = - 2(y_i - f(\boldsymbol{x}_i~|~\boldsymbol{\theta}))\times \underbrace{\frac{\partial f(\boldsymbol{x}_i|\boldsymbol{\theta})}{\partial \boldsymbol{a}_i^{(L - 1)}}}_{\text{Layer }L -1}\times \underbrace{\frac{\partial \boldsymbol{a}_i^{(L - 1)}}{\partial \boldsymbol{a}_i^{(L - 2)}}}_{\text{Layer }L -2}\times \underbrace{\frac{\partial \boldsymbol{a}_i^{(L - 2)}}{\partial \boldsymbol{a}_i^{(L - 3)}}}_{\text{Layer }L -3}\times \cdots \times \underbrace{\frac{\partial \boldsymbol{a}_i^{(2)}}{\partial \boldsymbol{\theta}}}_{\text{Layer }1}.∇(yi​−f(xi​ ∣ θ))2=−2(yi​−f(xi​ ∣ θ))×Layer L−1∂ai(L−1)​∂f(xi​∣θ)​​​×Layer L−2∂ai(L−2)​∂ai(L−1)​​​​×Layer L−3∂ai(L−3)​∂ai(L−2)​​​​×⋯×Layer 1∂θ∂ai(2)​​​​.

  • The gradient of the later layers is calculated earlier.

  • For parameter θ\thetaθ, we can show ∂(yi−f(xi ∣ θ))2∂θ=constant×∂hL(ai(L−1))∂ai(L−1)⏟Layer L−1×∂hL(ai(L−2))∂ai(L−2)⏟Layer L−2×∂hL(ai(L−3))∂ai(L−3)⏟Layer L−3×⋯×∂h2(xi∣θ)∂θ⏟Layer 1.\frac{\partial (y_i - f(\boldsymbol{x}_i~|~\boldsymbol{\theta}))^2}{\partial \theta} = \text{constant}\times \underbrace{\frac{\partial h_L(a_i^{(L-1)})}{\partial a_i^{(L-1)}}}_{\text{Layer }L -1}\times \underbrace{\frac{\partial h_L(a_i^{(L-2)})}{\partial a_i^{(L-2)}}}_{\text{Layer }L -2}\times \underbrace{\frac{\partial h_L(a_i^{(L-3)})}{\partial a_i^{(L-3)}}}_{\text{Layer }L -3}\times \cdots \times \underbrace{\frac{\partial h_2(x_i| \theta)}{\partial \theta}}_{\text{Layer }1}.∂θ∂(yi​−f(xi​ ∣ θ))2​=constant×Layer L−1∂ai(L−1)​∂hL​(ai(L−1)​)​​​×Layer L−2∂ai(L−2)​∂hL​(ai(L−2)​)​​​×Layer L−3∂ai(L−3)​∂hL​(ai(L−3)​)​​​×⋯×Layer 1∂θ∂h2​(xi​∣θ)​​​.

Vanishing gradient problem

∂(yi−f(xi ∣ θ))2∂θ=constant×∂hL(ai(L−1))∂ai(L−1)⏟Layer L−1×∂hL(ai(L−2))∂ai(L−2)⏟Layer L−2×∂hL(ai(L−3))∂ai(L−3)⏟Layer L−3×⋯×∂h2(xi∣θ)∂θ⏟Layer 1.\frac{\partial (y_i - f(\boldsymbol{x}_i~|~\boldsymbol{\theta}))^2}{\partial \theta} = \text{constant}\times \underbrace{\frac{\partial h_L(a_i^{(L-1)})}{\partial a_i^{(L-1)}}}_{\text{Layer }L -1}\times \underbrace{\frac{\partial h_L(a_i^{(L-2)})}{\partial a_i^{(L-2)}}}_{\text{Layer }L -2}\times \underbrace{\frac{\partial h_L(a_i^{(L-3)})}{\partial a_i^{(L-3)}}}_{\text{Layer }L -3}\times \cdots \times \underbrace{\frac{\partial h_2(x_i| \theta)}{\partial \theta}}_{\text{Layer }1}.∂θ∂(yi​−f(xi​ ∣ θ))2​=constant×Layer L−1∂ai(L−1)​∂hL​(ai(L−1)​)​​​×Layer L−2∂ai(L−2)​∂hL​(ai(L−2)​)​​​×Layer L−3∂ai(L−3)​∂hL​(ai(L−3)​)​​​×⋯×Layer 1∂θ∂h2​(xi​∣θ)​​​.

  • If hlh_lhl​ is the Sigmoid activation function then 0<∂hl(ai(l−1))∂ai(l−1)<10 < \frac{\partial h_l(a_i^{(l -1)})}{\partial a_i^{(l -1)}} < 10<∂ai(l−1)​∂hl​(ai(l−1)​)​<1.
  • If the LLL is large then this can result in ∇(yi−f(xi∣θ))2≈0\nabla (y_i - f(\boldsymbol{x}_i | \boldsymbol{\theta}))^2 \approx 0∇(yi​−f(xi​∣θ))2≈0.
  • This phenomenon is referred to as the vanishing gradient problem and results in difficulty with calibrating the neural network.
  • To avoid this problem, you can use a different activation function that does not have the same issue, e.g. ReLU.

An application to fashion MNIST data

Fashion MNIST data

scroll

Code 
library(tidyverse)
fashion <- read_csv("https://emitanaka.org/iml/data/fashion-mnist_train.csv.gz")
fashion_test <- read_csv("https://emitanaka.org/iml/data/fashion-mnist_test.csv")
  • The fashion MNIST data by Zalando SE contains 28×2828 \times 2828×28 pixel images of fashion items labelled as:
Label Description
0 T-shirt/top
1 Trouser
2 Pullover
3 Dress
4 Coat
5 Sandal
6 Shirt
7 Sneaker
8 Bag
9 Ankle boot

  • So this is a multi-class classification problem with m=10m = 10m=10 classes (or levels).
  • The training data contains 60,000 observations with 784 variables (pixel1, …, pixel784) and 1 response variable (label) labelled from 0-9.
  • The testing data contains 10,000 observations (note: this testing data also contains label!).

Neural network in R 1 prepare the data

  • To fit a neural network model, we use the keras package.
  • We first need to prepare the data:
    • convert categorical data to dummy variables, and
    • normalise the predictors.
fashion_train_y <- model.matrix(~factor(label) - 1, data = fashion)

fashion_train_x <- fashion %>% 
  select(-label) %>% 
  scale()

fashion_test_y <- model.matrix(~factor(label) - 1, data = fashion_test)

fashion_test_x <- fashion_test %>% 
  select(-label) %>% 
  scale()

Neural network in R 2 Define architechture

library(keras)
NN <- keras_model_sequential() %>% 
  # hidden layer
  layer_dense(units = 128, # number of nodes in hidden layer
              activation = "relu",
              # number of predictors
              input_shape = 784) %>% 
  # output layer
  layer_dense(units = 10, # the number of classes
              # we need to use softmax for multi-class classification
              activation = "softmax")

NN
Model: "sequential"
________________________________________________________________________________
 Layer (type)                       Output Shape                    Param #     
================================================================================
 dense_1 (Dense)                    (None, 128)                     100480      
 dense (Dense)                      (None, 10)                      1290        
================================================================================
Total params: 101,770
Trainable params: 101,770
Non-trainable params: 0
________________________________________________________________________________

Neural network in R 3 Choose loss function

  • For regression, you can use mean_squared_error.
  • For binary classification, you can use binary_crossentropy.
  • For multi-class classfication, you can use categorical_crossentropy.
  • For the full list of loss functions, see help("loss-functions").
NNl <- NN %>% 
  compile(loss = "categorical_crossentropy",
          metrics = "accuracy")

Neural network in R 4 Fit model

This model takes long to fit!

learnNNl <- NNl %>% 
  fit(x = fashion_train_x,
      y = fashion_train_y,
      epochs = 20,
      batch_size = 40,
      validation_split = 0.2)
  • Typically epochs is a large number.

Model diagnostic

  • We can plot the loss (and other metrics specified) of the neural network using the plot function on the training history.
class(learnNNl)
[1] "keras_training_history"
plot(learnNNl)

  • This model actually is not doing well for the validation set!
  • This suggests that the model is overfitted.

Predicting from neural network

  • Notice we are using NNl and not learnNNl here!
test_predict <- predict(NNl, fashion_test_x)

test_predict[1:3, ]
             [,1]         [,2]         [,3]         [,4]         [,5]
[1,] 9.959991e-01 4.470216e-15 2.498414e-06 1.280446e-06 6.907984e-09
[2,] 2.188681e-21 1.000000e+00 5.769808e-32 9.360376e-28 1.490198e-26
[3,] 6.637161e-03 6.112183e-16 9.681513e-01 3.647272e-13 6.959190e-06
             [,6]         [,7]         [,8]         [,9]        [,10]
[1,] 2.304835e-11 3.900097e-03 9.305310e-19 9.703544e-05 4.587546e-13
[2,] 0.000000e+00 8.628763e-31 0.000000e+00 4.123140e-29 1.746240e-37
[3,] 8.466854e-11 2.520457e-02 8.891557e-21 6.381754e-13 2.705846e-17
  • Each row corresponds to the observation in the test data.
  • Column ccc corresponds to the probability of the label c−1c - 1c−1.
  • Sum of each row is 1.

Visually checking the prediction: observation 1

Code 
tibble(label = 0:9,
       desc = c("T-shirt/top", "Trouser", "Pullover", "Dress", "Coat", "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot")) %>% 
  mutate(prob = test_predict[1, ]) %>% 
  ggplot(aes(prob, desc)) +
  geom_col() +
  labs(y = "", x= "Probability")

Visually checking the prediction: observation 2

Code 
tibble(label = 0:9,
       desc = c("T-shirt/top", "Trouser", "Pullover", "Dress", "Coat", "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot")) %>% 
  mutate(prob = test_predict[2, ]) %>% 
  ggplot(aes(prob, desc)) +
  geom_col() +
  labs(y = "", x= "Probability")

Visually checking the prediction: observation 3

Code 
tibble(label = 0:9,
       desc = c("T-shirt/top", "Trouser", "Pullover", "Dress", "Coat", "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot")) %>% 
  mutate(prob = test_predict[3, ]) %>% 
  ggplot(aes(prob, desc)) +
  geom_col() +
  labs(y = "", x= "Probability")

Visually checking the prediction: observation 4

Code 
tibble(label = 0:9,
       desc = c("T-shirt/top", "Trouser", "Pullover", "Dress", "Coat", "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot")) %>% 
  mutate(prob = test_predict[4, ]) %>% 
  ggplot(aes(prob, desc)) +
  geom_col() +
  labs(y = "", x= "Probability")

Visually checking the prediction: observation 5

Code 
tibble(label = 0:9,
       desc = c("T-shirt/top", "Trouser", "Pullover", "Dress", "Coat", "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot")) %>% 
  mutate(prob = test_predict[5, ]) %>% 
  ggplot(aes(prob, desc)) +
  geom_col() +
  labs(y = "", x= "Probability")

Limitations

Cons of neural network

  • Minimal interpretability.
  • Requires the number of observations to be larger than the number of features.
  • Computationally intensive
    • Many calculations are required to estimate all of the parameters in many neural networks.
    • Deep learning involves huge amounts of matrix multiplications and other operations.
    • Often used in conjuction with GPUs to parallellise computations.

Takeaways

  • To build a feed-forward neural network, we need the key components:
    1. Input data,
    2. A pre-defined network architecture,
    3. A feedback mechanism (e.g. loss function) to enable the network to learn, and
    4. Model training.
  • Deep neural networks are faster to calibrate than wide neural networks via stochastic gradient descent and backpropogation algorithms.

ETC3250/5250 Week 12

1 / 51
No notes on this slide.
Neural network II - table of contents ETC3250/5250 Neural network II Acknowledgement Wide and deep neural networks Neural network layers Wide neural network Deep neural network Feed forward neural networks Mathematical notation Forward evaluation General feed forward neural network Calibration Calibration Regression loss Binary classification loss Multi-class classification loss Optimisation Illustrative example Finding the zero slope Optimising a multi-variable function Searching for the optimal value Illustrative example: Step 0 Illustrative example: Step 1 Illustrative example: Step 2 Illustrative example: Step 3 Illustrative example: Step 4 Illustrative example: Step 20 Gradient descent Limitations of gradient descent Stochastic gradient descent Stochastic gradient descent Batch illustration Epoch Epoch Epoch Epoch Epoch Epoch Epoch Backpropagation Vanishing gradient problem An application to fashion MNIST data Fashion MNIST data Neural network in R 1 prepare the data Neural network in R 2 Define architechture Neural network in R 3 Choose loss function Neural network in R 4 Fit model Model diagnostic Predicting from neural network Visually checking the prediction: observation 1 Visually checking the prediction: observation 2 Visually checking the prediction: observation 3 Visually checking the prediction: observation 4 Visually checking the prediction: observation 5 Limitations Cons of neural network Takeaways

  1. Slides

  2. Tools

  3. Close
  • Neural network II - table of contents
  • ETC3250/5250
  • Acknowledgement
  • Wide and deep neural networks
  • Neural network layers
  • Wide neural network
  • Deep neural network
  • Feed forward neural networks
  • Mathematical notation
  • Forward evaluation
  • Forward evaluation
  • Forward evaluation
  • Forward evaluation
  • Forward evaluation
  • Forward evaluation
  • Forward evaluation
  • Forward evaluation
  • Forward evaluation
  • Forward evaluation
  • Forward evaluation
  • General feed forward neural network
  • Calibration
  • Calibration
  • Regression loss
  • Binary classification loss
  • Multi-class classification loss
  • Optimisation
  • Illustrative example
  • Finding the zero slope
  • Optimising a multi-variable function
  • Searching for the optimal value
  • Illustrative example: Step 0
  • Illustrative example: Step 1
  • Illustrative example: Step 2
  • Illustrative example: Step 3
  • Illustrative example: Step 4
  • Illustrative example: Step 20
  • Gradient descent
  • Limitations of gradient descent
  • Stochastic gradient descent
  • Stochastic gradient descent
  • Batch illustration
  • Epoch
  • Epoch
  • Epoch
  • Epoch
  • Epoch
  • Epoch
  • Epoch
  • Backpropagation
  • Vanishing gradient problem
  • An application to fashion MNIST data
  • Fashion MNIST data
  • Neural network in R 1 prepare the data
  • Neural network in R 2 Define architechture
  • Neural network in R 3 Choose loss function
  • Neural network in R 4 Fit model
  • Model diagnostic
  • Predicting from neural network
  • Visually checking the prediction: observation 1
  • Visually checking the prediction: observation 2
  • Visually checking the prediction: observation 3
  • Visually checking the prediction: observation 4
  • Visually checking the prediction: observation 5
  • Limitations
  • Cons of neural network
  • Takeaways
  • f Fullscreen
  • s Speaker View
  • o Slide Overview
  • e PDF Export Mode
  • b Toggle Chalkboard
  • c Toggle Notes Canvas
  • d Download Drawings
  • ? Keyboard Help