Course Website

Types of Neural Network

Figure adapted from slides by Andrew NG, Deep Learning Specialization

A Little Bit of History – Perceptron

• First concept of a simplified brain cell (1943): McCullock-Pitts (MCP) neuron
• Frank Rosenblatt published the first concept of the perceptron learning rule based on MCP (1957)
• Classification of N points into 2 classes: -1 and +1 (i.e., two different colors in the picture below)

• Fun video: https://www.youtube.com/watch?v=cNxadbrN_aI

• Two features: (x1 and x2); linear functions to separate classes; find (w0, w1, w2) such that:

z(i)=w0+w1x(i)1+w2x(i)2

pred(i)={1−1if z(i)>0if z(i)≤0

Perceptron Algorithm

Start with random weights. Set a maximum of epochs of M, for each epoch (permutation):

• For each data point i:
  • Predict class label

  z(i)=w0+w1x(i)1+w2x(i)2

  pred(i)={1−1if z(i)>0if z(i)≤0

  • For every data point, we update the weight based on the prediction correctness, learning rate and feature values.

w0=w0+η(actual(i)−pred(i))

w1=w1+η(actual(i)−pred(i))x(i)1 w2=w2+η(actual(i)−pred(i))x(i)2

• Calculate accuracy for the entire dataset to see whether it meets the criteria after each epoch.

Perceptron Algorithm

• Perceptron algorithm is easy to implement in any modern programming language.
  • Perceptron R notebook

• It is a linear classification function, and the weight is updated after we feed each data point to the algorithm (a concept similar to stochastic gradient descent).

• The algorithm continues to update when we feed the same data set again and again (i.e., epochs)

• It does not solve non-linearly spreadable problems.

Logistic Regression as A Neural Network

• m training samples: {(x(1),y(1)),(x(2),y(2)),...,(x(m),y(m))}

X=[x(1)1x(2)1⋯x(m)1x(1)2x(2)2⋯x(m)2⋮⋮⋮⋮x(1)nxx(2)nx…x(m)nx]∈Rnx×m

y=[y(1),y(2),…,y(m)]∈R1×m

ˆy(i)=σ(wTx(i)+b) where σ(z)=11+e−z

Logistic Regression as A Neural Network

• m training samples: {(x(1),y(1)),(x(2),y(2)),...,(x(m),y(m))}

X=[x(1)1x(2)1⋯x(m)1x(1)2x(2)2⋯x(m)2⋮⋮⋮⋮x(1)nxx(2)nx…x(m)nx]∈Rnx×m

y=[y(1),y(2),…,y(m)]∈R1×m

ˆy(i)=σ(wTx(i)+b) where σ(z)=11+e−z

• Loss function: L(ˆy,y)=−ylog(ˆy)−(1−y)log(1−ˆy)
• Cost function: J(w,b)=1mΣmi=1L(ˆy(i),y(i))=1mΣmi=1{ −y(i)log(ˆy(i))−(1−y(i))log(1−ˆy(i))}
• Goal: Find w,b that mininizes J(w,b)
  • w:=w−α∂J∂w
  • b:=b−α∂J∂b

Forward Propagation

Forward Propagation

Backward Propagation

Stochastic Gradient Descent (SGD)

Neural Network: 0 Hidden Layer Neural Network

Neural Network: 1 Hidden Layer Neural Network

Neural Network: 1 Layer Neural Network

Neural Network: 1 Layer Neural Network

Neural Network: 1 Hidden Layer Neural Network

Across m Samples

Across m Samples

MNIST Dataset

• Contains 70000 handwritten labeled digit images with label (60000 training + 10000 testing)
• Census Bureau employees and American high school students wrote these digits
• Each image is 28x28 pixel in greyscale
• Yann LeCun used convolutional network LeNet to achieve < 1% error rate at 1990s

Image Data

1 Hidden Layer Neural Network Example

Forward and Backward Propagation

Forward and Backward Propagation

• MINST with one hidden layer step by step

Deep Neural Network

Batch, Mini-batch, Stochastic Gradient Descent

• Mini-batch size = m: batch gradient descent, too long per iteration
• Mini-batch size = 1: stochastic gradient descent, lose speed from vectorization
• Mini-batch size in between: mini-batch gradient descent, make progress without processing all training set, typical batch sizes are 26=64, 27=128, 27=256, 28=512

x=[x(1),x(2),⋯,x(1000)⏟/⋯/⋯x(m)](nx,m)mini−batch 1

y=[y(1),y(2),⋯,y(1000)⏟/⋯/⋯y(m)](1,m)mini−batch 1

Batch, Mini-batch, Stochastic Gradient Descent

Activation Functions

• Representational power: combination of linear and non-linear transformation
• Some aspects that we can consider:
  • input and output range
  • gradients at initialization (usually small values)
  • gradients at extremes
  • computational complexity
• Bad news: no one activation function is the best.
• Good news: you can start with common use cases

Activation Functions

Activation Functions

• Intermediate layers
  • Relu (i.e. rectified linear unit) is usually a good choice which has the following good properties:
    

1. fast computation;
   
2. non-linear;
   
3. reduced likelihood of the gradient to vanish;
   
4. Unconstrained response
   • Sigmoid, studied in the past, not as good as Relu in deep learning, due to the gradient vanishing problem when there are many layers
   • hyperbolic tangent function (tanh)

• Last layer which connects to the output
  • Binary classification: sigmoid with binary cross entropy as loss function
  • Multiple class, single-label classification: softmax with categorical cross entropy for loss function
  • Continuous responses: identity function (i.e. y = x)

Deal with Overfitting: Regularization

For logistic regression,

minw,bJ(w,b)=1mΣmi=1L(ˆy(i),y(i))+penalty

where

L2 penalty=λ2∥w∥22=λ2Σnxi=1w2i L1 penalty=λΣnxi=1|w| For neural network,

J(w[1],b[1],…,w[L],b[L])=1mΣmi=1L(ˆy(i),y(i))+λ2ΣLl=1∥w[l]∥2F where ∥w[l]∥2F=Σn[l]i=1Σn[l−1]j=1(w[l]ij)2

Deal with Overfitting: Dropout

Deal with Overfitting

• Huge number of parameters, even with large amount of training data, there is a potential of overfitting
  • Overfitting due to size of the NN (i.e. total number of parameters)
  • Overfitting due to using the training data for too many epochs
• Solution for overfitting due to NN size
  • Dropout: randomly dropout some proportion (such as 0.3 or 0.5) of nodes at each layer, which is similar to random forest concept
  • Using L1 or L2 regularization in the activation function at each layer
• Solution for overfitting due to using too many epochs
  • Run NN with large number of epochs to reach overfitting region through cross validation from training/validation vs. epoch curve
  • Choose the model with number of epochs that have the minimum validation accuracy as the final NN model
  • The optimal number for epoch is determined by when the model is not overfitted (i.e. validation accuracy reaches the best performance)

Exponentially Weighted Averages

Suppose we have the following 100 days’ temperature data:

θ1=49F,θ2=53F,…,θ99=70F,θ100=69F

The weighted average is defined as:

Vt=βVt−1+(1−β)θt

And we have:

V0=0V1=βV0+(1−β)θ1V2=βV1+(1−β)θ2⋮V100=βV99+(1−β)θ100

Corrected Exponentially Weighted Averages

Vcorrectedt=Vt1−βt

For example,

β=0.95V0=0V1=0.05θ1Vcorrected1=V11−0.95=θ1V2=0.95V1+0.05θ1=0.0475θ1+0.05θ2Vcorrected2=V21−0.952=0.4872θ1+0.5128θ2

With v.s. Without Correction

Momentum

On iteration t, compute dw, db using samples in one mini-batch and update the parameters as follows:

Vdw=βVdw+(1−β)dw

Vdb=βVdb+(1−β)db

w=w−αVdw;  b=b−αVdb

• An overview of gradient descent optimization algorithms

RMSprop

It was proposed by Geoffrey Hinton at his Neural Networks Coursera course

On iteration t, compute dw, db using the current mini-batch.

Sdw=βSdw+(1−β)dw2

Sdb=βSdb+(1−β)db2

w=w−αdw√Sdw; b=b−αdb√Sdb

Adaptive Moment Estimation (Adam)

On iteration t, compute dw, db using the current mini-batch.

{Vdw=β1Vdw+(1−β1)dwVdb=β1Vdb+(1−β1)dbmomantum update β1

{Sdw=β2Sdw+(1−β2)dw2Sdb=β2Sdb+(1−β2)db2RMSprop update β2

{Vcorrecteddw=Vdw1−βt1Vcorrecteddb=Vdb1−βt1;  {Scorrecteddw=Sdw1−βt2Scorrecteddb=Sdb1−βt2

w=w−αVcorrecteddw√Scorrecteddw+ϵ; b=b−αVcorrecteddb√Scorrecteddb+ϵ

Recap of A Few Key Concepts

• Data: Require large well-labeled dataset
• Computation: intensive matrix-matrix operation
• Structure of fully connected feedforward NN
  • Size of the NN: total number of parameters
  • Depth: total number of layers (this is where deep learning comes from)
  • Width of a particular layer: number of nodes (i.e. neurons) in that layer
• Activation function
  • Intermediate layers
  • Last layer connecting to outputs
• Loss function
  • Classification (i.e. categorical response)
  • Regression (i.e. continuous response)
• Optimization methods (SGD)
  • Batch size
  • Learning rate
  • Epoch
• Deal with overfitting
  • Dropout
  • Regularization (L1 or L2)

Use Keras R Package

1. Data preprocessing (from image to list of input features)
   • One image of 28x28 grey scale value matrix → 784 column of features
   • Scale the value to between 0 and 1, by divide each value by 255
   • Make response categorical (i.e. 10 columns with the corresponding digit column 1 and rest columns zero.
2. Load keras package and build a neural network with a few layers
   • Define a placeholder object for the NN structure
   • 1st layer using 256 nodes, fully connected, using ‘relu’ activation function and connect from the input 784 features
   • 2nd layer using 128 nodes, fully connected, using ‘relu’ activation function
   • 3rd layer using 64 nodes, fully connected, using ‘relu’ activation function
   • 4th layer using 10 nodes, fully connected, using ‘softmax’ activation function and connect to the output 10 columns
   • Add drop out to the first three layers to prevent overfitting

3. Compile the NN model, define loss function, optimizer, and metrics to follow

4. Fit the NN model using the training dataset, define epoch, mini batch size, and validation size used in the training where the metrics will be checked

5. Predict using the fitted NN model using the testing dataset

R Scripts

Some links

• Introduction to Data Science (still writing): the content of the workshop is in this book
• Awesome-Data-Science-Materials
• Online books:
  • The Elements of Statistical Learning
  • An Introduction to Statistical Learning
• Hard copy books:
  • Applied Predictive Modeling
  • R for Marketing Research and Analytics