Course Website

Types of Neural Network

Figure adapted from slides by Andrew NG, Deep Learning Specialization

A Little Bit of History – Perceptron

• First concept of a simplified brain cell (1943): McCullock-Pitts (MCP) neuron
• Frank Rosenblatt published the first concept of the perceptron learning rule based on MCP (1957)
• Classification of N points into 2 classes: -1 and +1 (i.e., two different colors in the picture below)

• Fun video: https://www.youtube.com/watch?v=cNxadbrN_aI

• Two features: (x1 and x2); linear functions to separate classes; find (w0, w1, w2) such that:

z(i)=w0+w1x(i)1+w2x(i)2

pred(i)={1−1if z(i)>0if z(i)≤0

Perceptron Algorithm

Start with random weights. Set a maximum of epochs of M, for each epoch (permutation):

• For each data point i:
  • Predict class label

  z(i)=w0+w1x(i)1+w2x(i)2

  pred(i)={1−1if z(i)>0if z(i)≤0

  • For every data point, we update the weight based on the prediction correctness, learning rate and feature values.

w0=w0+η(actual(i)−pred(i))

w1=w1+η(actual(i)−pred(i))x(i)1 w2=w2+η(actual(i)−pred(i))x(i)2

• Calculate accuracy for the entire dataset to see whether it meets the criteria after each epoch.

Perceptron Algorithm

• Perceptron algorithm is easy to implement in any modern programming language.
  • Perceptron R notebook

• It is a linear classification function, and the weight is updated after we feed each data point to the algorithm (a concept similar to stochastic gradient descent).

• The algorithm continues to update when we feed the same data set again and again (i.e., epochs)

• It does not solve non-linearly spreadable problems.

Logistic Regression as A Neural Network

• m training samples: {(x(1),y(1)),(x(2),y(2)),...,(x(m),y(m))}

X=[x(1)1x(2)1⋯x(m)1x(1)2x(2)2⋯x(m)2⋮⋮⋮⋮x(1)nxx(2)nx…x(m)nx]∈Rnx×m

y=[y(1),y(2),…,y(m)]∈R1×m

ˆy(i)=σ(wTx(i)+b) where σ(z)=11+e−z

Logistic Regression as A Neural Network

• m training samples: {(x(1),y(1)),(x(2),y(2)),...,(x(m),y(m))}

X=[x(1)1x(2)1⋯x(m)1x(1)2x(2)2⋯x(m)2⋮⋮⋮⋮x(1)nxx(2)nx…x(m)nx]∈Rnx×m

y=[y(1),y(2),…,y(m)]∈R1×m

ˆy(i)=σ(wTx(i)+b) where σ(z)=11+e−z

• Loss function: L(ˆy,y)=−ylog(ˆy)−(1−y)log(1−ˆy)
• Cost function: J(w,b)=1mΣmi=1L(ˆy(i),y(i))=1mΣmi=1{ −y(i)log(ˆy(i))−(1−y(i))log(1−ˆy(i))}
• Goal: Find w,b that mininizes J(w,b)
  • w:=w−α∂J∂w
  • b:=b−α∂J∂b

Forward Propagation

Forward Propagation

Backward Propagation

Stochastic Gradient Descent (SGD)

Neural Network: 0 Hidden Layer Neural Network

Neural Network: 1 Hidden Layer Neural Network

Neural Network: 1 Layer Neural Network

Neural Network: 1 Layer Neural Network

Neural Network: 1 Hidden Layer Neural Network

Across m Samples

Across m Samples

MNIST Dataset

• Contains 70000 handwritten labeled digit images with label (60000 training + 10000 testing)
• Census Bureau employees and American high school students wrote these digits
• Each image is 28x28 pixel in greyscale
• Yann LeCun used convolutional network LeNet to achieve < 1% error rate at 1990s

Image Data

1 Hidden Layer Neural Network Example

Forward and Backward Propagation