Introduction to Deep Neural Networks

Deep neural networks contain input layers, hidden layers and, output layers.

NAND function example

We will use the above simple neural network with z=w_1x_1+w_2x_2+w_0 and the activation function f chosen to be the unit step function U(z) :

U(z)=\left{\begin{array}{ll}0 & z \leq 0 \ 1 & z>0\end{array}\right.

Hidden Layer Models

let’s consider a simple 2-dimensional classification task. The training set is made up of 4 points listed below:

x^{(1)}=(-1,-1), y^{(1)}=1
x^{(2)}=(1,-1), y^{(2)}=-1
x^{(3)}=(-1,1), y^{(3)}=-1
x^{(4)}=(1,1), y^{(4)}=1

The dataset is illustrated below (blue – positive, red – negative):

For simplicity, y(i) can be either -1 or 1.

f_1^{(i)},f_2^{(i)} denote the output of the hidden layer.

f_{1}^{(i)}=f\left(w_{01}+\left(w_{11} x_{1}^{(i)}+w_{21} x_{2}^{(i)}\right)\right)
f_{2}^{(i)}=f\left(w_{02}+\left(w_{12} x_{1}^{(i)}+w_{22} x_{2}^{(i)}\right)\right)

The weights of the network are given as follows:

w_{11}=1, w_{21}=-1, w_{01}=1
w_{12}=-1, w_{22}=1, w_{02}=1

If we consider a set:

D^{\prime}=\left{\left(\left[f_{1}^{(i)}, f_{2}^{(3)}\right], y^{(i)}\right), i=1,2,3,4\right}

f(z)=ReLU\funcapply(z), and f(z)=\tanh\funcapply(z) make D^\prime linearly separable.
f(z)=ReLU\funcapply(z) gives the following results:


f(z)=\tanh\funcapply(z) gives the following results:


Don’t miss these tips!

We don’t spam! Read our privacy policy for more info.

Open chat
Powered by