Introduction to Deep Neural Networks

Deep neural networks contain input layers, hidden layers and, output layers.

NAND function example

We will use the above simple neural network with $z=w_1x_1+w_2x_2+w_0$ and the activation function f chosen to be the unit step function U(z) :

$U(z)=\left{\begin{array}{ll}0 & z \leq 0 \ 1 & z>0\end{array}\right.$

Hidden Layer Models

let’s consider a simple 2-dimensional classification task. The training set is made up of 4 points listed below:

$x^{(1)}=(-1,-1), y^{(1)}=1$
$x^{(2)}=(1,-1), y^{(2)}=-1$
$x^{(3)}=(-1,1), y^{(3)}=-1$
$x^{(4)}=(1,1), y^{(4)}=1$

The dataset is illustrated below (blue – positive, red – negative):

For simplicity, y⁽ⁱ⁾ can be either -1 or 1.

$f_1^{(i)},f_2^{(i)}$ denote the output of the hidden layer.

$f_{1}^{(i)}=f\left(w_{01}+\left(w_{11} x_{1}^{(i)}+w_{21} x_{2}^{(i)}\right)\right)$
$f_{2}^{(i)}=f\left(w_{02}+\left(w_{12} x_{1}^{(i)}+w_{22} x_{2}^{(i)}\right)\right)$

The weights of the network are given as follows:

$w_{11}=1, w_{21}=-1, w_{01}=1$
$w_{12}=-1, w_{22}=1, w_{02}=1$

If we consider a set:

$D^{\prime}=\left{\left(\left[f_{1}^{(i)}, f_{2}^{(3)}\right], y^{(i)}\right), i=1,2,3,4\right}$

$f(z)=ReLU\funcapply(z)$ , and $f(z)=\tanh\funcapply(z)$ make $D^\prime$ linearly separable.
$f(z)=ReLU\funcapply(z)$ gives the following results:

$\left(1,1\right),\left(3,0\right),\left(0,3\right),\left(1,1\right)$

$f(z)=\tanh\funcapply(z)$ gives the following results:

$\left(0.76,0.76\right),\left(0.99,-0.76\right),\left(-0.76,0.99\right),\left(0.76,0.76\right)$