ReLU function 

Then there is an activation function called the Rectified Linear Unit, ReLU(z), that transforms any value, z, to 0 or a value above 0. In other words, it outputs any value below 0 as 0 and any value above 0 as the value itself:

Just to summarize our understanding so far, the perceptron is the traditional and outdated neuron that is rarely used in real implementations. They are great to get a simplistic understanding of the underlying principle; however, they had the problem of fast learning due to the drastic changes in output values.

We use activation functions to reduce the learning speed and determine finer changes in z or  . Let's sum up these activation functions:

• The sigmoid neuron is the neuron that uses the sigmoid activation function to transform the output to a value between 0 and 1.
• The tanh neuron is the neuron that uses the tanh activation function to transform the output to a value between -1 and 1.
• The ReLU neuron is the neuron that uses the ReLU activation function to transform the output to a value of either 0 or any value above 0.

The sigmoid function is used in practice but is slow compared to the tanh and ReLU functions. The tanh and ReLU functions are commonly used activation functions. The ReLU function is also considered state of the art and is usually the first choice of activation function that's used to build ANNs.

Here is a list of commonly used activation functions:

In the projects within this book, we will be primarily using either the sigmoid, tanh, or the ReLU neurons to build our ANN.