Of course, the most basic activation function would be a step function. If the value of x is more than a fixed value, a, then y is either 0 or 1, as shown in the following code:
func step(x) {
if x >= 0 {
return 1
} else {
return 0
}
}
As you can see in the following diagram, the step function is extremely simple; it takes a value and then returns 0 or 1:
This is a very simple function and one that is not particularly useful for deep learning. This is because the gradient of this function is a constant zero, meaning that, when we are doing backpropagation, it will constantly produce zeroes, which results in very little (if any at all) improvement when we are performing backpropagation.