However, scientists did not take long to conclude that a perceptron neural network could only be applied to simple tasks because of its simplicity. At that time, neural networks were being used for simple classification problems, but perceptrons usually failed when faced with more complex datasets. Let's review the first example of Chapter 2, How Neural Networks Learn, (AND) to better understand this issue.

The example consists of an AND function that takes two inputs x1 and x2. This function can be plotted in a two-dimensional chart as follows:

Now, let's examine how the neural network evolves in the training by using the perceptron rule, considering a pair of two weights w1 and w2, initially 0.5, and a bias value of 0.5. Assume that the learning rate η equals 0.2.

After 89 epochs, we find the network to produce values close to the desired output. Since in this example, the outputs are binary (zero or one), we can assume that any value produced by the network that is below 0.5 is considered to be 0 and any value above 0.5 is considered to be 1. So, we can draw a function Y = x₁w₁ + x₂w₂ + b=0.5, with the final weights and bias found by the learning algorithm w1 = 0.562, w2 = 0.5, and b = -0.375, defining the linear boundary as shown in the following chart:

This boundary is a definition of all classifications given by the network. You can see that the boundary is linear, given that the function is linear. Thus, the perceptron network is really suitable for problems whose patterns are linearly separable.

Let's analyze the XOR case, whose chart can be seen in the following figure:

We see that in two dimensions, it is impossible to draw a line to separate the two patterns. What would happen if we tried to train a single-layer perceptron to learn this function? Suppose that we tried; let's see what happened through the following table:

The perceptron just could not find any pair of weights that would drive the error below 0.625. This can be explained mathematically as we have already perceived from the chart that this function cannot be linearly separable in two dimensions. So, what if we add another dimension? Let's see the previous XOR chart in three dimensions:

In three dimensions, it is possible to draw a plane that would separate the patterns, provided that this additional dimension could properly transform the input data. Okay, but now, there is an additional problem: How can we derive this additional dimension since we have only two input variables? One obvious but "workaround" answer would be adding a third variable as a derivation from the two original ones. With this third variable a (derivation), our neural network would probably get the following shape:

Okay, now, the perceptron has three inputs, one of them being a composition of the other two. This also leads to a new question: How should this composition be processed? We can see that this component can act as a neuron, thereby giving the neural network a nested architecture. If so, there would be another new question: How would the weights of this new neuron be trained, since the error is on the output neuron?