Another method to make order matter within neural networks is to give the network some kind of memory. So far, all of our networks have done a forward pass without any memory of what happened before or after the pass. It's time to change that with a recurrent neural network (RNN):

The scheme of an RNN

RNNs contain recurrent layers. Recurrent layers can remember their last activation and use it as their own input:

A recurrent layer takes a sequence as an input. For each element, it then computes a matrix multiplication (W * in), just like a Dense layer, and runs the result through an activation function, such as relu. It then retains its own activation. When the next item of the sequence arrives, it performs the matrix multiplication as before, but this time it also multiplies its previous activation with a second matrix (). The recurrent layer adds the result of both operations together and passes it through the activation function again.

In Keras, we can use a simple RNN as follows:

from keras.layers import SimpleRNN

model = Sequential()
model.add(SimpleRNN(16,input_shape=(max_len,n_features)))
model.add(Dense(1))

model.compile(optimizer='adam',loss='mean_absolute_percentage_error')

The only parameter we need to specify is the size of the recurrent layer. This is basically the same as setting the size of a Dense layer, as SimpleRNN layers are very similar to Dense layers except that they feed their output back in as input. RNNs, by default, only return the last output of the sequence.

To stack multiple RNNs, we need to set return_sequences to True, which we can do by running the following code:

from keras.layers import SimpleRNN

model = Sequential()
model.add(SimpleRNN(16,return_sequences=True,input_shape=(max_len,n_features)))
model.add(SimpleRNN(32, return_sequences = True))
model.add(SimpleRNN(64))
model.add(Dense(1))

model.compile(optimizer='adam',loss='mean_absolute_percentage_error')

You can then fit the model on the generator as before:

model.fit_generator(train_gen,epochs=20,steps_per_epoch=n_train_samples // batch_size, validation_data= val_gen, validation_steps=n_val_samples // batch_size)

As a result of this code, we'll be able to see that a simple RNN does much better than the convolutional model, with a loss of around 1,548,653. You'll remember that previously our loss was at 12,793,928. However, we can do much better using a more sophisticated version of the RNN.