As we have seen, neural networks, and convolutional networks in particular, work by tuning the weights of the network as if they were coefficients of a large equation in order to get the correct output given a specific input. The tuning happens through back-propagation to move the weights towards the best solution given the chosen neural net architecture. One of the problems is therefore finding the best initialization values for the weights in the neural network. Libraries such as Keras can automatically take care of that. However, this topic is important enough to be worth discussing this point.

Restricted Boltzmann machines have been used to pre-train the network by using the input as the desired output to make the network automatically learn representations of the input and tune its weights accordingly, and this topic has already been discussed in Chapter 4, Unsupervised Feature Learning.

In addition, there exists many pre-trained networks that offer good results. As we have mentioned, many people have been working on convolutional neural networks and have been getting impressive results, and one can often save time by reutilizing the weights learnt by these networks and applying them to other projects.

The VGG-16 model used in K. Simonyan, A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition arXiv:1409.1556, http://arxiv.org/pdf/1409.1556v6.pdf, is an important model for image recognition. In this model, the input is a fixed 224 x 224 RGB-valued image where the only pre-processing is subtracting the mean RGB-value computed on the training set. We outline the architecture for this network in the attached diagram, and the user can try to implement by himself or herself such a network, but also keep in mind the computationally intensive nature of running such a network. In this network the architecture is as follows:

VGG-16 convolutional neural network architecture by Simonyan and Zisserman.

We also refer the interested reader to another noteworthy example, the AlexNet network, contained in Alex Krizhevsky, Ilya Sutskeve, Geoffrey Hinton, ImageNet Classification with Deep Convolutional Networks, in Advances in Neural Information Processing Systems 25 (NIPS 2012), https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf, that we will not be discussing this here for the sake of brevity, but we invite the interested reader to look at it. We also invite the interested reader to look at https://github.com/fchollet/deep-learning-models for code examples of the VGG-16 and other networks.