If the target dataset is small but of a different type to the original – for example, the original dataset is dog images and the new (target) dataset is flower images – then do the following:
- Slice most of the initial layers of the network
- Add to the remaining pre-trained layers a new fully connected layer that matches the number of classes of the target dataset
- Randomize the weights of the new fully connected layer and freeze all the weights from the pre-trained network
- Train the network to update the weights of the new fully connected layer
Since the dataset is small, overfitting is still a concern here as well. To overcome this, we will keep the weights of the original pre-trained network the same and update only the weights of the new fully connected layer:
Only fine tune the higher level portion of the network. This is because the beginning layers are designed to extract more generic features. In general, the first layer of a convolutional neural network is not specific to a dataset.