If the target dataset is small and similar to the original dataset:
- In this case, replace the last fully connected layer with a new fully connected layer that matches with the number of classes of the target dataset
- Initialize old weights with randomized weights
- Train the network to update the weights of the new, fully connected layer:
Transfer learning can be used as a strategy to avoid overfitting, especially when there is a small dataset.