Basics of Transfer Learning

Reading Time: 2 minutes

What is Transfer Learning

Is a method where a model developed for some tasks is reused and acts as the basis for a model on a second task which is similar in nature. In this way knowledge acquired in solving one problem is applied to a different but related problem.

It is quite popular in deep learning as the cost of resources required to train models is high and also due to the limitations in getting quality and quantity of relevant training datasets. In simple terms the objective is to transfer knowledge and learning gained in in one task and using it for another task.

Examples

In image recognition, an already trained model to recognise dog images can be re used to recognise cats or other animals.


In language processing weights generated in a model to understand general English language can be further reused for a domain specific model without having to train the latter to understand general English language.

Benefits

Reduction in effort and cost in creating new models.

Optimisation that allows rapid progress and improved performance when modelling the second task.


When to use

Transfer Learning can be best leveraged in the following situations.

Availability of data
In cases where there is limited availability of training data for training ML models , pre-trained models can be used

Rapid Development
When the turn around time required to build models is less , transfer learning can be utilised and pre-trained models can be used for further training and tuning for new requirements

Contextual AI
General pre trained models can be used to further train context specific models

Using a pre trained model

Prediction
Get a pre trained model and use it as it is to for new predictions.

Feature Extraction,
The pre-trained model’s final part is build to fit the new datasets. The output of the layer before the final layer is fed as input to a new model. The objective is to use the pre-trained model, to pre-process and get essential features. Pass these features to a new classifier.

Fine-tuning
Use weights from the pre-trained model to initialise weights in a new model.

References

http://research.aimultiple.com/transfer-learning

Written by 

Rohit Jagati is a technologist, experienced in designing enterprise grade solutions. Rohit has been designing and implementing solutions based on Machine Learning, IOT and custom/bespoke applications.