Model Distillation

Model Distillation is the process of taking a big model or ensemble of models and producing a smaller model that captures most of the performance of the original bigger model. It could be also better be described as a blind model replication method.

The reasons for doing so are:

1. improved run-time performance (FLOP operations)
2. (maybe) better generalization because of the model simplicity
3. you don't have access to the training of the original model.
4. you have access to a remotely deployed model and you want to replicate it (it happens more than you can imagine)
5. original model maybe is too complicated
6. insights that may arise from the process itself

How it works

Assume a MNIST classifier \(F_{MNIST}\) composed of an ensemble of \(N\) convolutional deep neural networks that produces a logit \(z_i\) which is then converted to a probability of an input image, \(x_i\), for each of the possible labels \(C_{0}-C_{9}\).

The distillation process will give us an  \(F_{MNIST_{distilled}}\) composed of a single deep neural network that will approximate the classification results of the bigger ensemble of models.

In distillation, knowledge is transferred from the teacher model to the student by minimizing a loss function in which the target is the distribution of class probabilities predicted by the teacher model. That is - the output of a softmax function on the teacher model's logits.

Logits \(z_j\) are converted to probabilities \(P(C_i|x)\) using the softmax layer:

 \(
p_i = \frac
{exp(z_i)}
{\sum_{j}exp(z_j)}
\)

However, in many cases, this probability distribution has the correct class at a very high probability, with all other class probabilities very close to 0. As such, it doesn't provide much information beyond the ground truth labels already provided in the dataset.

To tackle this issue, Hinton et al., 2015 introduced the concept of "softmax temperature". The probability \(q_i\) is computer by the logit \(z_i\) for the scalar softmax temperature \(T\):

 \(
q_i = \frac
{exp(\frac{z_i}{T})}
{\sum_{j}exp(\frac{z_j}{T})}
\)

where T is a temperature that is normally set to 1. Using a higher value for T produces a softer probability distribution over classes. Softer probability distribution means that the values are somewhat diffused and a 0.999 probability may become 0.9 and the rest spread to the other classes.

In the simplest form of distillation, knowledge is transferred to the distilled model by training it on a transfer set and using a soft target distribution for each case in the transfer set that is produced by using the cumbersome model with a high temperature in its softmax. The same high temperature is
used when training the distilled model, but after it has been trained it uses a temperature of 1. When the correct labels are known for all or some of the transfer set, this method can be significantly improved by also training the distilled model to produce the correct labels. One way to do this is to use the correct labels to modify the soft targets, but we found that a better way is to simply use a weighted average of two different objective functions.

1. The first objective function is the cross entropy with the soft targets and this cross entropy is computed using the same high temperature in the softmax of the distilled model as was used for generating the soft targets from the cumbersome model. 
2. The second objective function is the cross entropy with the correct labels. This is computed using exactly the same logits in softmax of the distilled model but at a temperature of 1. 

This very simple operation can have a multitude of knobs and parameters to adjust but the core essence is very simple and works quite well.

If you don't define it then how can you understand it ?

A model's representational capacity is its ability to fit a wide variety of functions. Models with low capacity may struggle to fit the training set (high training error). Models with high capacity can overfit by memorizing properties of the training set that do not serve them well on the test set.

Overfitting is the situation where a learning algorithm achieves low training error but high test error. Overfitting is sign of poor generalization.

Simpler models (smaller hypothesis space and smaller capacity) are more likely to generalize (small gap between training and test error) however complex models are more likely to achieve low training error.

In practise the learning algorithm may not be able to find the best model among the model's hypothesis space. This additional limitations such as the imperfection of the optimization algorithm mean that the learning's algorithm effective capacity may be less than the representational capacity of the model family.

Statistical learning theory provides a way to quantify a model's capacity. The Vapnik-Chervonenkis dimension or VC dimension measures the capacity of a binary classifier. It is defined as being the largest possible value of m for which there exists a training set of m different x points that the classifier can label arbitrarily.

Thus the discrepancy between training error and generalization error is bounded from above by the quantity that grows as the model capacity grows but shrinks as the number of training examples increases.

Regularization is any modification we make to a learning algorithm that is intended to reduce its generalization error but not its training error. Without regularization any search on the hyperparameters of a model would result on those that maximize the model's capacity resulting in overfitting.

Bias and variance measure two different sources of error in an estimator. 

Bias measures the expected deviation from the true value of the function or parameter. The bias is error from erroneous assumptions in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting).

Variance provides a measure of the deviation from the expected estimator value that any particular sampling of the data is likely to cause. The variance is error from sensitivity to small fluctuations in the training set. High variance can cause overfitting: modeling the random noise in the training data, rather than the intended outputs.

The relationship between bias and variance is tightly linked to the machine learning concepts of capacity, underfitting and overfitting. When regularization error is measured by Mean Square Error (where bias and variance are meaningful components of generalization error), increasing capacity tends to increase variance and decrease bias.

In the context of deep learning, most regularization strategies are based on regularizing estimators. Regularization of an estimator works by trading increased bias for reduced variance. An effective regularizer is one that makes a profitable trade, reducing variance significally while not overly increasing the bias.

* from the book Deep Learning