Compiling Tensorflow 1.15 from source

Following my previous post Compiling Tensorflow under Debian Linux with GPU support and CPU extensions I was trying to build version r1.15 when i stumbled to another peculiar bug. When reaching the end of the compilation you get a lovely error in the form of

from keras.preprocessing import image as image_utils
ImportError: No module named keras.preprocessing

To fix this you you just need to install the following packages prior to compiling




pip install keras_applications==1.0.4 --no-deps
pip install keras_preprocessing==1.0.2 --no-deps
pip install h5py==2.8.0

And that's it you can build again.


bazel build -c opt --config=v1 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

Software Commodification

Something that people outside the software industry don't fully grasp is the increasing pace of commodification. 

You had a Phd in CS 20 years ago ? congrats all your knowledge is a library. 

You were an excellent physicist 10 years ago ? Congrats we can import and run everything you knew without even understanding half of it. 

You studied Artificial Intelligence in depth 15 years ago ? A 17 year old with a weekend in keras can provide better solutions than you. Everything you know is getting obsolete in an increasing rate. 

As software people we are are used to packaging up our knowledge and re-learning new skills (tools, theories, frameworks) every couple of years, because the old have been completely commoditized. 

It is an exciting but tiresome journey that it shouldn't be required by all. However as everything is becoming software there may not be a choice by most people. 

Personally the increasing pace of commodification allows me to cut through the noise and add depth to my knowledge. Knowing that the low hanging fruits have been picked will clear up the space of Artificial Intelligence and Machine Learning.

What if you could push your AI models to be 10 times faster?

Deceptive Easy

Modern Machine Learning (ML) and especially Deep Learning (DL) have become deceptively easy. It is almost trivial to have something up and running with presentable results as the the tools hide most of the complexity and hard decisions. Whereas that might be good enough for a Proof of Concept (POC) or a Minimum Viable Product (MVP), ensuring stability, high performance and scalability is a whole different ball game.

High Sunk Cost 

Many CXO’s and seniors managers find themselves trapped into subpar solutions that cannot be used effectively because they lack the technical know-how to productionalize them.

At Electi we have just release a new brochure listing the services we can offer to companies already implementing Deep Learning. Check it out here.

Performance Measures for Machine Learning

Summary

In this blog spot I'm presenting a few performance measures for machine learning tasks. These performance measures come up a lot in the marketing domain.

Take at home lessons:

• the measure you optimize to makes a difference
• the measure you report makes a difference
• use measure appropriate for problem/community
• accuracy often is not sufficient/appropriate
• only accuracy generalizes to >2 classes
• this is not an exhaustive list of performance measures

 

Confusion Matrix

First we construct a confusion matrix for a binary classification problem. Given a classification function f(x)->R and a threshold T that can split the outcomes into {0, 1} we can create a confusion matrix that counts the occurrences of the predicted class given the true label.

 

Accuracy

Accuracy is then measured as the percentage of correct responses (True Positives + True Negatives) over the total amount of responses.

 

Problems with Accuracy

This measure is commonly used but it can be misleading. The problems arise from the domain we are modelling. If one of the class for example is poorly represented into the metric is meaningless as we could predict the same class always and still have a good results.

• Assumes equal cost for both kinds of errors 

• cost(b-type-error) = cost (c-type-error)

• is 99% accuracy good?

• can be excellent, good, mediocre, poor, terrible
• depends on problem

• Base Rate = accuracy of predicting predominant class

 

Weighted (Cost sensitive) Accuracy

A modified version of accuracy is "Weighted Accuracy" were we count the cost of misclassification.

In this scenario we aiming for a model and a threshold that can minimize the total cost.

 

Lift

If we are not interested in the accuracy on the entire dataset but want accurate predictions for 5%, 10% or 20% of the dataset then we can use the lift measure.

Lift measures how much better than random prediction on the fraction of the dataset predicted true (f(x) > threshold).

 

Precision / Recall

• The Precision measure counts how many of the interest class are correct.
• The Recall measure counts how many of the interest class does the model return.

In the case below the interest class is a(1). 

We can change the sweep over the threshold calculate Precision/Recall multiple times and graph out what is called the Precision/Recall curve.

At each different threshold we can see a different tradeoff between the two metrics.

• When the threshold is too high then c (everything is predicted as class 0) becomes zero and then the precision becomes zero.
• When the threshold is too low then b (everything is predicted as class 1) becomes zero and then the recall becomes zero.

Both of these metrics are flawed in isolation and it is the eye of the modeller on which one better represents the problem.

F-Measure

The F-Measure is an attempt to merge the two measures to construct a more meaningful performance measure.

Receiver Operating Characteristic (ROC)

• Developed in WWII to statistically model false positive and false negative detections of radar operators
• Better statistical foundations than most other measures
• Standard measure in medicine and biology
• Becoming more popular in ML 

Although ROC graphs are apparently simple, there are some common misconceptions and pitfalls when using them in practice.

One of the earliest adopters of ROC graphs in machine learning was Spackman (1989), who demonstrated the value of ROC curves in evaluating and comparing algorithms.

ROC graphs are conceptually simple, but there are some non-obvious complexities that arise when they are used in research. 

ROC Plot

 

• Sweep threshold and plot

• TPR vs. FPR
• Sensitivity vs. 1-Specificity
• P(true|true) vs. P(true|false)

• Sensitivity = a/(a+b) = Recall = LIFT numerator
• 1 - Specificity = 1 - d/(c+d)

A ROC graph depicts relative trade-offs between benefits (true positives) and costs (false positives).

• The lower left point (0,0) represents the strategy of never issuing a positive classiffication. 
• The opposite strategy is represented by the upper right point (1,1).
• The point (0,1) represents perfect classiffication.
• The diagonal line y = x represents the strategy of randomly guessing a class. 
• A random classifier will produce an ROC point that "slides" back and forth on the diagonal based on the frequency with which it guesses the positive class. In order to get away from this diagonal into the upper triangular region, the classifier must exploit some information in the data. 
• Any classifier that appears in the lower right triangle performs worse than random guessing. This triangle is therefore usually empty in ROC graphs.
• ROC curves have an attractive property: they are insensitive to changes in class distribution.
• Any performance metric that uses values from both columns of theconfusion matrix will be inherently sensitive to class skews. Metrics such as accuracy, precision, lift and F scores use values from both
  columns of the confusion matrix. 
• ROC graphs are based upon TP rate and FP rate, in which each dimension is a strict columnar ratio, so do not depend on class distributions.

Identity Mappings in Deep Residual Networks, 2016

A very nice improvement over the original ResNet.

In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation.

Machine Learning Engineering : Tests for Infrastructure

An ML system often relies on a complex pipeline rather than a single running binary.

Engineering checklist:

1. Test the reproducibility of training
2. Unit test model specification code
3. Integration test the full ML pipeline
4. Test model quality before attempting to serve it
5. Test that a single example or training batch can be sent to the model
6. Test models via a canary process before they enter production serving environments
7. Test how quickly and safely a model can be rolled back to a previous serving version

1. Test the reproducibility of training. Train two models on the same data, and observe any differences in aggregate metrics, sliced metrics, or example-by-example predictions. Large differences due to non-determinism can exacerbate debugging and troubleshooting.

2. Unit test model specification code. Although model specifications may seem like “configuration”, such files can have bugs and need to be tested. Useful assertions include testing that training results in decreased loss and that a model can restore from a checkpoint after a mid-training job crash.

3. Integration test the full ML pipeline. A good integration test runs all the way from original data sources, through feature creation, to training, and to serving. An integration test should run both continuously as well as with new releases of models or servers, in order to catch problems well before
they reach production.

4. Test model quality before attempting to serve it. Useful tests include testing against data with known correct outputs and validating the aggregate quality, as well as comparing predictions to a previous version of the model.

5. Test that a single example or training batch can be sent to the model, and changes to internal state can be observed from training through to prediction. Observing internal state on small amounts of data is a useful debugging strategy for issues like numerical instability.

6. Test models via a canary process before they enter production serving environments. Modeling code can change more frequently than serving code, so there is a danger that an older serving system will not be able to serve a model trained from newer code. This includes testing that a model can be loaded into the production serving binaries and perform inference on production input data at all. It also includes a canary process, in which a new version is tested on a small trickle of live data.


7. Test how quickly and safely a model can be rolled back to a previous serving version. A model “roll back” procedure is useful in cases where upstream issues might result in unexpected changes to model quality. Being able to quickly revert to a previous known-good state is as crucial with ML models as with any other aspect of a serving system.

* From  "What’s your ML Test Score? A rubric for ML production systems" NIPS, 2016

Machine Learning Engineering : Tests for Model Development

While the field of software engineering has developed a full range of best practices for developing reliable software systems, the set of standards and practices for developing ML models in a rigorous fashion is still developing. It can be all too tempting to rely on a single-number summary metric to judge performance, perhaps masking subtle areas of unreliability. Careful testing is needed to search for potential lurking issues.

Engineering checklist:

1. Test that every model specification undergoes a code review and is checked in to a repository
2. Test the relationship between offline proxy metrics and the actual impact metrics
3. Test the impact of each tunable hyperparameter
4. Test the effect of model staleness. Concept drift is real for non stationary processes
5. Test against a simpler model as a baseline
6. Test model quality on important data slices
7. Test the model for implicit bias

1. Test that every model specification undergoes a code review and is checked in to a repository.
It can be tempting to avoid, but disciplined code review remains an excellent method for avoiding
silly errors and for enabling more efficient incident response and debugging.

2. Test the relationship between offline proxy metrics and the actual impact metrics. For exam-
ple, how does a one-percent improvement in accuracy or AUC translate into effects on metrics of
user satisfaction, such as click through rates? This can be measured in a small scale A/B experiment
using an intentionally degraded model.

3. Test the impact of each tunable hyperparameter. Methods such as a grid search or a more
sophisticated hyperparameter search strategy not only improve predictive performance, but also
can uncover hidden reliability issues. For example, it can be surprising to observe the impact of
massive increases in data parallelism on model accuracy.

4. Test the effect of model staleness. If predictions are based on a model trained yesterday versus
last week versus last year, what is the impact on the live metrics of interest? All models need to be
updated eventually to account for changes in the external world; a careful assessment is important to
guide such decisions.

5. Test against a simpler model as a baseline. Regularly testing against a very simple baseline
model, such as a linear model with very few features, is an effective strategy both for confirming
the functionality of the larger pipeline and for helping to assess the cost to benefit tradeoffs of more
sophisticated techniques.

6. Test model quality on important data slices. Slicing a data set along certain dimensions of
interest provides fine-grained understanding of model performance. For example, important slices
might be users by country or movies by genre. Examining sliced data avoids having fine-grained
performance issues masked by a global summary metric.

7. Test the model for implicit bias. This may be viewed as an extension of examining important data
slices, and may reveal issues that can be root-caused and addressed. For example, implicit bias might
be induced by a lack of sufficient diversity in the training data.

* From  "What’s your ML Test Score? A rubric for ML production systems" NIPS, 2016

Machine Learning Engineering : Tests for Features and Data

Machine learning systems differ from traditional software-based systems in that the behavior of ML systems is not specified directly in code but is learned from data. Therefore, while traditional software can rely on unit tests and integration tests of the code, here we attempt to add a sufficient set of tests of the data.

Engineering checklist:

1. Test that the distributions of each feature match your expectations
2. Test the relationship between each feature and the target 
3. Test the cost of each feature
4. Test that a model does not contain any unsuitable for use feature
5. Test that your system maintains privacy controls across its entire data pipeline
6. Test all code that creates input features

1. Test that the distributions of each feature match your expectations. One example might be to test that Feature A takes on values 1 to 5, or that the two most common values of Feature B are "Harry" and "Potter" and they account for 10% of all values. This test can fail due to real external changes, which may require changes in your model.

2. Test the relationship between each feature and the target, and the pairwise correlations between individual signals. It is important to have a thorough understanding of the individual features used in a given model; this is a minimal set of tests, more exploration may be needed to develop a full understanding. These tests may be run by computing correlation coefficients, by training models with one or two features, or by training a set of models that each have one of k features individually removed.

3. Test the cost of each feature. The costs of a feature may include added inference latency and RAM usage, more upstream data dependencies, and additional expected instability incurred by relying on that feature. Consider whether this cost is worth paying when traded off against the provided improvement in model quality.

4. Test that a model does not contain any features that have been manually determined as unsuitable for use. A feature might be unsuitable when it’s been discovered to be unreliable, overly expensive, etc. Tests are needed to ensure that such features are not accidentally included (e.g. via copy-paste) into new models.

5. Test that your system maintains privacy controls across its entire data pipeline. While strict access control is typically maintained on raw data, ML systems often export and transform that data during training. Test to ensure that access control is appropriately restricted across the entire pipeline. Test the calendar time needed to develop and add a new feature to the production model. The faster a team can go from a feature idea to it running in production, the faster it can both improve the system and respond to external changes.

6. Test all code that creates input features, both in training and serving. It can be tempting to believe feature creation code is simple enough to not need unit tests, but this code is crucial for correct behavior and so its continued quality is vital.

* From  "What’s your ML Test Score? A rubric for ML production systems" NIPS, 2016

What is deep learning ?

Long Story Short

I like this definition, clean, consise and to the point with zero marketing fluff.

“A class of parametrized non-linear representations encoding appropriate domain knowledge (invariance and stationarity) that can be (massively) optimized efficiently using stochastic gradient descent”

What neural networks actually do ?