Tuesday, February 21, 2017

The Black Magic of Deep Learning - Tips and Tricks for the practitioner


Spirits guide us to find the correct hyperparameters
I first heard of Deep Learning in 2012 when they gained traction against traditional methods. I followed their evolution but thought it was mostly hype.
Then in January 2015 I was involved in a green field project and I was in charge of deciding the core Machine Learning algorithms to be used in a computer vision platform.

Nothing worked good enough and if it did it wouldn't generalize, required fiddling all the time and when introduced to similar datasets it wouldn't perform as well. I was lost. I needed what Deep Learning promised but I was skeptical, so I read the papers, the books and the notes. I then went and put to work everything I learned.

Suprisingly, it was no hype, Deep Learning works and it works well. However it is such a new concept (even though the foundations were laid in the 70's) that a lot of anecdotal tricks and tips started coming out on how to make the most of it (Alex Krizhevsky covered a lot of them and in some ways pre-discovered batch normalization).

Anyway to sum, these are my tricks (that I learned the hard way) to make DNN tick.
  • Always shuffle. Never allow your network to go through exactly the same minibatch. If your framework allows it shuffle at every epoch. 
  • Expand your dataset. DNN's need a lot of data and the models can easily overfit a small dataset. I strongly suggest expanding your original dataset. If it is a vision task, add noise, whitening, drop pixels, rotate and color shift, blur and everything in between. There is a catch though if the expansion is too big you will be training mostly with the same data. I solved this by creating a layer that applies random transformations so no sample is ever the same. If you are going through voice data shift it and distort it
  • This tip is from Karpathy, before training on the whole dataset try to overfit on a very small subset of it, that way you know your network can converge.
  • Always use dropout to minimize the chance of overfitting. Use it after large > 256 (fully connected layers or convolutional layers). There is an excellent thesis about that (Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning)
  • Avoid LRN pooling, prefer the much faster MAX pooling.
  • Avoid Sigmoid's , TanH's gates they are expensive and get saturated and may stop back propagation. In fact the deeper your network the less attractive Sigmoid's and TanH's are. Use the much cheaper and effective ReLU's and PreLU's instead. As mentioned in Deep Sparse Rectifier Neural Networks they promote sparsity and their back propagation is much more robust.
  • Don't use ReLU or PreLU's gates before max pooling, instead apply it after to save computation
  • Don't use ReLU's they are so 2012. Yes they are a very useful non-linearity that solved a lot of problems. However try fine-tuning a new model and watch nothing happen because of bad initialization with ReLU's blocking backpropagation. Instead use PreLU's with a very small multiplier usually 0.1. Using PreLU's converges faster and will not get stuck like ReLU's during the initial stages. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. ELU's are still good but expensive.
  • Use Batch Normalization (check paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift) ALWAYS. It works and it is great. It allows faster convergence ( much faster) and smaller datasets. You will save time and resources.
  • I don't like removing the mean as many do, I prefer squeezing the input data to [-1, +1]. This is more of  a training and deployment trick rather a performance trick.
  • Always go for the smaller models, if you are working and deploying deep learning models like me, you quickly understand the pain of pushing gigabytes of models to your users or to a server in the other side of the world. Go for the smaller models even if you lose some accuracy.
  • If you use the smaller models try ensembles. You can usually boost your accuracy by ~3% with an enseble of 5 networks. 
  • Use xavier initialization as much as possible. Use it only on large Fully Connected layers and avoid them on the CNN layers. An-explanation-of-xavier-initialization
  • If your input data has a spatial parameter try to go for CNN's end to end. Read and understand SqueezeNet , it is a new approach and works wonders, try applying the tips above. 
  • Modify your models to use 1x1 CNN's layers where it is possible, the locality is great for performance. 
  • Don't even try to train anything without a high end GPU.
  • If you are making templates out of models or your own layers, parameterize everything otherwise you will be rebuilding your binaries all the time. You know you will
  • And last but not least understand what you are doing, Deep Learning is the Neutron Bomb of Machine Learning. It is not to be used everywhere and always. Understand the architecture you are using and what you are trying to achieve don't mindlessly copy models.  
To get the math behind DL read Deep-Learning-Adaptive-Computation-Machine.
It is an excellent book and really clears things up. There is an free pdf on the net. But buy it to support the authors for their great work.
For a history lesson and a great introduction read Deep Learning: Methods and Applications (Foundations and Trends in Signal Processing) 
If your really want to start implementing from scratch, check out Deep Belief Nets in C++ and CUDA C, Vol. 1: Restricted Boltzmann Machines and Supervised Feedforward Networks
  
Suggested reading
If you are need help with you Deep Learning / Machine Learning pipeline contact us at  Electi Consulting

10 comments:

  1. He has been practicing for a long time and offering solutions to the problems of people who are suffering in their lives and feeling depressed. Consult this Famous Indian Astrologer in Toronto pandit rudra ji to get rid of all your hurdles.
    Famous Indian Astrologer in Toronto

    ReplyDelete
  2. I appreciate the wondrous work you did for me LORD ZAKUZA by getting my ex lover back and I know people out there will need your help also. He's spells are magnificent, powerful and permanent. To get your ex back now, WhatsApp or call LORD ZAKUZA for urgent response through this number +1 740 573 9483.

    Email: lordzakuza7@gmail.com
    Website: lordzakuzaspells.com

    ReplyDelete
  3. Please visit
    (http://dradodalovetemple.com)
    (https://youtu.be/B279mkEo_C4)
    i am here to give testimony on how this powerful spell caster called Dr Adoda helped me to fix my relationship. i was heartbroken when my fiancee told me he is no longer interested in marrying me because he was having an affair with the other woman where he work. I cried and sobbed every day, until it got so bad that I reached out to the Internet for help, that was when i read a review about the great work of Dr Adoda, then i contacted him for help to get my lover back, he helped me cast a powerful (reconciliation love spell) and to my greatest surprise after 48 hours of doing the love spell my Fiancee came back on his knees begging me to forgive him We are now happily married and all thanks goes to Dr Adoda Wonders for helping me save my relationship you can contact Dr Adoda through his email address: adodalovespelltemple440@gmail.com or WHATSAPP him on+2348112825421

    ReplyDelete
  4. Pandith Rudra is a renowned Famous, Top and Best Indian Astrologer in New York who provides astrological services such as Black Magic Removal, Psychic Reading, Ex Love Back, Husband & wife Relationship Problems, Love Specialist and Jealousy and Curse Removal. His followers have earned more benefits through his best forecasting and even after their problem solved, they meet him and ask him some general questions as a family member.
    Top Astrologer in New York

    ReplyDelete
  5. Hi everyone I'm Brooks Leon and am here to share the wonderful work Dr Ayola did for me. After 8 years in marriage with my Wife with 3 kids, my Wife started acting weird and going out with other Guys and showed me cold love, on several occasions she threatens to divorce me if I dare question her about his affair with other Guys, I was totally devastated and confused until a old friend of mine told me about a spell caster on the internet called Dr. Ayoola who help people with relationship and marriage problem by the powers of love spells, at first I doubted if such thing ever exists but decided to give it a try, when I contact him, he helped me cast a love spell and within 48hours my wife came back to me and started apologizing, now she has stopped going out with other Guys and she with me for good and for real. Contact this great love spell caster for your relationship or marriage problem to be solved today via email: drayoolasolutionhome@gmail.com He is the best spell caster that can help you with your problems or call him on whatsapp +31684065675 or website https://drayoolasolutionhome.com/

    ReplyDelete
  6. My name is Jelena Dorian I had struggle relationship with my husband in the past which led to divorce with my first husband. The memories of my husband was still in me and i realized how much i loved him and have missed him. I wept bitterly that night thinking i have lost the man that i have had so much love for. I asked for advise on what to do and a friend of mine gave me a contact of DR ISIKOLO, i consulted him and to his very best with his powerful spells he helped me with a reunion marriage love spell to returned my ex husband back to me, in just 2 days i had encounter with a spiritual reunion prayers with DR ISIKOLO everything turn around for good in my life, I am now happily living with my husband again and sort out the divorce issues. I testify here today that DR ISIKOLO is powerful it can resolve all problem in a broken relationship with love once. contact him for solutions to any problems. email him at: isikolosolutionhome@gmail.com You can also WhatsApp him on +1(725)240-4536

    ReplyDelete
  7. heartbreak is so frustrating, my wife abandoned me for another man for so many years, i was heartbroken and it was the worse experience of my life, but i thank god for using Dr Charles to bring her back to me, Dr Charles spell caster is the best i have ever seen, he helped me brought back my wife within 3 days, i promise to share a great testimony to the world how he helped me, if you are going through any problem in your relationship or marriage he can help you and he does any other spell as well. 

    his Email drcharlesspeltemple@enshiangs.com
    website:   https://drcharlesspeltempl.wixsite.com/info   
    WhatsApp: +16029609776

    ReplyDelete
  8. I was reading some of your content on this website and I conceive this internet site is really informative ! Keep on putting up. voodoo services

    ReplyDelete
  9. Contact Prof Remmy at psychicremmy@gmail.com if you want to get your Husband, Wife, Boyfriend and Girlfriend back now and regain your happiness. Hi i am Bianca from USA I have just experience the wonders of Prof. Remmy love spell, that have been spread on the internet and worldwide, How he marvellously helped people all over the world to restored back their marriage life and get back lost lovers, and also help to win lottery. I contacted him after going through so many testimonies from different people how he help to bring back ex-lover back, i told him about my husband that abandoned me about 8 months ago, and left home with all i had.. Prof Remmy only told me to smile and have a rest of mind he will handle all in just 24 hours, After the second day my husband called me, i was just so shocked, i pick the call and couldn't believe my ears, he was really begging me to forgive him and making promises on phone.. He came back home and also got me a new car just for him to proof his love for me. i was so happy and called Prof Remmy and thanked him, he only told me to share the good news all over the world .. Well if you need an effective and real spell caster contact Via email: psychicremmy@gmail.com Call +27660670249 or WhatsApp: +1-517-201-7336 or visit :https://www.psychicremmy.com/

    ReplyDelete
  10. Very interesting information!Perfect just what I was looking for! “Peace, commerce and honest friendship with all nations entangling alliances with none.

    Astrologer in Washington,
    Marriage Problems Solution,
    Astrologer and Spiritual Healer,

    ReplyDelete