Tuesday, February 21, 2017

The Black Magic of Deep Learning - Tips and Tricks for the practitioner


Spirits guide us to find the correct hyperparameters
I first heard of Deep Learning in 2012 when they gained traction against traditional methods. I followed their evolution but thought it was mostly hype.
Then in January 2015 I was involved in a green field project and I was in charge of deciding the core Machine Learning algorithms to be used in a computer vision platform.

Nothing worked good enough and if it did it wouldn't generalize, required fiddling all the time and when introduced to similar datasets it wouldn't perform as well. I was lost. I needed what Deep Learning promised but I was skeptical, so I read the papers, the books and the notes. I then went and put to work everything I learned.

Suprisingly, it was no hype, Deep Learning works and it works well. However it is such a new concept (even though the foundations were laid in the 70's) that a lot of anecdotal tricks and tips started coming out on how to make the most of it (Alex Krizhevsky covered a lot of them and in some ways pre-discovered batch normalization).

Anyway to sum, these are my tricks (that I learned the hard way) to make DNN tick.
  • Always shuffle. Never allow your network to go through exactly the same minibatch. If your framework allows it shuffle at every epoch. 
  • Expand your dataset. DNN's need a lot of data and the models can easily overfit a small dataset. I strongly suggest expanding your original dataset. If it is a vision task, add noise, whitening, drop pixels, rotate and color shift, blur and everything in between. There is a catch though if the expansion is too big you will be training mostly with the same data. I solved this by creating a layer that applies random transformations so no sample is ever the same. If you are going through voice data shift it and distort it
  • This tip is from Karpathy, before training on the whole dataset try to overfit on a very small subset of it, that way you know your network can converge.
  • Always use dropout to minimize the chance of overfitting. Use it after large > 256 (fully connected layers or convolutional layers). There is an excellent thesis about that (Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning)
  • Avoid LRN pooling, prefer the much faster MAX pooling.
  • Avoid Sigmoid's , TanH's gates they are expensive and get saturated and may stop back propagation. In fact the deeper your network the less attractive Sigmoid's and TanH's are. Use the much cheaper and effective ReLU's and PreLU's instead. As mentioned in Deep Sparse Rectifier Neural Networks they promote sparsity and their back propagation is much more robust.
  • Don't use ReLU or PreLU's gates before max pooling, instead apply it after to save computation
  • Don't use ReLU's they are so 2012. Yes they are a very useful non-linearity that solved a lot of problems. However try fine-tuning a new model and watch nothing happen because of bad initialization with ReLU's blocking backpropagation. Instead use PreLU's with a very small multiplier usually 0.1. Using PreLU's converges faster and will not get stuck like ReLU's during the initial stages. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. ELU's are still good but expensive.
  • Use Batch Normalization (check paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift) ALWAYS. It works and it is great. It allows faster convergence ( much faster) and smaller datasets. You will save time and resources.
  • I don't like removing the mean as many do, I prefer squeezing the input data to [-1, +1]. This is more of  a training and deployment trick rather a performance trick.
  • Always go for the smaller models, if you are working and deploying deep learning models like me, you quickly understand the pain of pushing gigabytes of models to your users or to a server in the other side of the world. Go for the smaller models even if you lose some accuracy.
  • If you use the smaller models try ensembles. You can usually boost your accuracy by ~3% with an enseble of 5 networks. 
  • Use xavier initialization as much as possible. Use it only on large Fully Connected layers and avoid them on the CNN layers. An-explanation-of-xavier-initialization
  • If your input data has a spatial parameter try to go for CNN's end to end. Read and understand SqueezeNet , it is a new approach and works wonders, try applying the tips above. 
  • Modify your models to use 1x1 CNN's layers where it is possible, the locality is great for performance. 
  • Don't even try to train anything without a high end GPU.
  • If you are making templates out of models or your own layers, parameterize everything otherwise you will be rebuilding your binaries all the time. You know you will
  • And last but not least understand what you are doing, Deep Learning is the Neutron Bomb of Machine Learning. It is not to be used everywhere and always. Understand the architecture you are using and what you are trying to achieve don't mindlessly copy models.  
To get the math behind DL read Deep-Learning-Adaptive-Computation-Machine.
It is an excellent book and really clears things up. There is an free pdf on the net. But buy it to support the authors for their great work.
For a history lesson and a great introduction read Deep Learning: Methods and Applications (Foundations and Trends in Signal Processing) 
If your really want to start implementing from scratch, check out Deep Belief Nets in C++ and CUDA C, Vol. 1: Restricted Boltzmann Machines and Supervised Feedforward Networks
  
Suggested reading
If you are need help with you Deep Learning / Machine Learning pipeline contact us at  Electi Consulting

21 comments:

  1. He has been practicing for a long time and offering solutions to the problems of people who are suffering in their lives and feeling depressed. Consult this Famous Indian Astrologer in Toronto pandit rudra ji to get rid of all your hurdles.
    Famous Indian Astrologer in Toronto

    ReplyDelete
    Replies
    1. Hello to everyone out here, I am here to share the unexpected miracle that happened to me … My name is Susan Christian , I live in London, UK. we got married for more than 9 years and have gotten two kids. thing were going well with us and we are always happy. until one day my husband started to behave in a way i could not understand, i was very confused by the way he treat me and the kids. later that month he did not come home again and he called me that he want a divorce, i asked him what have i done wrong to deserve this from him, all he was saying is that he want a divorce that he hate me and do not want to see me again in his life, i was mad and also frustrated do not know what to do, i was sick for more than 2 weeks because of the divorce. i love him so much he was everything to me without him my life is incomplete. i told my sister and she told me to contact a spell caster, i never believe in all this spell casting of a thing. i just want to try if something will come out of it. i contacted Dr Emu for the return of my husband to me, they told me that my husband have been taken by another woman, that she cast a spell on him that is why he hate me and also want us to divorce. then they told me that they have to cast a spell on him that will make him return to me and the kids, they casted the spell and after 24 hours my husband called me and he told me that i should forgive him, he started to apologize on phone and said that he still live me that he did not know what happen to him that he left me. it was the spell that he Dr Emu casted on him that make him come back to me today, me and my family are now happy again today. thank you Dr Emu for what you have done for me i would have been nothing today if not for your great spell. i want you my friends who are passing through all this kind of love problem of getting back their husband, wife , or ex boyfriend and girlfriend to contact Dr Emu ,if you need his help you can contact him through his private mail: emutemple@gmail.com or you can contact him through his website https://emutemple.wordpress.com/ fb page Https://web.facebook.com/Emu-Temple-104891335203341 and you will see that your problem will be solved without any delay.

      Delete
    2. DR. JOROMI IS TRULY A POWERFUL MAN. Contacting Dr Joromi is the best thing that has ever happened to me. My husband left me for another woman for the reason which i know nothing about. I tried every means to get him back but to no avail, all my efforts proved abortive. I was even contemplating suicide. On a lucky day when i was browsing the internet, i came across some amazing testimonies about Dr. Joromi on how he has helped a lot of people with his powerful love spells. I contacted him and explained all my problems to him and he assured me that he will help me. He told me what to do and assured me that my man will be back in 12 hours. I obeyed him and followed the instructions and to my Greatest surprise, my man called me in 12 hours time just as Dr. Joromi said. Thank you Dr Joromi. May the good God continue to bless you. If you are in need of help please contact Dr Joromi now on his website: https://www.joromispells.com and you can also reach him with his email: Joromispells@gmail.com and here is his WhatsApp number which you can also use to reach him +2348138695387 Thank you Dr Joromi. Thank you once again

      Delete
  2. I appreciate the wondrous work you did for me LORD ZAKUZA by getting my ex lover back and I know people out there will need your help also. He's spells are magnificent, powerful and permanent. To get your ex back now, WhatsApp or call LORD ZAKUZA for urgent response through this number +1 740 573 9483.

    Email: lordzakuza7@gmail.com
    Website: lordzakuzaspells.com

    ReplyDelete
  3. Please visit
    (http://dradodalovetemple.com)
    (https://youtu.be/B279mkEo_C4)
    i am here to give testimony on how this powerful spell caster called Dr Adoda helped me to fix my relationship. i was heartbroken when my fiancee told me he is no longer interested in marrying me because he was having an affair with the other woman where he work. I cried and sobbed every day, until it got so bad that I reached out to the Internet for help, that was when i read a review about the great work of Dr Adoda, then i contacted him for help to get my lover back, he helped me cast a powerful (reconciliation love spell) and to my greatest surprise after 48 hours of doing the love spell my Fiancee came back on his knees begging me to forgive him We are now happily married and all thanks goes to Dr Adoda Wonders for helping me save my relationship you can contact Dr Adoda through his email address: adodalovespelltemple440@gmail.com or WHATSAPP him on+2348112825421

    ReplyDelete
  4. Pandith Rudra is a renowned Famous, Top and Best Indian Astrologer in New York who provides astrological services such as Black Magic Removal, Psychic Reading, Ex Love Back, Husband & wife Relationship Problems, Love Specialist and Jealousy and Curse Removal. His followers have earned more benefits through his best forecasting and even after their problem solved, they meet him and ask him some general questions as a family member.
    Top Astrologer in New York

    ReplyDelete
  5. Hi everyone I'm Brooks Leon and am here to share the wonderful work Dr Ayola did for me. After 8 years in marriage with my Wife with 3 kids, my Wife started acting weird and going out with other Guys and showed me cold love, on several occasions she threatens to divorce me if I dare question her about his affair with other Guys, I was totally devastated and confused until a old friend of mine told me about a spell caster on the internet called Dr. Ayoola who help people with relationship and marriage problem by the powers of love spells, at first I doubted if such thing ever exists but decided to give it a try, when I contact him, he helped me cast a love spell and within 48hours my wife came back to me and started apologizing, now she has stopped going out with other Guys and she with me for good and for real. Contact this great love spell caster for your relationship or marriage problem to be solved today via email: drayoolasolutionhome@gmail.com He is the best spell caster that can help you with your problems or call him on whatsapp +31684065675 or website https://drayoolasolutionhome.com/

    ReplyDelete
  6. My name is Jelena Dorian I had struggle relationship with my husband in the past which led to divorce with my first husband. The memories of my husband was still in me and i realized how much i loved him and have missed him. I wept bitterly that night thinking i have lost the man that i have had so much love for. I asked for advise on what to do and a friend of mine gave me a contact of DR ISIKOLO, i consulted him and to his very best with his powerful spells he helped me with a reunion marriage love spell to returned my ex husband back to me, in just 2 days i had encounter with a spiritual reunion prayers with DR ISIKOLO everything turn around for good in my life, I am now happily living with my husband again and sort out the divorce issues. I testify here today that DR ISIKOLO is powerful it can resolve all problem in a broken relationship with love once. contact him for solutions to any problems. email him at: isikolosolutionhome@gmail.com You can also WhatsApp him on +1(725)240-4536

    ReplyDelete
  7. heartbreak is so frustrating, my wife abandoned me for another man for so many years, i was heartbroken and it was the worse experience of my life, but i thank god for using Dr Charles to bring her back to me, Dr Charles spell caster is the best i have ever seen, he helped me brought back my wife within 3 days, i promise to share a great testimony to the world how he helped me, if you are going through any problem in your relationship or marriage he can help you and he does any other spell as well. 

    his Email drcharlesspeltemple@enshiangs.com
    website:   https://drcharlesspeltempl.wixsite.com/info   
    WhatsApp: +16029609776

    ReplyDelete
  8. I was reading some of your content on this website and I conceive this internet site is really informative ! Keep on putting up. voodoo services

    ReplyDelete
  9. Contact Prof Remmy at psychicremmy@gmail.com if you want to get your Husband, Wife, Boyfriend and Girlfriend back now and regain your happiness. Hi i am Bianca from USA I have just experience the wonders of Prof. Remmy love spell, that have been spread on the internet and worldwide, How he marvellously helped people all over the world to restored back their marriage life and get back lost lovers, and also help to win lottery. I contacted him after going through so many testimonies from different people how he help to bring back ex-lover back, i told him about my husband that abandoned me about 8 months ago, and left home with all i had.. Prof Remmy only told me to smile and have a rest of mind he will handle all in just 24 hours, After the second day my husband called me, i was just so shocked, i pick the call and couldn't believe my ears, he was really begging me to forgive him and making promises on phone.. He came back home and also got me a new car just for him to proof his love for me. i was so happy and called Prof Remmy and thanked him, he only told me to share the good news all over the world .. Well if you need an effective and real spell caster contact Via email: psychicremmy@gmail.com Call +27660670249 or WhatsApp: +1-517-201-7336 or visit :https://www.psychicremmy.com/

    ReplyDelete
  10. Very interesting information!Perfect just what I was looking for! “Peace, commerce and honest friendship with all nations entangling alliances with none.

    Astrologer in Washington,
    Marriage Problems Solution,
    Astrologer and Spiritual Healer,

    ReplyDelete
  11. Hi Nikolas, let me drop my warm thanks for this awesome piece of fabulous project. I just revealed your post through googling and excited with what lesions and experiences you shared with us. This is cool. The black magic of deep learning tips and steps you unveiled for your fans and followers were incredible helpful. In our practical life we have to face so many unwanted and unhappy events that lead us to frustration and disheartened and we start to find our life meaningless. It may occur for the dissatisfaction in our working performance, lack of love, broken relationship and so on. In that situation, we must have to find out some trusted helps or supports around us to overcome. I think your analytical tips might be an exceptional inspiration for us. Also there are many honest help out there online who are really devoted to help people in crisis so that they can get back in their normal life. Right this moment I can remember warlock spells that's really perfect place to try out. Their goal and mission is to help people and make their lives better. Maybe some guys here are really in need of such kind of help.

    ReplyDelete
  12. Hello to everyone out here, I am here to share the unexpected miracle that happened to me … My name is Susan Christian , I live in London, UK. we got married for more than 9 years and have gotten two kids. thing were going well with us and we are always happy. until one day my husband started to behave in a way i could not understand, i was very confused by the way he treat me and the kids. later that month he did not come home again and he called me that he want a divorce, i asked him what have i done wrong to deserve this from him, all he was saying is that he want a divorce that he hate me and do not want to see me again in his life, i was mad and also frustrated do not know what to do, i was sick for more than 2 weeks because of the divorce. i love him so much he was everything to me without him my life is incomplete. i told my sister and she told me to contact a spell caster, i never believe in all this spell casting of a thing. i just want to try if something will come out of it. i contacted Dr Emu for the return of my husband to me, they told me that my husband have been taken by another woman, that she cast a spell on him that is why he hate me and also want us to divorce. then they told me that they have to cast a spell on him that will make him return to me and the kids, they casted the spell and after 24 hours my husband called me and he told me that i should forgive him, he started to apologize on phone and said that he still live me that he did not know what happen to him that he left me. it was the spell that he Dr Emu casted on him that make him come back to me today, me and my family are now happy again today. thank you Dr Emu for what you have done for me i would have been nothing today if not for your great spell. i want you my friends who are passing through all this kind of love problem of getting back their husband, wife , or ex boyfriend and girlfriend to contact Dr Emu ,if you need his help you can contact him through his private mail: emutemple@gmail.com or you can contact him through his website https://emutemple.wordpress.com/ fb page Https://web.facebook.com/Emu-Temple-104891335203341 and you will see that your problem will be solved without any delay.

    ReplyDelete
  13. What would i have done to get my husband back if not for the help of DR JOHN SOCO. my name is Sarah Nicky and I was in totally confused when my husband divorced me and said he was tired of me. My life was nothing without him and i tried all i could to make him cancel the divorce but he said his mind was made up and we went our separate ways. when my friend when searching for help and i was told to get in touch with DR JOHN SOCO for help by an New friend of mine when she heard what i was going through and i hurriedly contacted DR JOHN SOCO and explained everything to him and he told me that my husband will come back to me begging me to accept him back all within 24 hours just like everyone he had helped. DR JOHN SOCO restored the love and connection between me and my husband he truly returned back to me after 24 hours and it still amazes me. For more enquires on how to get in touch with DR JOHN SOCO, Email: drjohnsoco@gmail.com or WhatsApp him on +1 (706) 871 4571..

    ReplyDelete
  14. DR. JOROMI IS TRULY A POWERFUL MAN. Contacting Dr Joromi is the best thing that has ever happened to me. My husband left me for another woman for the reason which i know nothing about. I tried every means to get him back but to no avail, all my efforts proved abortive. I was even contemplating suicide. On a lucky day when i was browsing the internet, i came across some amazing testimonies about Dr. Joromi on how he has helped a lot of people with his powerful love spells. I contacted him and explained all my problems to him and he assured me that he will help me. He told me what to do and assured me that my man will be back in 12 hours. I obeyed him and followed the instructions and to my Greatest surprise, my man called me in 12 hours time just as Dr. Joromi said. Thank you Dr Joromi. May the good God continue to bless you. If you are in need of help please contact Dr Joromi now on his website: https://www.joromispells.com and you can also reach him with his email: Joromispells@gmail.com and here is his WhatsApp number which you can also use to reach him +2348138695387 Thank you Dr Joromi. Thank you once again

    ReplyDelete
  15. Hey, It really is incredibly fantastic and informative website. Good to discover your site Very well article! I’m simply in love with it.جلب الحبيب

    ReplyDelete
  16. I really appreciate Dr.Ogedegbe the powerful spell caster who help me cast a love spell that brought my ex husband within 48hours after providing him all the details he required from me and after the casting of the spell my ex husband came back to apologize that i should accept him and this will never happen again, for a long time now but since i met this spell caster online my story has change for good contact him via email dr.ogedegbe6@gmail.com or call/whatsapp +2348109374702. All thanks to Dr Ogedegbe for the good work... visit his youtube channel https://www.youtube.com/channel/UC-Hx8eErDaQ3gZce2U87GSQ

    ReplyDelete
  17. This comment has been removed by the author.

    ReplyDelete
  18. Best Indian Astrologer in New York
    I read your article, you give informative information about Astrology. I was searching for this particular information for a very long. 

    ReplyDelete
  19. I want to say thanks to this great man called Dr. Osasu who helped me in my marriage life. my name is Lilette Griffin lives in USA, So I was married to Albert we both love and like each other before our marriage, he care about my well being so i was so happy that i have found a man like this in my life my parents love him so much because of his kindness towards me and the way he care about me after four years in my marriage, no child he was not showing me much care anymore but i notice something is going on which i no. then i keep on with the relationship and i was hoping one day God will open my way to have a child in my home then i keep on going to church from one play to another by telling all the pastors about my problems that i don't have a child so many of them promise me that soon i we have a child in my home i keep on hoping in God's miracle on till one day when i went for a visit in my friend office then i was welcome by my friend we started dis causing about so many things on marriage life i was so shock she ask me about my wedding and what is going on till now i still don't have a child then i told her my dear sister i don't no what to do anymore and am scared of loosing my husband who have be caring for me for a long time now then she said i we not loosing him i ask how? then she said there is one man called Dr. Osasu who help in a relationship then i keep on asking how about him she told me this man can do all thing and make things possible i never believe her for once that this man she is talking about can do it so fast. also i ask her how do i get his contact she said i should not worry my self then she gave me his contact email ID and his number to contact him i said ok i will try my best to do so she wish me the best of luck in life then i went home gotten home, found that my door is open i was scared thinking so many things i don't no who is in my house now i look at the key in my hand i was thinking i did not lock my door shortly a word came into my mind that i should go inside and check if there is any body at home or my husband getting inside, i found that my husband move away his things in the house then i started calling his number refuses to pick i was like a mad Dog i cried and cried don't no what to do then i remember to that a friend of mine gave me a contact today i take my computer i emailed this man called Dr. Osasu also i called him on his cell phone which i received from my friend he spoke to me very well and i was happy my husband we come back to me so after the work was well done by Dr. Osasu, just in three days i heard my phone ringing not knowing it was my husband telling me his sorry about what he did to me then i accepted him again a month later, i was pregnant for him so i rest my testimony till i deliver safely i we give the best testimony again so my sisters and brothers if you are in such pain kindly contact this great man with this email drosasu25@gmail.com OR WhatsApp +2347064365391

    ReplyDelete