August 30th, 2019,
‘What I cannot create, I do not understand’ – Richard Feynman
This quote has haunted for the majority of this year. For the last few months, I had been feeling like an impostor. I felt that my reliance on high-level deep learning libraries had been an impediment to me fully understanding neural networks and what goes on under the hood. So, in May 2019 I decided to give up high-level libraries such as Tensorflow and not use them until I could write a neural network from scratch using only low lying mathematical and numerical operation libraries such as Numpy in Python.
As I set out to do this, I went to our trusty friend Google and I was overwhelmed by the sheer number of resources. So I ran away. After I gained my courage back I spent three months learning what goes on behind the curtain of deep learning. In the light of my experience, I decided to put the most useful resources in one place to help others. My goal is to put together a list of comprehensive resources that should help you mathematically recreate a neural network. After following and using these resources, I have been able to rederive all the equations of forward and, more significantly, backpropagation.
KHAN ACADEMY FOR DIFFERENTIAL EQUATIONS
Neural Networks involve a lot of differential equations so you need to have a working understanding of differential calculus. Khan Academy provides a comprehensive look at the concepts of differential calculus. You won’t need all the playlists to learn enough to master the concepts for neural networks. I suggest the following playlists:
- Derivatives: definition and basic rules This will introduce you to the basic ideas of derivatives. If you already know basics, I still think that it is worth skimming through these.
- Derivatives: chain rule and other advanced topics This one is the moneymaker. Chain rule forms the basis of backpropagation. You need to understand the use of chain rule to develop a full appreciation of neural networks.
- You must do all the practice questions associated with these playlists to formalize your understanding of these concepts.
MICHAEL NIELSEN’S BOOK ON NEURAL NETWORK.
Michael Nielsen has an exhaustive book on the workings of neural networks and it is free. To develop the understanding of the basics of neural networks, chapter 1 and chapter 2 are extremely important.
3Blue1Brown’s Neural Network Playlist.
The Neural Network Playlist by 3Blue1Brown is an amazing resource. It uses amazing animation to break down some hard concepts. This should be used in conjunction with the readings of Michael Nielsen’s book. It will let you visualize what Nielsen is talking about in his book.
Jonath Weisberg’s blog.
When you feel that you have a working understanding of forward propagtion, read through Jonathan Weisberg’s 2 part blog. He does an excellent job of breaking down the thinking behind creating the code for a neural network and writes it in easy to follow manner.
BRINGING IT ALL TOGETHER.
This is the approach I took to developing my understanding of the forward and backpropagation. Feel free to adapt, change and improve upon it.
- Start by reading the first 2 chapters of Nielsen’s book. This should give you an understanding of the level of difficulty you are dealing with. Don’t feel overwhelmed. This will all make sense soon.
- Watch the Khan Academy playlists listed above. Make sure to do all the exercises to cement your understanding.
- Read the Nielsen chapters periodically to see if things are making more sense to you as you develop a fuller understanding of the calculus presented.
- Once you finish both the playlists, read the chapters with the intent to recreate the math yourself. It’s okay if you can’t fully still.
- Use 3Blue1Brown playlist in conjunction with this series of reading. It will visualize a lot of the math for you.
RICHARD FEYNMAN TO THE RESCUE.
It is poetic, I think, that it was Feynman whose earlier quote threw me in this existential hold and it was Feynman Technique that got me out of it. This is what you should do once you feel that you have some level of understanding of what is going on.
- Describe the full workings of the neural network by writing it down. Try to explain it as if you were writing notes for someone who doesn’t understand any of this. So, no complicated words for the sake of complicated words.
- Derive all of the forward and backpropagation equations yourself.
- Where you feel like you struggle, use one of the resources mentioned earlier to clear up those concepts.
- Repeat step 1.
This might take a while for you to get to a point where you can derive all the math behind neural networks. It took me close to 3 months to do that. If you do it faster than this, awesome! If it takes you longer, there is no need to beat yourself up. Once you get to where you understand everything going on under the hood of deep learning, it will all have been worth it.
MY IMPLEMENTATION OF A NEURAL NETWORK FROM SCRATCH
Here is my implementation of a neural network from scratch to classify MNIST. My aim here was understanding and I am very happy with what I came up with.
https://www.kaggle.com/sanwal092/3-layer-neural-network-from-scratch
Happy Learning!!
deepjourney 