Skip to content

Deep Neural Networks

The Deep Learning textbook is a resource intended to help students and practitioners enter the field of machine learning in general and deep learning in particular. The online version of the book is now complete and will remain available online for free.

It is highly recommended to read the first and second parts of this book firstly, which will be of great help in reading some classic papers in the future.

History and Basics

  • LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." Nature 521.7553 (2015): 436. [pdf]. Three Giants' Survey
  • Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. "Learning representations by back-propagating errors." Nature volume 323, pages533–536 (1986). Back Propagation
  • Hochreiter, Sepp. "The vanishing gradient problem during learning recurrent neural nets and problem solutions." International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 6.02 (1998): 107-116. [pdf]. Problem discovered and analysed
  • Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18.7 (2006): 1527-1554. [pdf]. Deep Learning Eve
  • Hinton, Geoffrey E., and Ruslan R. Salakhutdinov. "Reducing the dimensionality of data with neural networks." Science 313.5786 (2006): 504-507. [pdf]. Milestone Pretraining Fine-tuning
  • Glorot, Xavier, and Yoshua Bengio. "Understanding the difficulty of training deep feedforward neural networks." Proceedings of the thirteenth international conference on artificial intelligence and statistics. 2010. [pdf]. Xavier Math
  • LeCun, Yann, et al. "Gradient-based learning applied to document recognition." Proceedings of the IEEE 86.11 (1998): 2278-2324. [pdf]. LeNet CNN
  • Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012. [pdf]. Breakthrough in computer vision AlexNet GPU
  • Lin, Min, Qiang Chen, and Shuicheng Yan. "Network in network." arXiv preprint arXiv:1312.4400 (2013). [pdf]. NiN Imporve CNN
  • Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014). [pdf]. VGGNet Get deeper
  • Szegedy, Christian, et al. "Going deeper with convolutions." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015. [pdf]. GoogLeNet InceptionV1
  • He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. [pdf]. ResNet Very very deep
  • Hinton, Geoffrey, et al. "Deep neural networks for acoustic modeling in speech recognition." IEEE Signal processing magazine 29 (2012). [pdf]. Breakthrough in speech recognition
  • Graves, Alex. "Generating sequences with recurrent neural networks." arXiv preprint arXiv:1308.0850 (2013). [pdf]. Milestone RNN
  • Hochreiter, Sepp, and Jürgen Schmidhuber. "Long short-term memory." Neural computation 9.8 (1997): 1735-1780. [pdf]. LSTM
  • Cho, Kyunghyun, et al. "Learning phrase representations using RNN encoder-decoder for statistical machine translation." arXiv preprint arXiv:1406.1078 (2014). [pdf]. GRU
  • Hinton, Geoffrey E., et al. "Improving neural networks by preventing co-adaptation of feature detectors." arXiv preprint arXiv:1207.0580 (2012). [pdf]. Dropout original
  • Srivastava, Nitish, et al. "Dropout: a simple way to prevent neural networks from overfitting." The journal of machine learning research 15.1 (2014): 1929-1958. [pdf]. Dropout definition
  • Ioffe, Sergey, and Christian Szegedy. "Batch normalization: Accelerating deep network training by reducing internal covariate shift." arXiv preprint arXiv:1502.03167 (2015). [pdf]. Batch normalization Math
  • Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014). [pdf]. Adam Math
  • Han, Song, Huizi Mao, and William J. Dally. "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding." arXiv preprint arXiv:1510.00149 (2015). [pdf]. Milestone Compression
  • Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems. 2014. [pdf]. Milestone GAN
  • Mnih, Volodymyr, et al. "Playing atari with deep reinforcement learning." arXiv preprint arXiv:1312.5602 (2013). [pdf]. Milestone DQN
  • Silver, David, et al. "Mastering the game of Go with deep neural networks and tree search." nature 529.7587 (2016): 484. [pdf]. AlphaGO AI be famous

Now you can read more related research papers. 😄

Helpful Resources

You should master using Google to find resources by yourself --

  • Deep Learning on Coursera: Learn Deep Learning from If you want to break into AI, this Specialization will help you do so.
  • Neural Networks and Deep Learning: A free online book.
  • e-Print archive: Submissions to arXiv should conform to Cornell University academic standards.
  • Arxiv Sanity Preserver: We develop a learning framework for building deformable templates, which play a fundamental role in many image analysis and computational anatomy tasks.
  • Papers With Code: The mission of Papers With Code is to create a free and open resource with Machine Learning papers, code and evaluation tables.
  • Distill: An Update from the Editorial Team ... Science is a human activity. When we fail to distill and explain research, we accumulate a kind of debt.
  • IMPORT AI: A weekly newsletter about artificial intelligence, read by more than ten thousand experts. Read past issues & subscribe here.

Follow others

Stanford Courses

Models and Optimism

Generative Adversarial Nets

GitHub: The classical paper list with code about generative adversarial nets

Ref: zhangqianhui/AdversarialNetsPapers

Model Compression and Acceleration

GitHub: A list of awesome papers on deep model compression and acceleration

Ref: sun254/awesome-model-compression-and-acceleration

Meta Learning

GitHub: Meta Learning / Learning to Learn / One Shot Learning / Few Shot Learning

Ref: floodsung/Meta-Learning-Papers

Auto Machine Learning

GitHub: A curated list of automated machine learning papers, articles, tutorials, slides and projects

Ref: hibayesian/awesome-automl-papers

Tasks and Applications

Image Recognition

GitHub Topics

Ref: topics/image-recognition

Tracking and Detection

Image Captioning

GitHub Collections

Ref: zhjohnchan/awesome-image-captioning

Super Resolution

GitHub Collections

Ref: ChaofWang/Awesome-Super-Resolution

Image Quality Assessment

Visual Question Answering