Multimedia Laboratory

Useful Sites

• http://deeplearning.net/ - reading list, software packages, and tutorials
• Deep Learning Methods for Vision - 1 day tutorial at CVPR 2012.
• Tutorial on Deep Learning - 1 day tutorial at ICML 2013.
• Deep Learning for Computer Vision - Half day tutorial at NIPS 2013. Tutorial slides

Papers

• Introduction / General
  • G. E. Hinton, Learning Multiple Layers of Representation, Trends in Cognitive Sciences, Vo;. 11, No. 10, 2007
  • A good introduction on backpropation (BP)
• Convolutional Neural Networks (CNN)
  • Y LeCun, L Bottou, Y Bengio, P Haffner, Gradient-based Learning Applied to Document Recognition. Proceedings of the IEEE 86 (11), 2278-2324, 1998.
  • Jake Bouvrie, Notes on Convolutional Neural Networks. A very brief and clear introduction of forward and backward propagation of CNN.
    Recommended for readers who understand neural networks but does not understand CNN.
• Deep Belief Networks
  • G. E. Hinton et al., A Fast Learning Algorithm for Deep Belief Nets, Trends in Cognitive Sciences, Vo;. 11, No. 10, 2007.
• Review / Survey
  • Yoshua Bengio, Aaron Courville, Pascal Vincent, Representation Learning: A Review and New Perspectives, Arxiv, 2012.
  • Yoshua Bengio, Learning Deep Architectures for AI, Foundations and Trends in Machine Learning, 2009.
    Recommended for startup readers. It is a very good illustration of deep learing models. Easy to understand.
• Sum-Product Networks
  • Hoifung Poon and Pedro Domingos, Sum-Product Networks: A New Deep Architecture, UAI 2011 (Best Paper)
  • Robert Gens and Pedro Domingos, Discriminative Learning of Sum-Product Networks, NIPS 2012 (Outstanding Student Paper Awards)
  • Robert Gens and Pedro Domingos, Learning the Structure of Sum-Product Networks, ICML 2013
• Practical Tricks and Guides
  • Yoshua Bengio, Practical Recommendations for Gradient-based Training of Deep Architectures, ArXiv, 2012
• How to Visualise Your Deep Model
  • Dumitru Erhan, Aaron Courville, and Yoshua Bengio, Understanding Representations Learned in Deep Architectures, Technical Report, 2010
  • Matthew D. Zeiler, Rob Fergus, Visualizing and Understanding Convolutional Neural Networks, arxiv.org, 2013
• Why Deep Model is Good / Motivation of Deep Model
  • Yoshua Bengio et al., Better Mixing via Deep Representations, ArXiv, 2012
  • Andrew Y. Ng et al., Sparse deep belief net model for visual area V2, NIPS, 2007
• Deep Model for Multi-Modality
  • Andrew Y. Ng et al., Convolutional-Recursive Deep Learning for 3D Object Classification, NIPS, 2012

Some Chinese forums suggest a few papers, which may be useful:

• Link 1
• Link 2

Codes

Toolbox	Description
Cuda-Convnet (Convnet)	CNN implementation, most recommended to configure and obtain many kinds of existing Deep models, but hard if you want to design your own deep model
Theano	Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently.
Cudacnn	Cudacnn is C++/CUDA library with Matlab frontend for implementation, training and simulation of Convolutional neural networks.
Hinton's code for their science paper (Hinton)	Relatively easy to use and learn
DeeplearningToolbox (DLtoolbox)	MATLAB - the easiest to use for both Convolutional Neural Nets (CNN) and Deep Belief Nets (DBN), but not fast.
Sum-Product Networks	Hoifung Poon and Pedro Domingos, Sum-Product Networks: A New Deep Architecture
Sum-Product Networks	Robert Gens and Pedro Domingos, Learning the Structure of Sum-Product Networks

Lectures

	Description
	Yoshua Bengio’s Google tech talk on Deep Learning Representations at Google Montreal (Google Montreal, 11/13/2012) ICML 2012 slides by Yoshua Bengio can be found here.
	Geoffrey Hinton’s GoogleTech Talk, March 2010.
	Geoffrey Hinton, A Tutorial on Deep Learning
	Robert Gens, Discriminative Learning of Sum-Product Networks Sum-product networks are a new deep learning architecture that yields tractable inference. Deep architectures are the most expressive machine learning models in existence, but are notoriously difficult to train. This paper shows how to discriminatively train sum-product networks, which leads to significantly improved prediction accuracy.