Editing NBDSM


== Schedule ==

=== 8/10/17 - Informal Meetup ===

Bring papers you've been reading, material/texts you've been working through, questions to ask, and Things You Understand to teach.

Dress code this week is all black.

Social events to follow.

=== 7/6/17 - Talk and Discussion: Dr. Steve Young - Boltzmann Machines and Statistical Mechanics. ===

PREREADINGS: 

[https://libgen.unblocked.srl/book/index.php?md5=E70CC484C7FF51073859B15779162C25 MacKay - Information Theory, Inference, and Learning Algorithms] Chapter 43 on the Boltzmann machine. Chapter 42 on Hopfield networks.

[[Media:hinton_lect11.pdf]] [[Media:hinton_lect12.pdf]] Lecture notes from Hinton's Coursera class. Good overview of Boltzmann machines and Hopfield nets. You can sign up for the free course and watch the accompanying videos [https://www.coursera.org/learn/neural-networks here]. They're also on [https://www.youtube.com/playlist?list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9 Youtube].

== What ==

nBDSM is the noiseBridge Deepnet and Statistical Mechanics working group. We meet periodically to learn, teach, and discuss topics at the intersection of AI/deep learning and statistical mechanics. Note that we have a non-trivial overlap with The One, The Only [https://www.noisebridge.net/wiki/DreamTeam Noisebridge DreamTeam].

We're focused on theory. Implementation is fun too, but has its own set of (mostly orthogonal) skills that we'll cover only lightly.

== Prerequisites ==

Our discussions are at upper division to grad level in machine learning and statistical mechanics. To be able to get something out of them, you should know

*linear algebra (at the level of [https://libgen.unblocked.srl/book/index.php?md5=79185948E755CC9B9611346E586CD050 D. Lay's book])
*single and multi-variable calculus, vector calculus, Lagrange multiplers, Taylor expansions (all of Stewart's textbook). 
*basics of statistics, including bayesian
*statistical mechanics (at the level of [http://mcgreevy.physics.ucsd.edu/s12/index.html McGreevy's MIT lecture notes])

There are plenty of other places to learn this stuff. Eg you can review your
probability, stats and linear algebra from chapters 2 and 3 of [https://libgen.unblocked.srl/book/index.php?md5=E4B2AB0EF22458F94C835D4D2397034E Goodfellow].

== Links ==
Check out these cool links

* A great [http://videolectures.net/deeplearning2016_ganguli_theoretical_neuroscience/ talk] by Ganguli at [https://sites.google.com/site/deeplearningsummerschool2016/ last year's deep learning summer school] in Montreal.
* Anything recent by [https://arxiv.org/find/stat/1/au:+Ganguli_S/0/1/0/all/0/1 Ganguli] at the [https://ganguli-gang.stanford.edu/ Neural Dynamics and Computation Lab] as well.
*[https://calculatedcontent.com/ Calculated Content]
* The venerable [http://colah.github.io/ colah's blog]
* Stat Mech//Machine Learning conference 2017 at Berkeley: [https://smml.io/ smml:2017]
* [https://libgen.unblocked.pub/book/index.php?md5=2E486AA1F672242DAA3D0B6116450D78 Proceedings] of the [http://www.lps.ens.fr/~krzakala/LESHOUCHES2013/home.htm Les Houches 2013 school] on Statistical physics, Optimization, Inference and Message-Passing algorithms. Fairly advanced. 
* [https://www.youtube.com/playlist?list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9 Videos] from Geoff Hinton's neural net course on Coursera.
* A [http://bactra.org/weblog/361.html blog] post about exponential families that demonstrates the sort of intuition we're trying to build.
* [[Media:maxEntChap10.pdf]] Intro to principle of Maximum Entropy

These are my ongoing personal written working notes. They are a mess, but you can at least use them to see what I'm working on.

* [https://1drv.ms/o/s!AmTN0QVCYp0Og-BvczbkIA4xN1FfKg Steve's AI Learning Notes (read only, so people don't draw butts or flowers all over everything)]

== Papers ==

=== Overviews ===

Good large scale overview of why the stat mech side is important
* Advani et al. - Stat mech of complex neural systems and high dimensional data - [https://arxiv.org/abs/1301.7115v1 arXiv:1301.7115v1]

Less emphasis on the physics, more emphasis on the stat mech <-> statistical inference
connection.
* Mastromatteo - On the typical properties of inverse problems in stat mech - [https://arxiv.org/abs/1311.0190v1 arXiv:1311.0910v1]

* Zdeborová et al. - [https://arxiv.org/abs/1511.02476 arXiv:1511.02476]

=== Interesting papers ===

* Chen et al. - On the Equivalence of Restricted Boltzmann Machines and Tensor Network States - [https://arxiv.org/abs/1701.04831v1 arXiv:1701:04831v1]
* Mehta et al. - An exact mapping between the Variational Renormalization Group and Deep Learning - [https://arxiv.org/abs/1410.3831 arXiv:1410.3831], Recreation at  [https://github.com/fineline179/MEHTA_project https://github.com/fineline179/MEHTA_project]
* Saxe et al. - Exact solutions to the nonlinear dynamics of learning in deep linear neural networks - [https://arxiv.org/abs/1312.6120 arXiv:1312.6120]
* Deng et al. - Quantum Entanglement in Neural Network States [https://arxiv.org/abs/1701.04844 arXiv:1701.04844]

== Books ==

=== IMPORTANT NOTICE ON PIRACY AND INTELLECTUAL PROPERTY ===
* A great place to find books and articles is [https://www.reddit.com/r/Scholar/comments/3bs1rm/meta_the_libgenscihub_thread_howtos_updates_and/ Library Genesis]. I use these links for books and articles: [https://libgen.unblocked.srl/],  [https://libgen.unblocked.li/scimag].

=== Stat Mech ===
* [https://libgen.unblocked.srl/book/index.php?md5=836019C9EC85D594F380D1D898B59713 Huang's] text is the bronze standard for grad level stat mech.

* [https://libgen.unblocked.srl/book/index.php?md5=05528E3012BCB03B7E0142343E58D0C6 Chandler's] text is supposedly great for stat mech, although I haven't read it.

=== Stat Mech and Stat Inference ===
* [https://libgen.unblocked.pub/book/index.php?md5=4CFDD37EEDB946D6E944750F746DB72B Bishop - Pattern Recognition and Machine Learning] Great pedagogical introduction to the basics. Good treatment of exponential family.

* [http://www.inference.org.uk/itila/p0.html MacKay - Information Theory, Inference, and Learning Algorithms] Link [https://libgen.unblocked.srl/book/index.php?md5=E70CC484C7FF51073859B15779162C25 here]

* [https://libgen.unblocked.srl/book/index.php?md5=E4B2AB0EF22458F94C835D4D2397034E Goodfellow et al. - Deep Learning]

* [https://libgen.unblocked.srl/book/index.php?md5=3E76F8F5189A047550CA9020D97848E4 Mezard - Information, Physics, and Computation]

* [https://libgen.unblocked.srl/book/index.php?md5=AFEE8F3BEF62DC5C7A191451259AD8EB Engel - Statistical Mechanics of Learning] I haven't looked at this yet, but it seems promising.

* [https://libgen.unblocked.srl/book/index.php?md5=0AF108CB2FEFA5A20F7B186BC2C88656 Jaynes - Probability Theory, The Logic of Science] An excellent introduction to Bayesian reasoning and probability theory in general, from a very pedagogical, opinionated point-of-view. Dives into the motivations behind some information theory as well.

= Stat Mech and Stat Inference =
* [https://libgen.unblocked.srl/book/index.php?md5=AFEE8F3BEF62DC5C7A191451259AD8EB Engel - Statistical Mechanics of Learning] I haven't looked at this yet, but it seems promising.
* [http://www.inference.org.uk/itila/p0.html MacKay - Information Theory, Inference, and Learning Algorithms] Link [https://libgen.unblocked.srl/book/index.php?md5=E70CC484C7FF51073859B15779162C25 here]
* [https://libgen.unblocked.pub/book/index.php?md5=4CFDD37EEDB946D6E944750F746DB72B Bishop - Pattern Recognition and Machine Learning]
* [https://libgen.unblocked.srl/book/index.php?md5=E4B2AB0EF22458F94C835D4D2397034E Goodfellow et al. - Deep Learning]
* [https://libgen.unblocked.srl/book/index.php?md5=3E76F8F5189A047550CA9020D97848E4 Mezard - Information, Physics, and Computation]
* [https://libgen.unblocked.srl/book/index.php?md5=0AF108CB2FEFA5A20F7B186BC2C88656 Jaynes - Probability Theory, The Logic of Science] An excellent introduction to Bayesian reasoning and probability theory in general, from a very pedagogical, opinionated point-of-view. Dives into the motivations behind some information theory as well.

== Ideas for future talks ==
Here's some ideas for future talks. If you want to present one of these,

A) Feel free to be advanced as you like -- assume an audience of graduate students.

but

B) Don't feel pressured to go any faster than you want. If you think you can give a pedagogical 'for dummies' talk in the course of an hour and a half, go for it!

* Derive capacity of Hopfield net and understand this limitation intuitively
* Explain similarity/relationship/identity of Bayesian inference and maximum entropy formalism.
* Deep intuitive dive on Lagrangian duals and what they really do/mean in the context of statistical inference/machine learning/stat mech