NBDSM: Difference between revisions

From Noisebridge
Jump to navigation Jump to search
Line 59: Line 59:


Good large scale overview of why the stat mech side is important
Good large scale overview of why the stat mech side is important
* Advani et. al. - Stat mech of complex neural systems and high dimensional data - [https://arxiv.org/abs/1301.7115v1 arXiv:1301.7115v1]
* Advani et al. - Stat mech of complex neural systems and high dimensional data - [https://arxiv.org/abs/1301.7115v1 arXiv:1301.7115v1]


Less emphasis on the physics, more emphasis on the stat mech <-> statistical inference
Less emphasis on the physics, more emphasis on the stat mech <-> statistical inference

Revision as of 00:24, 6 July 2017

UPDATE: FIRST MEETUP 7/6/17 AT 7PM

Our first meetup will be July 6th at 7pm, at Noisebridge Hackerspace, 2169 Mission st. San Francisco.

I'll talk about some of the key ideas at the intersection of stat mech and AI, in the context of what's known as the Boltzmann machine. A specialized version of this, the restricted Boltzmann machine (RBM), is the AI architecture behind the resurgence of interest in neural networks in the late aughts, prompted by Geoff Hinton "figuring out how to make them 10,000 times faster" via a procedure known as contrastive divergence. A significant amount of the buzz today in deep learning came from this development.

Some concepts to acquaint yourselves with before the first meeting: Ising Model, partition function, thermodynamic equilibrium, Lagrange multipliers (and Lagrangian dual), Bayesian inference, global versus local extrema, simulated annealing.

Before the first meeting, try to

  • be on top of the prerequisites (below)
  • check out the links (below)
  • skim the papers (below) (read the abstract, introduction and discussion/conclusions.)

But if you can't, come anyway!

SCHEDULE

7/6/17 - Talk and Discussion: Steve Young - Boltzmann Machines and Statistical Mechanics.

PREREADINGS:

MacKay - Information Theory, Inference, and Learning Algorithms Chapter 43 on the Boltzmann machine. Chapter 42 on Hopfield networks.

Media:hinton_lect11.pdf Media:hinton_lect12.pdf Lecture notes from Hinton's Coursera class. Good overview of Boltzmann machines and Hopfield nets. You can sign up for the free course and watch the accompanying videos here. They're also on Youtube.

What

nBDSM is the noiseBridge Deepnet and Statistical Mechanics working group. We meet weekly to learn, teach, and discuss topics at the intersection of AI/deep learning and statistical mechanics. Note that we have a non-trivial overlap with The One, The Only Noisebridge DreamTeam.

We're focused on theory. Implementation is fun too, but has its own set of (mostly orthogonal) skills that we'll cover only lightly.

Prerequisites

Our discussions are at upper division to grad level in machine learning and statistical mechanics. To be able to get something out of them, you should know

  • linear algebra (at the level of D. Lay's book)
  • single and multi-variable calculus, vector calculus, Lagrange multiplers, Taylor expansions (all of Stewart's textbook).
  • basics of statistics, including bayesian
  • statistical mechanics (at the level of McGreevy's MIT lecture notes)

There are plenty of other places to learn this stuff. Eg you can review your probability, stats and linear algebra from chapters 2 and 3 of Goodfellow.

Links

Check out these cool links

Papers

High Level Overviews

Good large scale overview of why the stat mech side is important

  • Advani et al. - Stat mech of complex neural systems and high dimensional data - arXiv:1301.7115v1

Less emphasis on the physics, more emphasis on the stat mech <-> statistical inference connection.

  • Mastromatteo - On the typical properties of inverse problems in stat mech - arXiv:1311.0910v1

Interesting papers

  • Chen et al. - On the Equivalence of Restricted Boltzmann Machines and Tensor Network States - arXiv:1701:04831v1
  • Mehta et al. - An exact mapping between the Variational Renormalization Group and Deep Learning - arXiv:1410.3831
  • Saxe et al. - Exact solutions to the nonlinear dynamics of learning in deep linear neural networks - arXiv:1312.6120

Books