Bounded Rationality (Posts about SGLD)http://bjlkeng.github.io/enSat, 23 Dec 2023 02:03:04 GMTNikola (getnikola.com)http://blogs.law.harvard.edu/tech/rssBayesian Learning via Stochastic Gradient Langevin Dynamics and Bayes by Backprophttp://bjlkeng.github.io/posts/bayesian-learning-via-stochastic-gradient-langevin-dynamics-and-bayes-by-backprop/Brian Keng<div><p>After a long digression, I'm finally back to one of the main lines of research
that I wanted to write about. The two main ideas in this post are not that
recent but have been quite impactful (one of the
<a class="reference external" href="https://icml.cc/virtual/2021/test-of-time/11808">papers</a> won a recent ICML
test of time award). They address two of the topics that are near and dear to
my heart: Bayesian learning and scalability. Dare I even ask who wouldn't be
interested in the intersection of these topics?</p>
<p>This post is about two techniques to perform scalable Bayesian inference. They
both address the problem using stochastic gradient descent (SGD) but in very
different ways. One leverages the observation that SGD plus some noise will
converge to Bayesian posterior sampling <a class="citation-reference" href="http://bjlkeng.github.io/posts/bayesian-learning-via-stochastic-gradient-langevin-dynamics-and-bayes-by-backprop/#welling2011" id="id1">[Welling2011]</a>, while the other generalizes the
"reparameterization trick" from variational autoencoders to enable non-Gaussian
posterior approximations <a class="citation-reference" href="http://bjlkeng.github.io/posts/bayesian-learning-via-stochastic-gradient-langevin-dynamics-and-bayes-by-backprop/#blundell2015" id="id2">[Blundell2015]</a>. Both are easily implemented in the modern deep
learning toolkit thus benefit from the massive scalability of that toolchain.
As usual, I will go over the necessary background (or refer you to my previous
posts), intuition, some math, and a couple of toy examples that I implemented.</p>
<p><a href="http://bjlkeng.github.io/posts/bayesian-learning-via-stochastic-gradient-langevin-dynamics-and-bayes-by-backprop/">Read moreā¦</a> (53 min remaining to read)</p></div>Bayes by BackpropBayesianelboHMCLangevinmathjaxrmspropsgdSGLDvariational inferencehttp://bjlkeng.github.io/posts/bayesian-learning-via-stochastic-gradient-langevin-dynamics-and-bayes-by-backprop/Wed, 08 Feb 2023 23:25:40 GMT