Bounded Rationality (Posts about variational calculus)http://bjlkeng.github.io/enTue, 04 Jun 2024 00:49:17 GMTNikola (getnikola.com)http://blogs.law.harvard.edu/tech/rssImportance Sampling and Estimating Marginal Likelihood in Variational Autoencodershttp://bjlkeng.github.io/posts/importance-sampling-and-estimating-marginal-likelihood-in-variational-autoencoders/Brian Keng<div><p>It took a while but I'm back! This post is kind of a digression (which seems
to happen a lot) along my journey of learning more about probabilistic
generative models. There's so much in ML that you can't help learning a lot
of random things along the way. That's why it's interesting, right?</p>
<p>Today's topic is <em>importance sampling</em>. It's a really old idea that you may
have learned in a statistics class (I didn't) but somehow is useful in deep learning,
what's old is new right? How this is relevant to the discussion is that when
we have a large latent variable model (e.g. a variational
autoencoder), we want to be able to efficiently estimate the marginal likelihood
given data. The marginal likelihood is kind of taken for granted in the
experiments of some VAE papers when comparing different models. I was curious
how it was actually computed and it took me down this rabbit hole. Turns out
it's actually pretty interesting! As usual, I'll have a mix of background
material, examples, math and code to build some intuition around this topic.
Enjoy!</p>
<p><a href="http://bjlkeng.github.io/posts/importance-sampling-and-estimating-marginal-likelihood-in-variational-autoencoders/">Read more…</a> (22 min remaining to read)</p></div>autoencodersautoregressiveCIFAR10generative modelsimportance samplingmathjaxMNISTMonte Carlovariational calculushttp://bjlkeng.github.io/posts/importance-sampling-and-estimating-marginal-likelihood-in-variational-autoencoders/Wed, 06 Feb 2019 12:20:11 GMTVariational Autoencoders with Inverse Autoregressive Flowshttp://bjlkeng.github.io/posts/variational-autoencoders-with-inverse-autoregressive-flows/Brian Keng<div><p>In this post, I'm going to be describing a really cool idea about how
to improve variational autoencoders using inverse autoregressive
flows. The main idea is that we can generate more powerful posterior
distributions compared to a more basic isotropic Gaussian by applying a
series of invertible transformations. This, in theory, will allow
your variational autoencoder to fit better by concentrating the
stochastic samples around a closer approximation to the true
posterior. The math works out so nicely while the results are kind of
marginal <a class="footnote-reference brackets" href="http://bjlkeng.github.io/posts/variational-autoencoders-with-inverse-autoregressive-flows/#id3" id="id1">1</a>. As usual, I'll go through some intuition, some math,
and have an implementation with few experiments I ran. Enjoy!</p>
<p><a href="http://bjlkeng.github.io/posts/variational-autoencoders-with-inverse-autoregressive-flows/">Read more…</a> (18 min remaining to read)</p></div>autoencodersautoregressiveCIFAR10generative modelsKullback-LeiblerMADEmathjaxMNISTvariational calculushttp://bjlkeng.github.io/posts/variational-autoencoders-with-inverse-autoregressive-flows/Tue, 19 Dec 2017 13:47:38 GMTSemi-supervised Learning with Variational Autoencodershttp://bjlkeng.github.io/posts/semi-supervised-learning-with-variational-autoencoders/Brian Keng<div><p>In this post, I'll be continuing on this variational autoencoder (VAE) line of
exploration
(previous posts: <a class="reference external" href="http://bjlkeng.github.io/posts/variational-autoencoders/">here</a> and
<a class="reference external" href="http://bjlkeng.github.io/posts/a-variational-autoencoder-on-the-svnh-dataset/">here</a>) by
writing about how to use variational autoencoders to do semi-supervised
learning. In particular, I'll be explaining the technique used in
"Semi-supervised Learning with Deep Generative Models" by Kingma et al.
I'll be digging into the math (hopefully being more explicit than the paper),
giving a bit more background on the variational lower bound, as well as
my usual attempt at giving some more intuition.
I've also put some notebooks on Github that compare the VAE methods
with others such as PCA, CNNs, and pre-trained models. Enjoy!</p>
<p><a href="http://bjlkeng.github.io/posts/semi-supervised-learning-with-variational-autoencoders/">Read more…</a> (25 min remaining to read)</p></div>autoencodersCIFAR10CNNgenerative modelsinceptionKullback-LeiblermathjaxPCAsemi-supervised learningvariational calculushttp://bjlkeng.github.io/posts/semi-supervised-learning-with-variational-autoencoders/Mon, 11 Sep 2017 12:40:47 GMTA Variational Autoencoder on the SVHN datasethttp://bjlkeng.github.io/posts/a-variational-autoencoder-on-the-svnh-dataset/Brian Keng<div><p>In this post, I'm going to share some notes on implementing a variational
autoencoder (VAE) on the
<a class="reference external" href="http://ufldl.stanford.edu/housenumbers/">Street View House Numbers</a>
(SVHN) dataset. My last post on
<a class="reference external" href="http://bjlkeng.github.io/posts/variational-autoencoders/">variational autoencoders</a>
showed a simple example on the MNIST dataset but because it was so simple I
thought I might have missed some of the subtler points of VAEs -- boy was I
right! The fact that I'm not really a computer vision guy nor a deep learning
guy didn't help either. Through this exercise, I picked up some of the basics
in the "craft" of computer vision/deep learning area; there are a lot of subtle
points that are easy to gloss over if you're just reading someone else's
tutorial. I'll share with you some of the details in the math (that I
initially got wrong) and also some of the implementation notes along with a
notebook that I used to train the VAE. Please check out my previous post
on <a class="reference external" href="http://bjlkeng.github.io/posts/variational-autoencoders/">variational autoencoders</a> to
get some background.</p>
<p><em>Update 2017-08-09: I actually found a bug in my original code where I was
only using a small subset of the data! I fixed it up in the notebooks and
I've added some inline comments below to say what I've changed. For the most
part, things have stayed the same but the generated images are a bit blurry
because the dataset isn't so easy anymore.</em></p>
<p><a href="http://bjlkeng.github.io/posts/a-variational-autoencoder-on-the-svnh-dataset/">Read more…</a> (19 min remaining to read)</p></div>autoencodersgenerative modelsKullback-Leiblermathjaxsvhnvariational calculushttp://bjlkeng.github.io/posts/a-variational-autoencoder-on-the-svnh-dataset/Thu, 13 Jul 2017 12:13:03 GMTVariational Autoencodershttp://bjlkeng.github.io/posts/variational-autoencoders/Brian Keng<div><p>This post is going to talk about an incredibly interesting unsupervised
learning method in machine learning called variational autoencoders. It's main
claim to fame is in building generative models of complex distributions like
handwritten digits, faces, and image segments among others. The really cool
thing about this topic is that it has firm roots in probability but uses a
function approximator (i.e. neural networks) to approximate an otherwise
intractable problem. As usual, I'll try to start with some background and
motivation, include a healthy does of math, and along the way try to convey
some of the intuition of why it works. I've also annotated a
<a class="reference external" href="https://github.com/bjlkeng/sandbox/blob/master/notebooks/variational-autoencoder.ipynb">basic example</a>
so you can see how the math relates to an actual implementation. I based much
of this post on Carl Doersch's <a class="reference external" href="https://arxiv.org/abs/1606.05908">tutorial</a>,
which has a great explanation on this whole topic, so make sure you check that
out too.</p>
<p><a href="http://bjlkeng.github.io/posts/variational-autoencoders/">Read more…</a> (25 min remaining to read)</p></div>autoencodersgenerative modelsKullback-Leiblermathjaxvariational calculushttp://bjlkeng.github.io/posts/variational-autoencoders/Tue, 30 May 2017 12:19:36 GMTVariational Bayes and The Mean-Field Approximationhttp://bjlkeng.github.io/posts/variational-bayes-and-the-mean-field-approximation/Brian Keng<div><p>This post is going to cover Variational Bayesian methods and, in particular,
the most common one, the mean-field approximation. This is a topic that I've
been trying to understand for a while now but didn't quite have all the background
that I needed. After picking up the main ideas from
<a class="reference external" href="http://bjlkeng.github.io/posts/the-calculus-of-variations/">variational calculus</a> and
getting more fluent in manipulating probability statements like
in my <a class="reference external" href="http://bjlkeng.github.io/posts/the-expectation-maximization-algorithm/">EM</a> post,
this variational Bayes stuff seems a lot easier.</p>
<p>Variational Bayesian methods are a set of techniques to approximate posterior
distributions in <a class="reference external" href="https://en.wikipedia.org/wiki/Bayesian_inference">Bayesian Inference</a>.
If this sounds a bit terse, keep reading! I hope to provide some intuition
so that the big ideas are easy to understand (which they are), but of course we
can't do that well unless we have a healthy dose of mathematics. For some of the
background concepts, I'll try to refer you to good sources (including my own),
which I find is the main blocker to understanding this subject (admittedly, the
math can sometimes be a bit cryptic too). Enjoy!</p>
<p><a href="http://bjlkeng.github.io/posts/variational-bayes-and-the-mean-field-approximation/">Read more…</a> (24 min remaining to read)</p></div>BayesianKullback-Leiblermathjaxmean-fieldvariational calculushttp://bjlkeng.github.io/posts/variational-bayes-and-the-mean-field-approximation/Mon, 03 Apr 2017 13:02:46 GMTThe Calculus of Variationshttp://bjlkeng.github.io/posts/the-calculus-of-variations/Brian Keng<div><p>This post is going to describe a specialized type of calculus called
variational calculus.
Analogous to the usual methods of calculus that we learn in university,
this one deals with functions <em>of functions</em> and how to
minimize or maximize them. It's used extensively in physics problems such as
finding the minimum energy path a particle takes under certain conditions. As
you can also imagine, it's also used in machine learning/statistics where you
want to find a density that optimizes an objective <a class="footnote-reference brackets" href="http://bjlkeng.github.io/posts/the-calculus-of-variations/#id4" id="id1">1</a>. The explanation I'm
going to use (at least for the first part) is heavily based upon Svetitsky's
<a class="reference external" href="http://julian.tau.ac.il/bqs/functionals/functionals.html">Notes on Functionals</a>, which so far is
the most intuitive explanation I've read. I'll try to follow Svetitsky's
notes to give some intuition on how we arrive at variational calculus from
regular calculus with a bunch of examples along the way. Eventually we'll
get to an application that relates back to probability. I think with the right
intuition and explanation, it's actually not too difficult, enjoy!</p>
<p><a href="http://bjlkeng.github.io/posts/the-calculus-of-variations/">Read more…</a> (16 min remaining to read)</p></div>differentialsentropylagrange multipliersmathjaxprobabilityvariational calculushttp://bjlkeng.github.io/posts/the-calculus-of-variations/Sun, 26 Feb 2017 15:08:38 GMT