Bounded Rationality (Posts about autoregressive)http://bjlkeng.github.io/enTue, 04 Jun 2024 00:49:16 GMTNikola (getnikola.com)http://blogs.law.harvard.edu/tech/rssPixelCNNhttp://bjlkeng.github.io/posts/pixelcnn/Brian Keng<div><p>It's been a long time coming but I'm finally getting this post out! I read
this paper a couple of years ago and wanted to really understand it because it
was state of the art at the time (still pretty close even now). As usual
though, once I started down the variational autoencoder line of posts, there
was always <em>yet</em> another VAE paper to look into so I never got around to
looking at this one.</p>
<p>This post is all about a proper probabilistic generative model called Pixel
Convolutional Neural Networks or PixelCNN. It was originally proposed
as a side contribution of Pixel Recurrent Neural Networks in [1] and later
expanded upon in [2,3] (and I'm sure many other papers). The real cool thing
about it is that it's (a) probabilistic, and (b) autoregressive. It's still
counter-intuitive to me that you can generate images one pixel at at time, but
I'm jumping ahead of myself here. We'll go over some background material, the
method, and my painstaking attempts at an implementation (and what I learned
from it). Let's get started!</p>
<p><a href="http://bjlkeng.github.io/posts/pixelcnn/">Read more…</a> (23 min remaining to read)</p></div>autoregressiveCIFAR10generative modelsmathjaxhttp://bjlkeng.github.io/posts/pixelcnn/Mon, 22 Jul 2019 11:11:09 GMTImportance Sampling and Estimating Marginal Likelihood in Variational Autoencodershttp://bjlkeng.github.io/posts/importance-sampling-and-estimating-marginal-likelihood-in-variational-autoencoders/Brian Keng<div><p>It took a while but I'm back! This post is kind of a digression (which seems
to happen a lot) along my journey of learning more about probabilistic
generative models. There's so much in ML that you can't help learning a lot
of random things along the way. That's why it's interesting, right?</p>
<p>Today's topic is <em>importance sampling</em>. It's a really old idea that you may
have learned in a statistics class (I didn't) but somehow is useful in deep learning,
what's old is new right? How this is relevant to the discussion is that when
we have a large latent variable model (e.g. a variational
autoencoder), we want to be able to efficiently estimate the marginal likelihood
given data. The marginal likelihood is kind of taken for granted in the
experiments of some VAE papers when comparing different models. I was curious
how it was actually computed and it took me down this rabbit hole. Turns out
it's actually pretty interesting! As usual, I'll have a mix of background
material, examples, math and code to build some intuition around this topic.
Enjoy!</p>
<p><a href="http://bjlkeng.github.io/posts/importance-sampling-and-estimating-marginal-likelihood-in-variational-autoencoders/">Read more…</a> (22 min remaining to read)</p></div>autoencodersautoregressiveCIFAR10generative modelsimportance samplingmathjaxMNISTMonte Carlovariational calculushttp://bjlkeng.github.io/posts/importance-sampling-and-estimating-marginal-likelihood-in-variational-autoencoders/Wed, 06 Feb 2019 12:20:11 GMTVariational Autoencoders with Inverse Autoregressive Flowshttp://bjlkeng.github.io/posts/variational-autoencoders-with-inverse-autoregressive-flows/Brian Keng<div><p>In this post, I'm going to be describing a really cool idea about how
to improve variational autoencoders using inverse autoregressive
flows. The main idea is that we can generate more powerful posterior
distributions compared to a more basic isotropic Gaussian by applying a
series of invertible transformations. This, in theory, will allow
your variational autoencoder to fit better by concentrating the
stochastic samples around a closer approximation to the true
posterior. The math works out so nicely while the results are kind of
marginal <a class="footnote-reference brackets" href="http://bjlkeng.github.io/posts/variational-autoencoders-with-inverse-autoregressive-flows/#id3" id="id1">1</a>. As usual, I'll go through some intuition, some math,
and have an implementation with few experiments I ran. Enjoy!</p>
<p><a href="http://bjlkeng.github.io/posts/variational-autoencoders-with-inverse-autoregressive-flows/">Read more…</a> (18 min remaining to read)</p></div>autoencodersautoregressiveCIFAR10generative modelsKullback-LeiblerMADEmathjaxMNISTvariational calculushttp://bjlkeng.github.io/posts/variational-autoencoders-with-inverse-autoregressive-flows/Tue, 19 Dec 2017 13:47:38 GMTAutoregressive Autoencodershttp://bjlkeng.github.io/posts/autoregressive-autoencoders/Brian Keng<div><p>You might think that I'd be bored with autoencoders by now but I still
find them extremely interesting! In this post, I'm going to be explaining
a cute little idea that I came across in the paper <a class="reference external" href="https://arxiv.org/pdf/1502.03509.pdf">MADE: Masked Autoencoder
for Distribution Estimation</a>.
Traditional autoencoders are great because they can perform unsupervised
learning by mapping an input to a latent representation. However, one
drawback is that they don't have a solid probabilistic basis
(of course there are other variants of autoencoders that do, see previous posts
<a class="reference external" href="http://bjlkeng.github.io/posts/variational-autoencoders/">here</a>,
<a class="reference external" href="http://bjlkeng.github.io/posts/a-variational-autoencoder-on-the-svnh-dataset/">here</a>, and
<a class="reference external" href="http://bjlkeng.github.io/posts/semi-supervised-learning-with-variational-autoencoders/">here</a>).
By using what the authors define as the <em>autoregressive property</em>, we can
transform the traditional autoencoder approach into a fully probabilistic model
with very little modification! As usual, I'll provide some intuition, math and
an implementation.</p>
<p><a href="http://bjlkeng.github.io/posts/autoregressive-autoencoders/">Read more…</a> (17 min remaining to read)</p></div>autoencodersautoregressivegenerative modelsMADEmathjaxMNISThttp://bjlkeng.github.io/posts/autoregressive-autoencoders/Sat, 14 Oct 2017 14:02:15 GMT