Bounded Rationality (Posts about Bits-Back)http://bjlkeng.github.io/enSat, 23 Dec 2023 02:03:06 GMTNikola (getnikola.com)http://blogs.law.harvard.edu/tech/rssLossless Compression with Latent Variable Models using Bits-Back Codinghttp://bjlkeng.github.io/posts/lossless-compression-with-latent-variable-models-using-bits-back-coding/Brian Keng<div><p>A lot of modern machine learning is related to this idea of "compression", or
maybe to use a fancier term "representations". Taking a huge dimensional space
(e.g. images of 256 x 256 x 3 pixels = 196608 dimensions) and somehow compressing it into
a 1000 or so dimensional representation seems like pretty good compression to
me! Unfortunately, it's not a lossless compression (or representation).
Somehow though, it seems intuitive that there must be a way to use what is learned in
these powerful lossy representations to help us better perform <em>lossless</em>
compression, right? Of course there is! (It would be too anti-climatic of a
setup otherwise.)</p>
<p>This post is going to introduce a method to perform lossless compression that
leverages the learned "compression" of a machine learning latent variable
model using the Bits-Back coding algorithm. Depending on how you first think
about it, this <em>seems</em> like it should either be (a) really easy or (b) not possible at
all. The reality is kind of in between with an elegant theoretical algorithm
that is brought down by the realities of discretization and imperfect learning
by the model. In today's post, I'll skim over some preliminaries (mostly
referring you to previous posts), go over the main Bits-Back coding algorithm
in detail, and discuss some of the implementation details and experiments that
I did while trying to write a toy version of the algorithm.</p>
<p><a href="http://bjlkeng.github.io/posts/lossless-compression-with-latent-variable-models-using-bits-back-coding/">Read moreā¦</a> (25 min remaining to read)</p></div>asymmetric numeral systemsBits-BackcompressionlosslessmathjaxMNISTvariational autoencoderhttp://bjlkeng.github.io/posts/lossless-compression-with-latent-variable-models-using-bits-back-coding/Tue, 06 Jul 2021 16:00:00 GMT