http://bjlkeng.github.io/enTue, 10 Mar 2026 20:54:59 GMTNikola (getnikola.com)http://blogs.law.harvard.edu/tech/rsshttp://bjlkeng.github.io/posts/label-refinery/Brian Keng<div><p>This post is going to be about a really simple idea that is surprisingly effective from a paper by Bagherinezhad et al. called <a class="reference external" href="https://arxiv.org/abs/1805.02641">Label Refinery: Improving ImageNet Classification through Label Progression</a>. The title pretty much says it all but I'll also discuss some intuition and show some experiments on the CIFAR10 and SVHN datasets. The idea is both simple and surprising, my favourite kind of idea! Let's take a look.</p> <p><a href="http://bjlkeng.github.io/posts/label-refinery/">Read more…</a> (10 min remaining to read)</p></div>CIFAR10label refinerymathjaxresidual networkssvhnhttp://bjlkeng.github.io/posts/label-refinery/Tue, 04 Sep 2018 11:26:02 GMThttp://bjlkeng.github.io/posts/universal-resnet-the-one-neuron-approximator/Brian Keng<div><p><em>"In theory, theory and practice are the same. In practice, they are not."</em></p> <p>I read a very interesting paper titled <em>ResNet with one-neuron hidden layers is a Universal Approximator</em> by Lin and Jegelka [1]. The paper describes a simplified Residual Network as a universal approximator, giving some theoretical backing to the wildly successful ResNet architecture. In this post, I'm going to talk about this paper and a few of the related universal approximation theorems for neural networks. Instead of going through all the theoretical stuff, I'm simply going introduce some theorems and play around with some toy datasets to see if we can get close to the theoretical limits.</p> <p>(You might also want to checkout my previous post where I played around with ResNets: <a class="reference external" href="http://bjlkeng.github.io/posts/residual-networks/">Residual Networks</a>)</p> <p><a href="http://bjlkeng.github.io/posts/universal-resnet-the-one-neuron-approximator/">Read more…</a> (11 min remaining to read)</p></div>hidden layersmathjaxneural networksresidual networksResNetuniversal approximatorhttp://bjlkeng.github.io/posts/universal-resnet-the-one-neuron-approximator/Fri, 03 Aug 2018 12:03:28 GMThttp://bjlkeng.github.io/posts/residual-networks/Brian Keng<div><p>Taking a small break from some of the heavier math, I thought I'd write a post (aka learn more about) a very popular neural network architecture called Residual Networks aka ResNet. This architecture is being very widely used because it's so simple yet so powerful at the same time. The architecture's performance is due its ability to add hundreds of layers (talk about deep learning!) without degrading performance or adding difficulty to training. I really like these types of robust advances where it doesn't require fiddling with all sorts of hyper-parameters to make it work. Anyways, I'll introduce the idea and show an implementation of ResNet on a few runs of a variational autoencoder that I put together on the CIFAR10 dataset.</p> <p><a href="http://bjlkeng.github.io/posts/residual-networks/">Read more…</a> (9 min remaining to read)</p></div>autoencodersCIFAR10mathjaxresidual networksResNethttp://bjlkeng.github.io/posts/residual-networks/Sun, 18 Feb 2018 18:55:13 GMT