Bounded Rationality (Posts about ResNet)http://bjlkeng.github.io/enSat, 03 Aug 2024 01:42:51 GMTNikola (getnikola.com)http://blogs.law.harvard.edu/tech/rssUniversal ResNet: The One-Neuron Approximatorhttp://bjlkeng.github.io/posts/universal-resnet-the-one-neuron-approximator/Brian Keng<div><p><em>"In theory, theory and practice are the same. In practice, they are not."</em></p>
<p>I read a very interesting paper titled <em>ResNet with one-neuron hidden layers is
a Universal Approximator</em> by Lin and Jegelka [1].
The paper describes a simplified Residual Network as a universal approximator,
giving some theoretical backing to the wildly successful ResNet architecture.
In this post, I'm going to talk about this paper and a few of the related
universal approximation theorems for neural networks.
Instead of going through all the theoretical stuff, I'm simply going introduce
some theorems and play around with some toy datasets to see if we can get close
to the theoretical limits.</p>
<p>(You might also want to checkout my previous post where I played around with
ResNets: <a class="reference external" href="http://bjlkeng.github.io/posts/residual-networks/">Residual Networks</a>)</p>
<p><a href="http://bjlkeng.github.io/posts/universal-resnet-the-one-neuron-approximator/">Read more…</a> (11 min remaining to read)</p></div>hidden layersmathjaxneural networksresidual networksResNetuniversal approximatorhttp://bjlkeng.github.io/posts/universal-resnet-the-one-neuron-approximator/Fri, 03 Aug 2018 12:03:28 GMTResidual Networkshttp://bjlkeng.github.io/posts/residual-networks/Brian Keng<div><p>Taking a small break from some of the heavier math, I thought I'd write a post
(aka learn more about) a very popular neural network architecture called
Residual Networks aka ResNet. This architecture is being very widely used
because it's so simple yet so powerful at the same time. The architecture's
performance is due its ability to add hundreds of layers (talk about deep
learning!) without degrading performance or adding difficulty to training. I
really like these types of robust advances where it doesn't require fiddling
with all sorts of hyper-parameters to make it work. Anyways, I'll introduce
the idea and show an implementation of ResNet on a few runs of a variational
autoencoder that I put together on the CIFAR10 dataset.</p>
<p><a href="http://bjlkeng.github.io/posts/residual-networks/">Read more…</a> (9 min remaining to read)</p></div>autoencodersCIFAR10mathjaxresidual networksResNethttp://bjlkeng.github.io/posts/residual-networks/Sun, 18 Feb 2018 18:55:13 GMT