Bounded Rationality (Posts about hidden layers)http://bjlkeng.github.io/enSat, 23 Dec 2023 02:03:09 GMTNikola (getnikola.com)http://blogs.law.harvard.edu/tech/rssUniversal ResNet: The One-Neuron Approximatorhttp://bjlkeng.github.io/posts/universal-resnet-the-one-neuron-approximator/Brian Keng<div><p><em>"In theory, theory and practice are the same. In practice, they are not."</em></p>
<p>I read a very interesting paper titled <em>ResNet with one-neuron hidden layers is
a Universal Approximator</em> by Lin and Jegelka [1].
The paper describes a simplified Residual Network as a universal approximator,
giving some theoretical backing to the wildly successful ResNet architecture.
In this post, I'm going to talk about this paper and a few of the related
universal approximation theorems for neural networks.
Instead of going through all the theoretical stuff, I'm simply going introduce
some theorems and play around with some toy datasets to see if we can get close
to the theoretical limits.</p>
<p>(You might also want to checkout my previous post where I played around with
ResNets: <a class="reference external" href="http://bjlkeng.github.io/posts/residual-networks/">Residual Networks</a>)</p>
<p><a href="http://bjlkeng.github.io/posts/universal-resnet-the-one-neuron-approximator/">Read moreā¦</a> (11 min remaining to read)</p></div>hidden layersmathjaxneural networksresidual networksResNetuniversal approximatorhttp://bjlkeng.github.io/posts/universal-resnet-the-one-neuron-approximator/Fri, 03 Aug 2018 12:03:28 GMT