Bounded Rationality (Posts about Jaynes)http://bjlkeng.github.io/enSat, 03 Aug 2024 01:42:51 GMTNikola (getnikola.com)http://blogs.law.harvard.edu/tech/rssThe Logic Behind the Maximum Entropy Principlehttp://bjlkeng.github.io/posts/the-logic-behind-entropy/Brian Keng<div><p>For a while now, I've really enjoyed diving deep to understand
probability and related fundamentals (see
<a class="reference external" href="http://bjlkeng.github.io/posts/probability-the-logic-of-science/">here</a>,
<a class="reference external" href="http://bjlkeng.github.io/posts/maximum-entropy-distributions/">here</a>, and
<a class="reference external" href="http://bjlkeng.github.io/posts/an-introduction-to-stochastic-calculus/">here</a>).
Entropy is a topic that comes up all over the place from physics to information
theory, and of course, machine learning. I written about it in various
different forms but always taken it as a given as the "expected information".
Well I found a few of good explanations about how to "derive" it and thought
that I should share.</p>
<p>In this post, I'll be showing a few of derivations of the maximum entropy
principle, where entropy appears as part of the definition. These derivations
will show why it is a reasonable and natural thing to maximize, and how it is
determined from some well thought out reasoning. This post will be more math
heavy but hopefully it will give you more insight into this wonderfully
surprising topic.</p>
<p><a href="http://bjlkeng.github.io/posts/the-logic-behind-entropy/">Read more…</a> (16 min remaining to read)</p></div>entropyinformationJaynesmathjaxShannonWallishttp://bjlkeng.github.io/posts/the-logic-behind-entropy/Sat, 03 Aug 2024 00:44:59 GMTProbability as Extended Logichttp://bjlkeng.github.io/posts/probability-the-logic-of-science/Brian Keng<div><p>Modern probability theory is typically derived from the
<a class="reference external" href="https://en.wikipedia.org/wiki/Probability_axioms">Kolmogorov axioms</a>,
using measure theory with concepts like events and sample space.
In one way, it's intuitive to understand how this works as Laplace
<a class="reference external" href="https://en.wikipedia.org/wiki/Classical_definition_of_probability">wrote</a>:</p>
<blockquote>
<p>The probability of an event is the ratio of the number of cases favorable
to it, to the number of all cases possible, when [the cases are] equally
possible. ... Probability is thus simply a fraction whose numerator is the
number of favorable cases and whose denominator is the number of all the
cases possible.</p>
</blockquote>
<p>However, the intuition of this view of probability breaks down when we want to
do more complex reasoning. After learning probability from the lens of coins,
dice and urns full of red and white balls, I still didn't feel that I had
have a strong grasp about how to apply it to other situations -- especially
ones where it was difficult or too abstract to apply the idea of <em>"a fraction
whose numerator is the number of favorable cases and whose denominator is the
number of all the cases possible"</em>. And then I read <a class="reference external" href="http://www.cambridge.org/gb/academic/subjects/physics/theoretical-physics-and-mathematical-physics/probability-theory-logic-science">Probability Theory: The Logic of Science</a> by E. T. Jaynes.</p>
<p>Jaynes takes a drastically different approach to probability, not with events and
sample spaces, but rather as an extension of Boolean logic. Taking this view made
a great deal of sense to me since I spent a lot of time <a class="reference external" href="http://bjlkeng.github.io/posts/accessible-satisfiability/">studying and reasoning</a> in Boolean logic. The following post
is my attempt to explain Jaynes' view of probability theory, where he derives
it from "common sense" extensions to Boolean logic. (<em>Spoiler alert: he ends
up with pretty much the same mathematical system as Kolmogorov's probability
theory.</em>) I'll stay away from any heavy derivations and stick with the
intuition, which is exactly where I think this view of probability theory is most
useful.</p>
<p><a href="http://bjlkeng.github.io/posts/probability-the-logic-of-science/">Read more…</a> (14 min remaining to read)</p></div>Jayneslogicmathjaxprobabilityhttp://bjlkeng.github.io/posts/probability-the-logic-of-science/Thu, 15 Oct 2015 00:30:05 GMT