The Logic Behind the Maximum Entropy Principle

Brian Keng — Sat, 03 Aug 2024 00:44:59 GMT

For a while now, I've really enjoyed diving deep to understand probability and related fundamentals (see here, here, and here). Entropy is a topic that comes up all over the place from physics to information theory, and of course, machine learning. I written about it in various different forms but always taken it as a given as the "expected information". Well I found a few of good explanations about how to "derive" it and thought that I should share.

In this post, I'll be showing a few of derivations of the maximum entropy principle, where entropy appears as part of the definition. These derivations will show why it is a reasonable and natural thing to maximize, and how it is determined from some well thought out reasoning. This post will be more math heavy but hopefully it will give you more insight into this wonderfully surprising topic.

Probability as Extended Logic

Brian Keng — Thu, 15 Oct 2015 00:30:05 GMT

Modern probability theory is typically derived from the Kolmogorov axioms, using measure theory with concepts like events and sample space. In one way, it's intuitive to understand how this works as Laplace wrote:

The probability of an event is the ratio of the number of cases favorable to it, to the number of all cases possible, when [the cases are] equally possible. ... Probability is thus simply a fraction whose numerator is the number of favorable cases and whose denominator is the number of all the cases possible.

However, the intuition of this view of probability breaks down when we want to do more complex reasoning. After learning probability from the lens of coins, dice and urns full of red and white balls, I still didn't feel that I had have a strong grasp about how to apply it to other situations -- especially ones where it was difficult or too abstract to apply the idea of "a fraction whose numerator is the number of favorable cases and whose denominator is the number of all the cases possible". And then I read Probability Theory: The Logic of Science by E. T. Jaynes.

Jaynes takes a drastically different approach to probability, not with events and sample spaces, but rather as an extension of Boolean logic. Taking this view made a great deal of sense to me since I spent a lot of time studying and reasoning in Boolean logic. The following post is my attempt to explain Jaynes' view of probability theory, where he derives it from "common sense" extensions to Boolean logic. (Spoiler alert: he ends up with pretty much the same mathematical system as Kolmogorov's probability theory.) I'll stay away from any heavy derivations and stick with the intuition, which is exactly where I think this view of probability theory is most useful.

Bounded Rationality (Posts about Jaynes)

The Logic Behind the Maximum Entropy Principle

Probability as Extended Logic