I first saw the term "entropy" in a chemistry course while studying thermodynamics.
During my graduate studies I encountered the term in many different areas of mathematics.
Can anyone explain why this term is used and what it means. What I am looking for is a few examples where the term "entropy" is used to describe some mathematical object/quantity and its meaning there.



What is entropy, really?
Here is a simple story one can tell about the entropy 


$$H = -\sum\_{i=1}^n p\_i \log p\_i$$ 


of a discrete probability distribution. Suppose you wanted to describe how surprised you are upon learning that some event $E$ happened. Call your surprise upon learning that $E$ happened $s(E)$, the "surprisal." Here are some plausible conditions that $s$ could satisfy:


* $s(E)$ is a decreasing function of the probability $\mathbb{P}(E)$. That is, the less likely something it is to happen, the more surprising it is that it ends up happening, and the likelihood of something happening is the only thing determining how surprising it is. For example, flipping $10$ heads in a row is more surprising than flipping $5$ heads in a row.
* If $E\_1$ and $E\_2$ are independent, then $s(E\_1 \cap E\_2) = s(E\_1) + s(E\_2)$. That is, your surprise at learning that two independent events happened should be the sum of your surprises at learning that each individual event happened. For example, flipping $10$ coins heads in a row is twice as surprising as flipping $5$ coins heads in a row.


**Exercise:** These conditions imply that $s$ must be a positive scalar multiple of $- \log \mathbb{P}(E)$. 


Then the expected surprisal is a positive scalar multiple of the entropy. Note in particular that $H$ is minimized if some $p\_i = 1$, which corresponds to the same thing always happening and which is not surprising at all.


