I hope to understand the connection between statistics and information theory in a deep philosophical sense.


I suppose the best place to start would be David MacKay's Information Theory, Inference, and Learning Algorithms, but I was curious what else there may be. Are there any other good books out there like this? What are your recommendations?



Books to develop a unified view of statistics and information theory?
The booklength tutorial by Shannon award winner Imre Csiszár and Paul Shields is freely available online [here](https://www.nowpublishers.com/article/Details/CIT-004):


*Information Theory and Statistics: A Tutorial*
I. Csiszár, Rényi Institute of Mathematics, Hungarian Academy of Sciences


There is also an [article](https://link.springer.com/referenceworkentry/10.1007%2F978-3-642-04898-2_643) in the International Encyclopeadia of Statistical Science by E. Haroutunian which seems to be behind a paywall.


