Definition of information entropy

source ：https://www.zhihu.com/question/30828247/answer/64816509

Let me explain why the definition of entropy is as shown in the title from a very intuitive point of view .

First of all , Suppose there is a random variable , You can ask yourself when we observe a sample of this random variable , How much information can we receive ？ without doubt , When we are told that a highly unlikely thing has happened , Then we will receive more information ; And when we observe a very common thing happened , Then we receive a relatively small amount of information . Therefore, the measurement of information should depend on the probability distribution , So entropy The definition of should be a monotonic function of probability .

second , Suppose two random variables and It's independent of each other , Then the amount of information obtained by observing two variables separately should be the same as that obtained by observing two variables at the same time , namely ：. And in terms of probability , Two independent random variables mean , So it can be concluded here that the definition of entropy It should be probability Of function . Therefore, the entropy of a random variable can be defined as follows ：

The minus sign here is only used to guarantee entropy （ The amount of information ） Is a positive number or zero . and The choice of function basis is arbitrary （ In information theory, the base is often chosen as 2, Therefore, the unit of information is bit bits; In machine learning, the base is often chosen as a natural constant , Therefore, units are often called nats）.

Last , We use entropy to evaluate the entire random variable The average amount of information , And the best measure of average is the expectation of random variables , The definition of entropy is as follows ：