|
How much one has learned from an experiment is quantifiable by the information gain, also known as the Kullback–Leibler divergence. The narrowing of the posterior parameter distribution P ( θ ∣ D ) compared with the prior parameter distribution π ( θ ), is quantified in units of bits, as: D KL ( P ∣ π ) = ∫ log 2 P ( θ ∣ D ) π ( θ ) P ( θ ∣ D ) d θ bits This research note gives an intuition what one bit of information gain means. It corresponds to a Gaussian shrinking its standard deviation by a factor of three.
|
