#Maths/Probability-Theory #Maths/Information-Theory #Note

Kullback-Leiber Divergence

Definition

For two distributions $p$ and $q$ , the Kullback-Leiber divergence (KL divergence) is defined as

D_{KL} (p ∣∣ q) := H (p, q) - H (p) = E_{p} [\log \frac{p}{q}] .

The KL divergence can be used to measure how much $p$ differs from $q$ .

Properties

From Gibb's inequality, $D_{KL} (p ∣∣ q) \geq 0$ , and the equality holds if and only if $p = q$ .
The KL divergence is additive for independent distributions $p_{1} ⊥ p_{2}$ and $q_{1} ⊥ q_{2}$ :

D_{KL} (p_{1} ∣∣ q_{1}) + D_{KL} (p_{2} ∣∣ q_{2}) = D_{KL} (p_{1} p_{2} ∣∣ q_{1} q_{2}) .

KL divergence is not a Metric Distance

KL divergence is not a metric distance, because it does not satisfies the symmetric axiom and the triangle inequality. Jensen–Shannon Divergence, or JS divergence, is a symmetrized version of it.