Lagmental Vicfred

Kullback--Leibler Divergence Is Nonnegative by Vicfred

tags: mathematics, stochastic processes, order statistics, information theory

Relative entropy measures the expected log-likelihood ratio and vanishes only when the distributions agree. I will separate the object being defined from the consequence being claimed.

The mathematical object

Continuous-time processes are described by finite-dimensional distributions plus path regularity. A Poisson process \(N_t\) has independent increments, while Brownian motion \(B_t\) has Gaussian increments.

$$ D_{\mathrm{KL}}(P\|Q)=\sum_xP(x)\log\frac{P(x)}{Q(x)} $$

The first display fixes the mathematical data. I label it \(\mathsf{data}\) mentally, while the next is the \(\mathsf{claim}\); the bridge between them is the displayed \(\Longrightarrow\), not an automatic implication.

$$ D_{\mathrm{KL}}(P\|Q)\ge0 $$

One explicit computation

Now evaluate one representative case. The result should agree with the structural law above, but it is obtained without assuming the conclusion.

$$ -D_{\mathrm{KL}}(P\|Q)=\mathbf E_P\!\left[\log\frac{Q(X)}{P(X)}\right]\le\log\mathbf E_P\!\left[\frac{Q(X)}{P(X)}\right]=\log1=0 $$

Why the identity matters

The aligned summary deliberately puts the datum and conclusion on different rows. Mathematically, this is the distinction between specifying an object and proving a property of it.

$$ \begin{aligned} \mathsf{D}\;&:\quad D_{\mathrm{KL}}(P\|Q)=\sum_xP(x)\log\frac{P(x)}{Q(x)},\\[5pt] \mathsf{C}\;&:\quad D_{\mathrm{KL}}(P\|Q)\ge0. \end{aligned} $$

Where it can fail

Matching means and variances does not identify a distribution. Independence, increment laws, and sample-path properties are separate ingredients.

$$ \boxed{\begin{gathered} \text{compact conclusion}\\[-2pt] D_{\mathrm{KL}}(P\|Q)\ge0 \end{gathered}} $$

A symbolic summary is trustworthy only because the example and limitation remain visible beside it. The box compresses the conclusion without hiding its origin.

This article was posted on Sat 04 January 2025. Facts and circumstances may have changed since publication.
Please contact me before jumping to conclusions if something seems wrong or unclear.