diff --git a/_subfiles/_exm-cross-entropy-decomposition.qmd b/_subfiles/_exm-cross-entropy-decomposition.qmd index 0fc04ee..b45013e 100644 --- a/_subfiles/_exm-cross-entropy-decomposition.qmd +++ b/_subfiles/_exm-cross-entropy-decomposition.qmd @@ -21,9 +21,9 @@ c( ) ``` -The cross-entropy is $H_2(p, q) = `r round(cross_entropy, 3)`$ bits. -The entropy is $H_2(p) = `r shannon_entropy`$ bits, -and the divergence is $D_2(p \,\|\, q) = `r round(divergence, 3)`$ bits. +The cross-entropy is $\crossentropyf{p}{q} = `r round(cross_entropy, 3)`$ bits. +The entropy is $\entropyf{p} = `r shannon_entropy`$ bits, +and the divergence is $\kldivf{p}{q} = `r round(divergence, 3)`$ bits. They satisfy @thm-cross-entropy-decomposition: `r shannon_entropy` + `r round(divergence, 3)` = `r round(cross_entropy, 3)`. diff --git a/_subfiles/_rem-cross-entropy-loss.qmd b/_subfiles/_rem-cross-entropy-loss.qmd index c4ed051..838dc57 100644 --- a/_subfiles/_rem-cross-entropy-loss.qmd +++ b/_subfiles/_rem-cross-entropy-loss.qmd @@ -10,9 +10,11 @@ toward the distribution the labels came from. For a single labeled example, let $e$ be the distribution that puts all its mass on the observed label. -Then $H_2(e) = 0$ +Then $\entropyf{e} = 0$ ([as for a coin that always lands heads](https://morrison-lab.github.io/pds/expectation.html#exm-entropy-coin)), -so the example's cross-entropy loss $H_2(e, q)$ equals $D_2(e \,\|\, q)$: +so the example's cross-entropy +[loss](https://morrison-lab.github.io/pds/expectation.html#def-loss-function) $\crossentropyf{e}{q}$ +equals $\kldivf{e}{q}$: every bit the loss reports is a wasted one. ::: diff --git a/_subfiles/_rem-cross-entropy-wrong-code.qmd b/_subfiles/_rem-cross-entropy-wrong-code.qmd index dad7f9e..b93aa3c 100644 --- a/_subfiles/_rem-cross-entropy-wrong-code.qmd +++ b/_subfiles/_rem-cross-entropy-wrong-code.qmd @@ -2,13 +2,13 @@ #### Cross-entropy is the cost of the wrong code @thm-cross-entropy-decomposition splits the [cross-entropy](https://morrison-lab.github.io/pds/expectation.html#def-cross-entropy) into the [entropy](https://morrison-lab.github.io/pds/expectation.html#def-entropy) and the divergence. -By @thm-source-coding, no uniquely decodable code averages fewer than $H_2(p)$ bits. +By @thm-source-coding, no [uniquely decodable code](#def-uniquely-decodable) averages fewer than $\entropyf{p}$ [bits](entropy.qmd#def-bit). By @thm-wrong-code, a code built for $q$ -costs about $D_2(p \,\|\, q)$ bits per value more than that. +costs about $\kldivf{p}{q}$ bits per value more than that. By the [divergence theorem](entropy.qmd#thm-kl-divergence-nonnegative), the excess vanishes only when the code was built for $p$ [@cover2006elements, section 2.6; @yeung2008information, section 2.5]. @cover2006elements [section 5.4] state @thm-wrong-code -in this form, with $H_2(p) + D_2(p \,\|\, q)$ in place of $H_2(p, q)$. +in this form, with $\entropyf{p} + \kldivf{p}{q}$ in place of $\crossentropyf{p}{q}$. ::: diff --git a/_subfiles/_thm-cross-entropy-decomposition.qmd b/_subfiles/_thm-cross-entropy-decomposition.qmd index 3dc46eb..13d50bc 100644 --- a/_subfiles/_thm-cross-entropy-decomposition.qmd +++ b/_subfiles/_thm-cross-entropy-decomposition.qmd @@ -3,13 +3,13 @@ Let $p$ and $q$ be [probability mass functions](https://morrison-lab.github.io/pds/random-variables.html#def-pmf) on the values $x_1, \ldots, x_n$, with $q(x) > 0$ wherever $p(x) > 0$. -Write $H_2(p)$ for the [entropy](https://morrison-lab.github.io/pds/expectation.html#def-entropy) of $p$, -$H_2(p, q)$ for the [cross-entropy](https://morrison-lab.github.io/pds/expectation.html#def-cross-entropy) of $q$ relative to $p$, -and $D_2(p \,\|\, q)$ for the [Kullback--Leibler divergence](https://morrison-lab.github.io/pds/expectation.html#def-kl-divergence) of $q$ from $p$, +Write $\entropyf{p}$ for the [entropy](https://morrison-lab.github.io/pds/expectation.html#def-entropy) of $p$, +$\crossentropyf{p}{q}$ for the [cross-entropy](https://morrison-lab.github.io/pds/expectation.html#def-cross-entropy) of $q$ relative to $p$, +and $\kldivf{p}{q}$ for the [Kullback--Leibler divergence](https://morrison-lab.github.io/pds/expectation.html#def-kl-divergence) of $q$ from $p$, all with base-2 [logarithms](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-logarithm). Then -$$H_2(p, q) = H_2(p) + D_2(p \,\|\, q).$$ +$$\crossentropyf{p}{q} = \entropyf{p} + \kldivf{p}{q}.$$ ::: @@ -30,6 +30,6 @@ $$ = -\sum_x p(x) \log_2 p(x) + \sum_x p(x) \log_2 \frac{p(x)}{q(x)}. $$ -Reading left to right, these are $H_2(p, q)$, $H_2(p)$ and $D_2(p \,\|\, q)$. +Reading left to right, these are $\crossentropyf{p}{q}$, $\entropyf{p}$ and $\kldivf{p}{q}$. ::: diff --git a/_subfiles/_thm-kl-divergence-nonnegative.qmd b/_subfiles/_thm-kl-divergence-nonnegative.qmd index 88c9e21..19e5b7a 100644 --- a/_subfiles/_thm-kl-divergence-nonnegative.qmd +++ b/_subfiles/_thm-kl-divergence-nonnegative.qmd @@ -7,11 +7,13 @@ and suppose that $$\sum_x p(x) \, \bigl\lvert \log_2 \bigl( p(x) / q(x) \bigr) \bigr\rvert < \infty.$$ -Then the [Kullback--Leibler divergence](https://morrison-lab.github.io/pds/expectation.html#def-kl-divergence) satisfies +Then the [Kullback--Leibler divergence](https://morrison-lab.github.io/pds/expectation.html#def-kl-divergence), +with base-2 [logarithms](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-logarithm), +satisfies -$$D_2(p \,\|\, q) \ge 0,$$ +$$\kldivf{p}{q} \ge 0,$$ -with $D_2(p \,\|\, q) = 0$ exactly when $q = p$. +with $\kldivf{p}{q} = 0$ exactly when $q = p$. ::: @@ -38,9 +40,11 @@ $$ \le 0. $$ -The left side is $-\ln 2 \cdot D_2(p \,\|\, q)$, so $D_2(p \,\|\, q) \ge 0$. +The left side is $-\ln 2 \cdot \kldivf{p}{q}$, +because the divergence uses base-2 [logarithms](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-logarithm), +so $\kldivf{p}{q} \ge 0$. -If $D_2(p \,\|\, q) = 0$, both relations hold with equality. +If $\kldivf{p}{q} = 0$, both relations hold with equality. The first forces $q(x) = p(x)$ at every $x$ with $p(x) > 0$. The second then gives $\sum_x q(x) = 1$, so $q$ has no mass where $p$ has none, and $q = p$. diff --git a/_subfiles/_thm-source-coding.qmd b/_subfiles/_thm-source-coding.qmd index 15925cf..2072b2f 100644 --- a/_subfiles/_thm-source-coding.qmd +++ b/_subfiles/_thm-source-coding.qmd @@ -3,18 +3,18 @@ Let $X$ be a discrete random variable that takes finitely many values, at least two of them with positive probability, -and let $H_2(X)$ be its +and let $\entropyf{X}$ be its [entropy](https://morrison-lab.github.io/pds/expectation.html#def-entropy) with base-2 logarithms. 1. Every [uniquely decodable](#def-uniquely-decodable) binary code for $X$ - has [expected length](#def-expected-length) $L \ge H_2(X)$. + has [expected length](#def-expected-length) $L \ge \entropyf{X}$. 2. Some uniquely decodable binary code for $X$ has expected length - $L < H_2(X) + 1$. + $L < \entropyf{X} + 1$. 3. Coding blocks of $n$ independent copies of $X$ at a time, some uniquely decodable binary code spends, on average, - fewer than $H_2(X) + 1/n$ binary digits per copy. + fewer than $\entropyf{X} + 1/n$ [binary digits](entropy.qmd#def-bit) per copy. ::: diff --git a/_subfiles/_thm-wrong-code.qmd b/_subfiles/_thm-wrong-code.qmd index b66b776..5b0815f 100644 --- a/_subfiles/_thm-wrong-code.qmd +++ b/_subfiles/_thm-wrong-code.qmd @@ -8,9 +8,9 @@ Suppose a code gives each value $x$ with $p(x) > 0$ the length $\ell(x) = \lceil -\log_2 q(x) \rceil$. Then its [expected length](#def-expected-length) $L$ under $p$ satisfies -$$H_2(p, q) \le L < H_2(p, q) + 1,$$ +$$\crossentropyf{p}{q} \le L < \crossentropyf{p}{q} + 1,$$ -where $H_2(p, q)$ is the +where $\crossentropyf{p}{q}$ is the [cross-entropy](https://morrison-lab.github.io/pds/expectation.html#def-cross-entropy) of $q$ relative to $p$, with base-2 logarithms. @@ -32,6 +32,7 @@ $$ < -\sum_{x} p(x) \log_2 q(x) + 1. $$ -The middle sum is $L$ by @thm-expected-length-sum, and the outer sums are $H_2(p, q)$. +The middle sum is $L$ by @thm-expected-length-sum, +and the outer sums are $\crossentropyf{p}{q}$. :::