Write the base-2 entropy, cross-entropy and divergence with their macros

_subfiles/_exm-cross-entropy-decomposition.qmd

@@ -21,9 +21,9 @@ c(
 )
 ```
 
-The cross-entropy is $H_2(p, q) = `r round(cross_entropy, 3)`$ bits.
-The entropy is $H_2(p) = `r shannon_entropy`$ bits,
-and the divergence is $D_2(p \,\|\, q) = `r round(divergence, 3)`$ bits.
+The cross-entropy is $\crossentropyf{p}{q} = `r round(cross_entropy, 3)`$ bits.
+The entropy is $\entropyf{p} = `r shannon_entropy`$ bits,
+and the divergence is $\kldivf{p}{q} = `r round(divergence, 3)`$ bits.
 They satisfy @thm-cross-entropy-decomposition:
 `r shannon_entropy` + `r round(divergence, 3)` = `r round(cross_entropy, 3)`.
 

_subfiles/_rem-cross-entropy-loss.qmd

@@ -10,9 +10,11 @@ toward the distribution the labels came from.
 
 For a single labeled example,
 let $e$ be the distribution that puts all its mass on the observed label.
-Then $H_2(e) = 0$
+Then $\entropyf{e} = 0$
 ([as for a coin that always lands heads](https://morrison-lab.github.io/pds/expectation.html#exm-entropy-coin)),
-so the example's cross-entropy loss $H_2(e, q)$ equals $D_2(e \,\|\, q)$:
+so the example's cross-entropy
+[loss](https://morrison-lab.github.io/pds/expectation.html#def-loss-function) $\crossentropyf{e}{q}$
+equals $\kldivf{e}{q}$:
 every bit the loss reports is a wasted one.
 
 :::

_subfiles/_rem-cross-entropy-wrong-code.qmd

@@ -2,13 +2,13 @@
 #### Cross-entropy is the cost of the wrong code
 
 @thm-cross-entropy-decomposition splits the [cross-entropy](https://morrison-lab.github.io/pds/expectation.html#def-cross-entropy) into the [entropy](https://morrison-lab.github.io/pds/expectation.html#def-entropy) and the divergence.
-By @thm-source-coding, no uniquely decodable code averages fewer than $H_2(p)$ bits.
+By @thm-source-coding, no [uniquely decodable code](#def-uniquely-decodable) averages fewer than $\entropyf{p}$ [bits](entropy.qmd#def-bit).
 By @thm-wrong-code, a code built for $q$
-costs about $D_2(p \,\|\, q)$ bits per value more than that.
+costs about $\kldivf{p}{q}$ bits per value more than that.
 By the [divergence theorem](entropy.qmd#thm-kl-divergence-nonnegative), the excess vanishes
 only when the code was built for $p$
 [@cover2006elements, section 2.6; @yeung2008information, section 2.5].
 @cover2006elements [section 5.4] state @thm-wrong-code
-in this form, with $H_2(p) + D_2(p \,\|\, q)$ in place of $H_2(p, q)$.
+in this form, with $\entropyf{p} + \kldivf{p}{q}$ in place of $\crossentropyf{p}{q}$.
 
 :::

_subfiles/_thm-cross-entropy-decomposition.qmd

@@ -3,13 +3,13 @@
 
 Let $p$ and $q$ be [probability mass functions](https://morrison-lab.github.io/pds/random-variables.html#def-pmf) on the values $x_1, \ldots, x_n$,
 with $q(x) > 0$ wherever $p(x) > 0$.
-Write $H_2(p)$ for the [entropy](https://morrison-lab.github.io/pds/expectation.html#def-entropy) of $p$,
-$H_2(p, q)$ for the [cross-entropy](https://morrison-lab.github.io/pds/expectation.html#def-cross-entropy) of $q$ relative to $p$,
-and $D_2(p \,\|\, q)$ for the [Kullback--Leibler divergence](https://morrison-lab.github.io/pds/expectation.html#def-kl-divergence) of $q$ from $p$,
+Write $\entropyf{p}$ for the [entropy](https://morrison-lab.github.io/pds/expectation.html#def-entropy) of $p$,
+$\crossentropyf{p}{q}$ for the [cross-entropy](https://morrison-lab.github.io/pds/expectation.html#def-cross-entropy) of $q$ relative to $p$,
+and $\kldivf{p}{q}$ for the [Kullback--Leibler divergence](https://morrison-lab.github.io/pds/expectation.html#def-kl-divergence) of $q$ from $p$,
 all with base-2 [logarithms](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-logarithm).
 Then
 
-$$H_2(p, q) = H_2(p) + D_2(p \,\|\, q).$$
+$$\crossentropyf{p}{q} = \entropyf{p} + \kldivf{p}{q}.$$
 
 :::
 
@@ -30,6 +30,6 @@ $$
 = -\sum_x p(x) \log_2 p(x) + \sum_x p(x) \log_2 \frac{p(x)}{q(x)}.
 $$
 
-Reading left to right, these are $H_2(p, q)$, $H_2(p)$ and $D_2(p \,\|\, q)$.
+Reading left to right, these are $\crossentropyf{p}{q}$, $\entropyf{p}$ and $\kldivf{p}{q}$.
 
 :::

_subfiles/_thm-kl-divergence-nonnegative.qmd

@@ -7,11 +7,13 @@ and suppose that
 
 $$\sum_x p(x) \, \bigl\lvert \log_2 \bigl( p(x) / q(x) \bigr) \bigr\rvert < \infty.$$
 
-Then the [Kullback--Leibler divergence](https://morrison-lab.github.io/pds/expectation.html#def-kl-divergence) satisfies
+Then the [Kullback--Leibler divergence](https://morrison-lab.github.io/pds/expectation.html#def-kl-divergence),
+with base-2 [logarithms](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-logarithm),
+satisfies
 
-$$D_2(p \,\|\, q) \ge 0,$$
+$$\kldivf{p}{q} \ge 0,$$
 
-with $D_2(p \,\|\, q) = 0$ exactly when $q = p$.
+with $\kldivf{p}{q} = 0$ exactly when $q = p$.
 
 :::
 
@@ -38,9 +40,11 @@ $$
 \le 0.
 $$
 
-The left side is $-\ln 2 \cdot D_2(p \,\|\, q)$, so $D_2(p \,\|\, q) \ge 0$.
+The left side is $-\ln 2 \cdot \kldivf{p}{q}$,
+because the divergence uses base-2 [logarithms](https://morrison-lab.github.io/mds/algebra-exponentials.html#def-logarithm),
+so $\kldivf{p}{q} \ge 0$.
 
-If $D_2(p \,\|\, q) = 0$, both relations hold with equality.
+If $\kldivf{p}{q} = 0$, both relations hold with equality.
 The first forces $q(x) = p(x)$ at every $x$ with $p(x) > 0$.
 The second then gives $\sum_x q(x) = 1$,
 so $q$ has no mass where $p$ has none, and $q = p$.

_subfiles/_thm-source-coding.qmd

@@ -3,18 +3,18 @@
 
 Let $X$ be a discrete random variable
 that takes finitely many values, at least two of them with positive probability,
-and let $H_2(X)$ be its
+and let $\entropyf{X}$ be its
 [entropy](https://morrison-lab.github.io/pds/expectation.html#def-entropy)
 with base-2 logarithms.
 
 1. Every [uniquely decodable](#def-uniquely-decodable) binary code for $X$
-   has [expected length](#def-expected-length) $L \ge H_2(X)$.
+   has [expected length](#def-expected-length) $L \ge \entropyf{X}$.
 
 2. Some uniquely decodable binary code for $X$ has expected length
-   $L < H_2(X) + 1$.
+   $L < \entropyf{X} + 1$.
 
 3. Coding blocks of $n$ independent copies of $X$ at a time,
    some uniquely decodable binary code spends, on average,
-   fewer than $H_2(X) + 1/n$ binary digits per copy.
+   fewer than $\entropyf{X} + 1/n$ [binary digits](entropy.qmd#def-bit) per copy.
 
 :::

_subfiles/_thm-wrong-code.qmd

@@ -8,9 +8,9 @@ Suppose a code gives each value $x$ with $p(x) > 0$
 the length $\ell(x) = \lceil -\log_2 q(x) \rceil$.
 Then its [expected length](#def-expected-length) $L$ under $p$ satisfies
 
-$$H_2(p, q) \le L < H_2(p, q) + 1,$$
+$$\crossentropyf{p}{q} \le L < \crossentropyf{p}{q} + 1,$$
 
-where $H_2(p, q)$ is the
+where $\crossentropyf{p}{q}$ is the
 [cross-entropy](https://morrison-lab.github.io/pds/expectation.html#def-cross-entropy)
 of $q$ relative to $p$, with base-2 logarithms.
 
@@ -32,6 +32,7 @@ $$
 < -\sum_{x} p(x) \log_2 q(x) + 1.
 $$
 
-The middle sum is $L$ by @thm-expected-length-sum, and the outer sums are $H_2(p, q)$.
+The middle sum is $L$ by @thm-expected-length-sum,
+and the outer sums are $\crossentropyf{p}{q}$.
 
 :::