IV. Integration and convergence

$$

$$

Find these notes at gregorkb.github.io/probtheory/

Integration

• Simple function: Given a measurable space \((\Omega,\mathcal{F})\), a function \(f:\Omega \to [-\infty,\infty]\) is called a simple function if there exist sets \(A_1,\dots,A_k\in \mathcal{F}\) forming a partition of \(\Omega\) and \(c_1,\dots,c_k \in [-\infty,\infty]\), \(k < \infty\), such that \(f = \sum_{i=1}^k c_i 1_{A_i}\).

• Check: Are simple functions measurable?

• Integral of a simple function: Given a measure space \((\Omega, \mathcal{F}, \mu)\), the integral with respect to \(\mu\) of a simple nonnegative function \(f:\Omega \to [0,\infty]\) with representation \(f = \sum_{i=1}^k c_i 1_{A_i}\) is defined as \[ \int f d\mu \equiv \sum_{i=1}^k c_i \mu (A_i), \] where we use the convention \(\infty \cdot 0 = 0\). See Definition 2.3.2 on page 49 of A&L. Compare to Section 5.2.1 starting on page 119 of Resnick. One can show that the integral is invariant to the representation of \(f\); see Lemma 4.1 on page 49 of Gut.

• Results on integrals of simple functions: Let \(f,g: \Omega \to \mathbb{R}\) be nonnegative simple functions on a measure space \((\Omega,\mathcal{F},\mu)\).

  1. Show \(\int (a f + b g) d \mu = a \int f d \mu + b \int g d \mu\) for any \(a,b \geq 0\) (Linearity).
  2. Show that \(f \geq g\) almost everywhere implies \(\int f d\mu \geq \int g d \mu\) (Monotonicity).
  3. Show that \(f = g\) almost everywhere implies \(\int f d \mu = \int g d \mu\).

  See Proposition 2.3.1 of page 49 of A&L. Compare to Theorem 4.1 on page 49 of Gut.

• Meaning of “almost everywhere”: If a condition holds almost everywhere (a.e.) with respect to a measure, it means that the subset of \(\omega \in \Omega\) for which it does not hold has measure zero. For example, \(f \geq g\) almost everywhere with respect to \(\mu\) means \(\mu(\{\omega : f(\omega) < g(\omega)\}) = 0\). Note that to say “almost everywhere”, you must have a measure in mind; in the context of a measure space \((\Omega,\mathcal{F},\mu)\), “almost everywhere” will mean almost everywhere with respect to \(\mu\).

• Integral of a nonnegative function: Let \(f:\Omega \to [0,\infty]\) be a nonnegative measurable function on the measure space \((\Omega,\mathcal{F},\mu)\) and let \(\{f_n\}_{n\geq 1}:\Omega \to [0,\infty]\) be a sequence of simple nonnegative functions on the same measure space such that \(f_n(\omega) \uparrow f(\omega)\) as \(n \to \infty\) for all \(\omega \in \Omega\). Then the integral of \(f\) with respect to \(\mu\) is defined as \[ \int f d \mu \equiv \lim_{n \to \infty} \int f_n d\mu. \]

• Invariance of integral to the sequence of simple functions: In the setup of the previous definition, let \(\{f_n\}_{n \geq 1}\) and \(\{g_n\}_{n \geq 1}\) be sequences of simple nonnegative functions such that \(f_n(\omega) \uparrow f(\omega)\) and \(g_n(\omega) \uparrow f(\omega)\) as \(n \to \infty\) for all \(\omega \in \Omega\). Show that \[ \lim_{n \to \infty} \int f_n d\mu = \lim_{n \to \infty} \int g_n d\mu. \]

• Note that we really only need \(f_n \uparrow f\) and \(g_n \uparrow f\) as \(n \to \infty\) almost everywhere.

• Example of a sequence of simple nonnegative functions: Given any nonnegative measurable function \(f\) on a measure space \((\Omega,\mathcal{F})\), one can construct a sequence of simple functions \(f_n \uparrow f\) as \[ f_n(\omega) = \left\{\begin{array}{ll} j(1/2)^n,& j(1/2)^n \leq f(\omega) < (j+1)(1/2)^n, \quad j = 0,1,\dots,n2^n - 1\\ n,& f(\omega) \geq n \end{array} \right. \] Draw a picture of this sequence of functions for \(n =1,2\) for a measurable nonnegative function \(f\) on \((\mathbb{R},\mathcal{B}(\mathbb{R}))\). Note that subdividing the domain as in Riemann integration is not guaranteed to result in a sequence of functions approaching \(f\) from below. See page 51 of A&L.

• Results on integrals of nonnegative measurable functions: Let \(f\) and \(g\) be nonnegative measurable functions on a measure space \((\Omega,\mathcal{F},\mu)\). Then one can show

  1. \(\int (a f + b g) d \mu = a \int f d \mu + b \int g d \mu\) for any \(a,b \geq 0\) (Linearity).
  2. \(f \geq g\) almost everywhere implies \(\int f d\mu \geq \int g d \mu\) (Monotonicity).
  3. \(f = g\) almost everywhere implies \(\int f d \mu = \int g d \mu\).

  These are the same as above for simple nonnegative functions.

• Monotone convergence theorem (MCT): Let \(\{f_n\}_{n \geq 1}\) and \(f\) be nonnegative measurable functions on a measure space \((\Omega,\mathcal{F},\mu)\) such that \(f_n \uparrow f\) as \(n \to \infty\) almost everywhere. Show that \[ \lim_{n \to \infty} \int f_n d \mu = \int \lim_{n \to \infty} f_n d\mu. \] See Corollary 2.3.5 on page 53 of A&L.

• Integral of infinite sum of nonnegative functions: Let \(\{f_n\}_{n \geq 1}\) be nonnegative measurable functions on a measure space \((\Omega, \mathcal{F},\mu)\). Show (use the MCT) that \[ \int \sum_{n \geq 1} f_n d \mu = \sum_{n\geq 1} \int f_n d \mu. \] See Corollary 2.3.5 on page 53 of A&L.

• Fatou’s Lemma: Let \(\{f_n\}_{n \geq 1}\) be nonnegative measurable functions on a measure space \((\Omega,\mathcal{F},\mu)\). Show that \[ \liminf_{n \to \infty} \int f_n d \mu \geq \int \liminf_{n \to \infty}f_n d \mu. \]

  See Theorem 2.3.7 on page 54 of A&L. Compare to Theorem 5.3.2 on page 132 of Resnick.

• Integral of any measurable function: Let \(f\) be a measurable function (not necessarily nonnegative) on a measure space \((\Omega,\mathcal{F},\mu)\). Define \(A \equiv \{\omega : f(\omega) \geq 0\}\) as well as the nonnegative functions \(f^+ \equiv f 1_A\) and \(f^- \equiv -f 1_{A^c}\). Then the integral of \(f\) with respect to \(\mu\) is defined as \[ \int f d \mu \equiv \int f^+ d \mu - \int f^- d \mu, \] provided at least one of the integrals on the right hand side is finite.

• Integral over a subset of the sample space: For a function \(f\) on a measure space \((\Omega,\mathcal{F},\mu)\) and a set \(A \in \mathcal{F}\), the integral of \(f\) over \(A\) with respect to \(\mu\) is defined as \[ \int_A f d \mu \equiv \int f 1_A d \mu. \]

• Integrability of a function: A function \(f\) on a measure space \((\Omega,\mathcal{F},\mu)\) is integrable with respect to \(\mu\) if \(\int |f|d \mu\) is finite.

• Notation for integration: The integral of \(f\) with respect to \(\mu\) can be written as \[ \int f d \mu \quad \text{ or } \quad \int f(\omega) d \mu(\omega) \quad \text{ or } \quad \int f(\omega) \mu(d \omega). \]

• \(L_p\) spaces: Given a measure space \((\Omega,\mathcal{F},\mu)\), define the collections of functions \[ \begin{align} L^p(\Omega,\mathcal{F},\mu) &\equiv \{f: \int |f|^p d \mu < \infty\}, \quad p \in (0,\infty)\\ L^\infty(\Omega,\mathcal{F},\mu) &\equiv \{f: \mu(\{\omega : f(\omega) > C \}) = 0 \text{ for some } C \in (0,\infty)\}. \end{align} \]

• \(L_p\) norms: Given a function \(g\) on a measure space \((\Omega,\mathcal{F},\mu)\), define \[ \|g\|_p = \left\{\begin{array}{ll} \int |g|^p d \mu& 0 < p < 1\\ (\int |g|^p d \mu)^{1/p},& 1 \leq p<\infty \\ \inf\{c \in (0,\infty) : \mu(\{\omega : g(\omega) > c\}) = 0\},& p = \infty \end{array}\right. \]

• Results on integrals of integrable functions: Let \(f,g \in L^1(\Omega,\mathcal{F},\mu)\). Then one can show

  1. \(\int (a f + b g) d \mu = a \int f d \mu + b \int g d \mu\) for any \(a,b \in \mathbb{R}\) (Linearity).
  2. \(f \geq g\) almost everywhere implies \(\int f d\mu \geq \int g d \mu\) (Monotonicity).
  3. \(f = g\) almost everywhere implies \(\int f d \mu = \int g d \mu\).

  These are the same as above for simple nonnegative functions.

• Identification of zero function: Let \(f\) be a measurable function on a measure space \((\Omega,\mathcal{F},\mu)\). Show that \(f = 0\) almost everywhere if and only if \(\int |f| d \mu = 0\). See Proposition 2.3.9 on page 56 of A&L.

• Finiteness almost everywhere of an integrable function: Let \(f\in L^1(\Omega,\mathcal{F},\mu)\). Show that \(f\) is finite almost everywhere. See Proposition 2.3.10 on page 57 of A&L.

• Two modes of convergence: Let \(\{f_n\}_{n \geq 1}\) and \(f\) be measurable functions on the measurable space \((\Omega,\mathcal{F},\mu)\). We say \(f_n\) converges to \(f\)

  • pointwise if \(\lim_{n \to \infty} f_n(\omega) = f(\omega)\) for all \(\omega \in \Omega\). We write \(f_n \to f\).

  • almost everywhere if there is a set \(B \in \mathcal{F}\) such that \(\lim_{n \to \infty} f_n(\omega) = f(\omega)\) for all \(\omega \in B\) and \(\mu(B^c)=0\). We write \(f_n \stackrel{\operatorname{a.e.}}{\longrightarrow}f\).

• Dominated convergence theorem (DCT): Let \(\{f_n\}_{n \geq 1}\), \(f\), and \(g\) be measurable functions on a measurable space \((\Omega,\mathcal{F},\mu)\) such that \(|f_n| < g\) almost everywhere for all \(n \geq 1\), \(g\) is integrable, and \(f_n \to f\) almost everywhere.

  1. Show that \(f\) is integrable.
  2. Show that \(\lim_{n\to \infty} \int f_nd \mu = \int f d \mu\).

  See Corollary 2.3.12 on page 57 of A&L or Theorem 16.4 on page 209 of Billingsley. Compare to Theorem 5.3.3 on page 133 of Resnick.

Convergence

• More modes of convergence: Let \(\{f_n\}_{n \geq 1}\) and \(f\) be measurable functions on a measure space \((\Omega,\mathcal{F},\mu)\). We say \(f_n\) converges to \(f\)

  • in measure if \(\lim_{n \to \infty} \mu(\{\omega: |f_n(\omega) - f(\omega)| > \epsilon\}) = 0\) for each \(\epsilon > 0\). We write \(f_n \stackrel{\mu}{\longrightarrow}f\).

  • in \(L^p\) if \(\lim_{n\to \infty} \|f_n - f\|_p = 0\). We write \(f_n \stackrel{L_p}{\longrightarrow}f\).

  • uniformly over \(~\Omega\) if \(\lim_{n \to \infty} \sup_{\omega \in \Omega} |f_n(\omega) - f(\omega)| = 0\).

• Limit infima and suprema of sequences of sets: For a sequence of sets \(\{A_n\}_{n \geq 1}\) define \[ \begin{align} \liminf_{n \to \infty} A_n &\equiv \bigcup_{k \geq 1} \bigcap_{n \geq k} A_n \\ &= \{\omega:\ \text{for some $k \geq 1$, $\omega \in A_n$ for all $n \geq k$}\}\\ &= \{\omega: \omega \in A_n \text{ for all but finitely many $n$}\} \end{align} \] as well as \[ \begin{align} \limsup_{n \to \infty} A_n &\equiv \bigcap_{k \geq 1} \bigcup_{n \geq k} A_n \\ &= \{\omega:\ \text{for all $k \geq 1$, $\omega \in A_n$ for some $n \geq k$}\}\\ &= \{\omega: \omega \in A_n \text{ for infinitely many $n$, ``infinitely often'' (i.o.)}\} \end{align} \]

• Limits of sequences of sets: If \(\liminf_{n\to \infty} A_n = \limsup_{n \to \infty} A_n = A\), then we define \(\lim_{n \to \infty} A_n \equiv A\).

• Limits of monotone sequences of sets: Let \(\{A_n\}_{n \geq 1}\) be a collection of sets.

  1. Show that \(A_n \subset A_{n+1}\) for all \(n\geq 1\) implies \(\lim_{n \to \infty } A_n = \bigcup_{n \geq 1} A_n\).
  2. Show that \(A_{n+1} \subset A_n\) for all \(n\geq 1\) implies \(\lim_{n \to \infty } A_n = \bigcap_{n \geq 1} A_n\).

• Some relationships between modes of convergence: Let \(\{f_n\}_{n \geq 1}\) and \(f\) be measurable functions on a measure space \((\Omega,\mathcal{F},\mu)\).

  1. Show that \(f_n \stackrel{\operatorname{a.e.}}{\longrightarrow}f\) implies \(f_n \stackrel{\mu}{\longrightarrow}f\).

  2. Show that \(f_n \stackrel{\mu}{\longrightarrow}f\) implies that there exists a subsequence \(\{n_k\}_{k \geq 1}\) such that \(f_{n_k} \stackrel{\operatorname{a.e.}}{\longrightarrow}f\).

  3. Show that \(f_n \stackrel{L_p}{\longrightarrow}f\) for \(0<p<\infty\) implies \(f_n \stackrel{\mu}{\longrightarrow}f\).

  See Theorems 2.5.1, 2.5.2, and 2.5.3 on page 63 of A&L. Compare to Theorems 6.2.1 and 6.3.1 on pages 170 and 171 of Resnick. Note to myself: See Page 31-32 of handwritten notes.

• Modes of convergence renamed for random variables: Let \(\{X_n\}_{n \geq 1}\) and \(X\) be random variables on a probability space \((\Omega,\mathcal{F},P)\). Then

  1. \(X_n \stackrel{\operatorname{a.e.}}{\longrightarrow}X\) is usually written as \(X_n \stackrel{\text{a.s.}}{\longrightarrow}X\) and read as “\(X_n\) converges almost surely to \(X\),” or, equivalently, as \(X_n \stackrel{\operatorname{w.p.1}}{\longrightarrow}X\) and read as “\(X_n\) converges to \(X\) with probability one.”

  2. \(X_n \stackrel{P}{\longrightarrow}X\), which is convergence in measure with respect to the probability measure \(P\), is usually written \(X_n \stackrel{\text{p}}{\longrightarrow}X\), and read as “\(X_n\) converges in probability to \(X\).”

  3. \(X_n \stackrel{L_2}{\longrightarrow}X\) is usually read as “\(X_n\) converges to \(X\) in mean square”.

• Scheffé’s theorem: Let \(\{f_n\}_{n \geq 1}\) and \(f\) be measurable functions on a measurable space \((\Omega,\mathcal{F},\mu)\) such that \(f_n \stackrel{\operatorname{a.e.}}{\longrightarrow}f\), \(\int f_n d \mu \to \int f d \mu\), and \(\int f d \mu < \infty\). Show that \(f_n\stackrel{L_1}{\rightarrow} f\).

  See Theorem 2.5.4 on page 64 of A&L or Theorem 16.12 on page 215 of Billingsley.

Uniform integrability

• Some results similar to those above can be obtained under a weaker condition called uniform integrability.

• Tails of integrable functions: Show that \(f\in L^1(\Omega,\mathcal{F},\mu)\) implies \(\lim_{t\to \infty} \int_{\{|f| > t\}}|f|d\mu = 0\).

  This result leads to the definition of uniform integrability.

• Uniform integrability of a family of functions: Let \(\{f_\alpha, \alpha \in I\}\) be an arbitrarily indexed family of measurable functions on a measure space \((\Omega,\mathcal{F},\mu)\). The family \(\{f_\alpha, \alpha \in I\}\) is uniformly integrable if \[ \lim_{t \to \infty} \sup_{\alpha \in I} \int_{\{|f_\alpha| > t\}}|f_\alpha| d\mu = 0. \]

  See Definition 2.5.8 on page 65 of A&L, the section “uniform integrability” beginning on page 216 of Billingsley, or Section 6.5.1 of Resnick beginning on page 182. The definition is represented next for a sequence of random variables.

• Uniform integrability of a sequence of random variables: Let \(\{X_n\}_{n\geq 1}\) be a sequence of random variables on a probability space \((\Omega,\mathcal{F},P)\). Then the sequence \(\{X_n\}_{n \geq 1}\) is uniformly integrable if for every \(\epsilon > 0\) there exists \(M_\epsilon > 0\) such that \[ \sup_{n \geq 1} \mathbb{E}[|X_n| 1(|X_n| > M_\epsilon)] < \epsilon. \]

Differentiation

• Absolute continuity and mutual singularity: Let \(\mu\) and \(\nu\) be measures on a measurable space \((\Omega,\mathcal{F})\).

  • If \(\mu(A) = 0\) implies \(\nu(A) = 0\) for all \(A \in \mathcal{F}\) then \(\nu\) is absolutely continuous (a.c.) with respect to \(\mu\) or dominated by \(\mu\). We write \(\mu \ll \nu\).
  • If there exists a set \(B \in \mathcal{F}\) such that \(\mu(B) = 0\) and \(\nu(B^c) = 0\) then \(\mu\) and \(\nu\) are mutually singular. We write \(\mu \perp \nu\). Here the relation is symmetric, i.e. \(\mu \perp \nu \iff \nu \perp \mu\). See Example 4.1.4 on page 114 of A&L.

  See Definitions 4.1.1 and 4.1.2 on pages 113 and 114 of A&L.

• Make a new measure by integrating a nonnegative function: Let \(f\) be a nonnegative measurable function on a measure space \((\Omega,\mathcal{F},\mu)\). Show that the set function defined as \[ \nu(A) \equiv \int_A f d\mu \] for all \(A\in \mathcal{F}\) is a measure on \(\mathcal{F}\).

  See Corollary 2.3.6 on page 53 of A&L. Note that \(\nu \ll \mu\), i.e. \(\nu\) is absolutely continuous with respect to \(\mu\).

• Radon-Nikodym Theorem: Let \((\Omega,\mathcal{F})\) be a measure space and let \(\mu\) and \(\nu\) be \(\sigma\)-finite measures on \(\mathcal{F}\) such that \(\nu \ll \mu\). Then there exists a nonnegative measurable function \(f\) such that \(\nu(A) = \int_A f d\mu\) for all \(A \in \mathcal{F}\). Moreover, for two such functions \(f\) and \(g\) it holds that \(f = g\) almost everywhere.

  See Theorem 32.2 of Billingsley. The function \(f\) is called the Radon-Nikodym derivative of \(\nu\) with respect to \(\mu\). We may also call \(f\) the density of \(\nu\) with respect to \(\mu\). Often let \(\mu\) be the Lebesgue measure and set \(f\) to be a probability density function. Or set \(\mu\) to be the counting measure and \(f\) to be a probability mass function.

• Relation between Riemann and Lebesgue integrals: Let \(f\) be a bounded measurable function on \(([a,b],\mathcal{B}([a,b]),\mu_L)\), where \(\mu_L\) is the Lebesgue measure. Then \(f\) is Riemann integrable on \([a,b]\) if and only if it is continuous almost everywhere (with respect to \(\mu_L\)). In this case the integral \(\int f d\mu_L\) is equal to the Riemann integral \(\int_a^b f(x)dx\).

  See Theorem 2.4.1 on page 61 of A&L. Compare to Theorem 5.6.1 on page 139 of Resnick and Section 17 of Billingsley beginning on page 221.

• Example: Consider the measure space \(([0,1],\mathcal{B}([0,1]),\mu_L)\), where \(\mu_L\) is the Lebesgue measure, and let \(f(\omega) = 6 \omega(1-\omega)\) for \(\omega \in [0,1]\). Now define \(P(A) = \int_A f d\mu_L\).

  • Give an expression for \(P((a,b))\) for any interval \((a,b) \subset [0,1]\).
  • Show that \(P\) is a probability measure.
  • Let \(X(\omega) = \omega\) be a random variable on the probablity space \(([0,1],\mathcal{B}([0,1]),P)\). Give \(\mathbb{E}X\).

• Example: Consider the measure space \((\{0,1,\dots,N\},\mathcal{P}(\{0,1,\dots,N\}),\mu)\), where \(\mu(A) = |A|\) for all \(A \subset \{0,1,\dots,N\}\) is the counting measure, and let \(p(\omega) = (1/2)^N {N\choose\omega}\) for \(\omega \in \{0,1,\dots,N\}\). Now define \(P(A) = \int_A p d\mu\) for each \(A \subset \{0,1,\dots,N\}\).

  • Give an expression for \(P(A)\) for any \(A \subset \{0,1,\dots,N\}\)
  • Show that \(P\) is a probability measure.
  • Let \(X(\omega) = \omega\) be a random variable on the probability space \((\{0,1,\dots,N\},\mathcal{P}(\{0,1,\dots,N\}),P)\). Give \(\mathbb{E}X\).