II. Measures and extensions

Find these notes at gregorkb.github.io/probtheory/

Measures

Measure on an algebra: Let $\mathcal{F}$ be an algebra (field) on $\Omega$. A set function $\mu$ on $\mathcal{F}$ is called a measure if
1. $\mu(A) \in [0,\infty]$ for all $A \in \mathcal{F}$ (Non-negativity).
2. $\mu(\emptyset) = 0$ (Nullity of empty set measure).
3. $A_1,A_2,\dots \in \mathcal{F}$ with $A_i \cap A_j = \emptyset$ for all $i \neq j$ and $\bigcup_{i=1}^\infty A_i \in \mathcal{F}$ implies $\mu(\bigcup_{i = 1}^\infty A_i) = \sum_{i = 1}^\infty \mu(A_i)$ (Countable additivity or $\sigma$-additivity).
Show (see Proposition 1.2.1 of A&L) that we obtain an equivalent definition if we replace iii. above with
1. $A_1,A_2\in \mathcal{F}$ with $A_1\cap A_2 = \emptyset$ implies $\mu(A_1 \cup A_2) = \mu(A_1) + \mu(A_2)$ (Finite additivity).
2. $\{A_n\}_{n\geq 1}\subset \mathcal{F}$ with $A_n \subset A_{n+1}$ for all $n\geq 1$ and $A \equiv \bigcup_{n \geq 1}A_n \in \mathcal{F}$ implies $\lim_{n\to \infty}\mu(A_n) = \mu(A)$ (Monotone continuity from below).
Since every $\sigma$-algebra is an algebra, the above suffices to define measures on $\sigma$-algebras.
Finite, infinite, $\sigma$-finite, and probability measures: Given a sample space $\Omega$, a $\sigma$-algebra $\mathcal{F}$ on $\Omega$, and a measure $\mu$ on $\mathcal{F}$:
- If $\mu(\Omega) < \infty$ then $\mu$ is finite.
- If $\mu(\Omega) = \infty$ then $\mu$ is infinite.
- If there exists a collection of sets $A_1,A_2,\dots \in \mathcal{F}$ such that $\bigcup_{i=1}^\infty A_i = \Omega$ and $\mu(A_i) < \infty$ for all $i=1,2,\dots$ then $\mu$ is $\sigma$-finite.
- If $\mathcal{A}$ is a subclass of $\mathcal{F}$ then $\mu$ is $\sigma$-finite on $\mathcal{A}$ if there exists a collection of sets $A_1,A_2,\dots\in \mathcal{A}$ such that $\bigcup_{i=1}^\infty A_i = \Omega$ and $\mu(A_i) < \infty$ for all $i=1,2,\dots$
- If $\mu(\Omega)=1$ then $\mu$ is a probability measure.
See page 160 of Billingsley.
Two examples of measures:
- Counting measure: Define on $\mathcal{F}= \mathcal{P}(\Omega)$ the set function $\mu(A) = |A|$ for all $A \in \mathcal{F}$, where $|A|$ gives the number of elements in $A$. This is the counting measure. Show that $\mu$ is a measure on $\mathcal{F}$. Give sample spaces $\Omega$ such that $\mu$ is a finite measure, an infinite measure, and an infinite but a $\sigma$-finite measure.
- Discrete probability measure: Let $\Omega = \{\omega_1,\omega_2,\dots\}$ and, given a set of numbers $p_1,p_2,\dots \in [0,1]$ such that $\sum_{i = 1}^n p_i = 1$, define on $\mathcal{F}= \mathcal{P}(\Omega)$ the set function $\mu(A) = \sum_{i=1}^\infty p_i 1_{A}(\omega_i)$. Show that $\mu$ is a measure on $\mathcal{F}$.
Monotonicity, finite sub-additivity, and the inclusion-exclusion formula: Let $\mathcal{F}$ be an algebra on $\Omega$ and let $\mu$ be a measure on $\mathcal{F}$. Further, let $A,B, A_1,\dots,A_n \in \mathcal{F}$, $1 \leq n < \infty$.
1. Show that $A \subset B$ implies $\mu(A) \leq \mu(B)$ (Monotonicity).
2. Show that $\mu(\bigcup_{i=1}^n A_i) \leq \sum_{i=1}^n \mu(A_i)$ (Finite subadditivity).
3. Show by induction that \[ \begin{align} \mu(\bigcup_{i=1}^nA_i) = \sum_{i=1}^n\mu(A_i) &- \sum_{1\leq i < j \leq n}\mu(A_i \cap A_j) + \sum_{1 \leq i < j < k \leq n}\mu(A_i \cap A_j \cap A_k) \\ &- \dots +(-1)^{n-1}\mu(A_1 \cap \dots \cap A_n) \end{align} \] (inclusion-exclusion formula).
See Proposition 1.2.2 of A&L.
Monotone continuity from above (MCFA) and countable sub-additivity: Let $\mathcal{F}$ be an algebra on $\Omega$ and $\mu$ be a measure on $\mathcal{F}$.
1. Show that if $\{A_n\}_{n \geq 1} \subset \mathcal{F}$ with $A_{n+1}\subset A_n$ for all $n \geq 1$ and $A \equiv \bigcap_{n \geq 1} A_n \in \mathcal{F}$ then $\lim_{n\to \infty}\mu(A_n) = \mu(A)$, provided $\mu(A_{n_0}) < \infty$ for some $n_0 < \infty$ (monotone continuity from above).
2. Show that for any collection of sets $A_1,A_2,\dots \in \mathcal{F}$ such that $\bigcup_{i= 1}^\infty A_i \in \mathcal{F}$ we have $\mu(\bigcup_{i = 1}^\infty A_i) \leq \sum_{i =1}^\infty \mu(A_i)$ (countable subadditivity).
See Proposition 1.2.3 on page 18 of A&L.
Measurable space: A pair $(\Omega,\mathcal{F})$, where $\Omega$ is a non-empty set and $\mathcal{F}$ is a $\sigma$-algebra on $\Omega$ is called a measurable space.
Measure space: A triple $(\Omega,\mathcal{F},\mu)$, where $\Omega$ is a non-empty set, $\mathcal{F}$ is a $\sigma$-algebra on $\Omega$, and $\mu$ is a measure on $\mathcal{F}$ is called a measure space.
Probability space: A triple $(\Omega,\mathcal{F},P)$, where $\Omega$ is a non-empty set, $\mathcal{F}$ is a $\sigma$-algebra on $\Omega$, and $P$ is a probability measure on $\mathcal{F}$ is called a probability space.

Uniqueness of measures: Let $\mu_1$ and $\mu_2$ be finite measures on $\sigma \langle \mathcal{C}\rangle$, where $\mathcal{C}$ is a $\pi$-system on $\Omega$. Show that $\mu_1(\Omega) = \mu_2(\Omega)$ and $\mu_1(A) = \mu_2(A)$ for all $A \in \mathcal{C}$ implies $\mu_1(A) = \mu_2(A)$ for all $A \in \sigma \langle \mathcal{C}\rangle$.

If two measures agree on a $\pi$-system, they will agree on the $\sigma$-algebra generated by the $\pi$-system. See Theorem 1.2.4 on page 19 of A&L. For the case of $\sigma$-finite measures see Theorem 10.3 on page 163 of Billingsley.
Measure on a semi-algebra: Let $\mathcal{C}$ be a semi-algebra on $\Omega$. A set function $\mu$ on $\mathcal{C}$ is called a measure if
1. $\mu(A) \in [0,\infty]$ for all $A \in \mathcal{C}$.
2. $\mu(\emptyset) = 0$ (Nullity of empty set measure).
3. $A_1,A_2,\dots \in \mathcal{C}$ with $A_i \cap A_j = \emptyset$ for all $i \neq j$ and $\bigcup_{i=1}^\infty A_i \in \mathcal{F}$ implies $\mu(\bigcup_{i = 1}^\infty A_i) = \sum_{i = 1}^\infty \mu(A_i)$ (Countable additivity).
This is exactly how we define a measure on an algebra, but a new definition is needed here, since not every semi-algebra is an algebra.
Outer measure on the power set: Given a sample space $\Omega$, a set function $\mu^*$ on $\mathcal{P}(\Omega)$ is called an outer measure if it has the properties:
1. $\mu^*(A) \in [0,\infty]$ for all $A \subset \Omega$.
2. $\mu^*(\emptyset) = 0$ (Nullity of empty set probability).
3. $A \subset B$ implies $\mu^*(A) \leq \mu^*(B)$ (Monotonicity).
4. $\mu^*(\bigcup_{i=1}^\infty A_i) \leq \sum_{i=1}^\infty \mu^*(A_i)$ for any collection of sets $A_1,A_2,\dots \subset \Omega$ (Countable sub-additivity).
See page 165 of Billingsley.

Extensions of measures

Extension of a measure on a semi-algebra to a measure on the algebra generated by the semi-algebra: Let $\mathcal{C}$ be a semi-algebra of subsets of $\Omega$ and let $\mu$ be a measure on $\mathcal{C}$. Moreover, let $\mathcal{A}\equiv \mathcal{A}\langle \mathcal{C}\rangle$ be the smallest algebra containing $\mathcal{C}$ and, for each $A \in \mathcal{A}$, set \[ \bar \mu(A) \equiv \sum_{i=1}^k \mu(B_i) \] if $A$ has the representation $A = \bigcup_{i=1}^k B_i$, where $B_1,\dots,B_k$ is a finite collection of disjoint sets in $\mathcal{C}$.
1. Show that $\bar \mu$ does not depend on the collection of sets $B_1,\dots,B_k$ used in the representation $A = \bigcup_{i=1}^kB_i$.
2. Show that $\bar \mu$ is a measure on $\mathcal{A}$.
See Proposition 1.3.1 on page 20 of A&L or Theorem 2.4.1 of page 46 of Resnick.
Outer measure induced by a measure on a semi-algebra: Let $\mathcal{C}$ be a semi-algebra on $\Omega$ and let $\mu$ be a measure on $\mathcal{C}$. Then, for each $A \subset \Omega$, set \[ \mu^*(A) \equiv \inf\Big\{\sum_{i=1}^\infty \mu(B_i) : B_1,B_2,\dots \in \mathcal{C}\text{ and } A \subset \bigcup_{i=1}^\infty B_i \Big\}. \] Show that $\mu^*$ is an outer measure on $\Omega$.

The set function $\mu^*$ is called the outer measure on $\Omega$ induced by $\mu$.

The outer measure assigns to $A \subset \Omega$ the smallest sum of measures of sets in a collection of sets in the semi-algebra covering $A$ (a collection of sets covers a set $A$ if their union contains $A$). See page 38 of Billingsley.
Show that $\mu^*(A) = \bar \mu(A)$ for all $A \in \mathcal{A}\langle \mathcal{C}\rangle$.
Measurability of a set with respect to an outer measure: A set $A\subset \Omega$ is said to be $\mu^*$-measurable if \[ \mu^*(E) = \mu^*(E \cap A) + \mu^*(E \cap A^c) \quad \text{ for all } E \subset \Omega. \]
Caratheodory Extension Theorem Part 1: Let $\mu^*$ be an outer measure on a sample space $\Omega$ and set $\mathcal{M}_{\mu^*} \equiv \{A: \text{$A$ is $\mu^*$-measurable}\}$.
1. Show that $\mathcal{M}_{\mu^*}$ is a $\sigma$-algebra.
2. Show that $\mu^*$ with domain restricted to $\mathcal{M}_{\mu^*}$ is a measure on $\mathcal{M}_{\mu^*}$.
Showing the above establishes that $(\Omega,\mathcal{M}_{\mu^*},\mu^*)$ is a measure space. See Theorem 1.3.2 on page 22 of A&L. For now we do not discuss completeness of measure spaces.
Caratheodory Extension Theorem Part 2: Let $\mathcal{C}$ be a semi-algebra on $\Omega$, $\mu$ a measure on $\mathcal{C}$, and $\mu^*$ the outer measure on $\Omega$ induced by $\mu$.
1. Show that $\mathcal{C}\subset \mathcal{M}_{\mu^*}$.
2. Show that $\mu^*(A) = \mu(A)$ for all $A \in \mathcal{C}$.
3. If another measure $\mu^{**}$ on $\sigma\langle \mathcal{C}\rangle$ satisfies $\mu^{**}(A) = \mu(A)$ for all $A \in \mathcal{C}$, then $\mu^{**}(A) = \mu^*(A)$ for all $A \in \sigma\langle \mathcal{C}\rangle$, provided $\mu$ is $\sigma$-finite on $\mathcal{C}$.
In consequence $(\Omega, \sigma \langle \mathcal{C}\rangle, \mu^*)$, with $\mu^*$ restricted to $\sigma \langle \mathcal{C}\rangle$, is a measure space, where $\mu^*$ is the unique extension of $\mu$ to $\sigma \langle \mathcal{C}\rangle$. Compare to Theorems 1.3.3 and 1.3.6 on pages 24 and 29 of A&L or Theorem 2.4.3 on page 48 of Resnick.

Some authors begin with a semi-ring instead of a semi-algebra. See Theorem 11.3 on page 166 of Billingsley or the nice videos of Adam B. Kashlak at www.youtube.com/@cachelackmathstatslectures7001.
Lebesgue-Stieltjes measure on a semi-algebra of intervals: Given a non-decreasing function $F: \mathbb{R}\to \mathbb{R}$, set $F(x+) \equiv \lim_{y \downarrow x}F(y)$ and $F(x-)\equiv \lim_{y \uparrow x}F(y)$ for all $x \in \mathbb{R}$ as well as $F(\infty) \equiv \lim_{x \uparrow \infty}(x)$ and $F(-\infty)\equiv \lim_{x \downarrow \infty}F(x)$. Then, for sets in the collection \[ \mathcal{C}= \{(a,b]:-\infty \leq a \leq b < \infty\} \cup \{(a,\infty): - \infty \leq a < \infty\} \] define the set function $\mu_F((a,b]) = F(b+) - F(a+)$ and $\mu_F((a,\infty)) = F(\infty) - F(a+)$.

Show that $\mu_F$ is a measure on $\mathcal{C}$.
Lebesgue-Stieltjes measure space: For a non-decreasing function $F:\mathbb{R}\to \mathbb{R}$, the measure space $(\mathbb{R}, \mathcal{M}_{\mu^*_F},\mu_F^*)$ is called the Lebesgue-Stieltjes measure space and $\mu_F^*$ is called the Lebesgue-Stieltjes measure generated by the function $F$.

Since $\sigma\langle\mathcal{C}\rangle = \mathcal{B}(\mathbb{R})$, the triple $(\mathbb{R}, \mathcal{B}(\mathbb{R}),\mu_F^*)$, with $\mu_F^*$ restricted to Borel sets in $\mathbb{R}$, is also a measure space.
Show that $\mu_F^*(\{x\}) = F(x+) - F(x-)$ for any $x \in \mathbb{R}$.
Lebesgue measure: The Lebesgue measure is a special case of the Lebesgue-Stieltjes measure when $F(x) = x$, $x \in \mathbb{R}$. Denote the Lebesgue measure by $\mu_L$. Then:
- $\mu_L(\{x\}) = 0$ for any $x \in \mathbb{R}$.
- $\mu_L(A) = 0$ for any countable set $A \subset \mathbb{R}$.
- For any $A \subset \mathbb{R}$ we have \[ \mu_L(A) = \mu_L(A + c) \quad \text{ and } \quad \mu_L(-A) = \mu_L(A) \] for all $c \in \mathbb{R}$, where $A + c \equiv \{x : x - c \in A\}$ and $-A = \{x: -x \in A\}$.
One can define Lebesgue-Stieltjes measures and Lebesgue measures in higher-dimensional Euclidean space.

Measures

Extensions of measures

Practice