VI. Probability spaces and independence
$$
$$
Find these notes at gregorkb.github.io/probtheory/
Probability spaces, random variables, vectors
Recall the definitions of a probability space, random variable, and probability distribution.
Random vector of finite dimension: Given a probability space \((\Omega,\mathcal{F},P)\) a function \(X:\Omega \to \mathbb{R}^k\), \(k<\infty\) that is \(\langle \mathcal{F},\mathcal{B}(\mathbb{R}^k)\rangle\)-measurable is called a random vector of dimension \(k\). We may write \(X = (X_1,\dots,X_k)\).
Joint and marginal probability distributions: Let \(X = (X_1,\dots,X_k)\) be a random vector of dimension \(k\) on a probability space \((\Omega,\mathcal{F},P)\). Then:
- The probability measure \[ P_X(A) = P(\{\omega : X(\omega) \in A\}), \quad A \in \mathcal{B}(\mathbb{R}^k) \] is called the joint probability distribution of \(X = (X_1,\dots,X_k)\).
- For each \(j = 1,\dots,k\), the probability measure \[ P_{X_j}(A) = P(\{\omega : X_j(\omega) \in A\}), \quad A \in \mathcal{B}(\mathbb{R})\] is called the marginal distribution of \(X_j\).
Joint and marginal cumulative distribution functions: Let \(X = (X_1,\dots,X_k)\) be a random vector of dimension \(k\) on a probability space \((\Omega,\mathcal{F},P)\). Then:
- The function \[ \begin{align} F_X(x_1,\dots,x_k) &= P_X((-\infty,x_1] \times \dots \times (-\infty,x_k]) \\ &= P(\{\omega : X_1(\omega) \leq x_1, \dots, X_k(\omega) \leq x_k\}) \end{align} \] is called the joint cumulative distribution function of \(X\).
- For each \(j= 1,\dots,k\) the function \[ F_{X_j}(x_j) = P_{X_j}((-\infty,x_j]) = P(\{\omega : X_j(\omega) \leq x_j\}) \] is called the marginal cumulative distribution function of \(X_j\).
Note that \(F_{X_1}(x_1) = F_X(x_1,\infty,\dots,\infty)\) and so on.
Expected value: Let \(X\) be a random variable on a probability space \((\Omega,\mathcal{F},P)\). Then the expected value of \(X\) is defined as \[ \mathbb{E}X \equiv \int X dP, \] provided at least one of the integrals \(\int X^+dP\) and \(\int X^{-}dP\) is finite. We often call \(\mathbb{E}X\) the mean of \(X\) and denote it as \(\mu_X = \mathbb{E}X\).
Change of variable formula: Let \(X\) be a random variable on a probability space \((\Omega,\mathcal{F},P)\) and let \(Y = h(X)\), for some Borel measurable function \(h: \mathbb{R}\to \mathbb{R}\). If \(\int |Y|dP < \infty\) then \[ \int Y dP = \int h(x) dP_X(x) = \int y dP_Y(y). \]
Moments of a random variable: Given a random variable \(X\), the quantity \(\mu_k = \mathbb{E}X^k\) is called the \(k\)th moment of \(X\) for \(k=1,2,\dots\)
Variance of a random variable: The variance of a random variable \(X\) is defined as \(\operatorname{Var}X = \mathbb{E}X^2 - (\mathbb{E}X)^2 = \mu_2 - \mu_1^2\).
Covariance between two random variables: Let \(X\) and \(Y\) be random variables on \((\Omega,\mathcal{F},P)\). Then the covariance between \(X\) and \(Y\) is defined as \(\operatorname{Cov}(X,Y) = \mathbb{E}XY - \mathbb{E}X \mathbb{E}Y\).
Moment generating function of a random variable: The moment generating function of a random variable \(X\) is defined as \[ M_X(t) = \mathbb{E}e^{tX}, \quad t \in \mathbb{R}. \] Note that the expectation may not be finite.
Relation between the moment generating function and moments: Let \(X\) be a random variable with moment generating function \(M_X(t)\) such that \(M_X(t) < \infty\) for all \(|t| < \varepsilon\) for some \(\varepsilon > 0\).
- Show that \(E|X|^k < \infty\) for all \(k=1,2,\dots\)
- Show that \(M_X(t) = \sum_{k=0}^\infty t^k \frac{\mu_k}{k!}\) for all \(|t| < \varepsilon\).
- Show that \(M_X(t)\) is infinitely differentiable on \((-\varepsilon,\varepsilon)\) with \[ M_X^{(r)}(t) = \sum_{k=0}^\infty \frac{t^k}{k!}\mu_{k + r} \] for \(|t| < \varepsilon\), whereby \(M^{(r)}_X(0) = \mu_r\).
See Proposition 6.2.3 on page 195 of A&L. For differentiating a power series see Section 8-2 starting on page 219 of Kirkwood.
Inequalities: Let \(X\) and \(Y\) be random variables on \((\Omega,\mathcal{F},P)\).
Markov’s: Show that \[ P_X(|X| > \epsilon) \leq \frac{\mathbb{E}|X|}{\epsilon} \] for each \(\epsilon > 0\).
Chebychev’s: Show that \[ P_X(|X - \mu_X| > k \sigma_X) \leq \frac{1}{k^2} \] for each \(k >0\), provided \(\sigma_X\) is finite.
Jensen’s: Let \(\phi\) be a convex function on \((a,b)\) and suppose \(P_X(a < X < b) = 1\). Show that \(\phi(\mathbb{E}X) \leq \mathbb{E}\phi(X)\).
See page 188 of Resnick.
Hölder’s: Show that \[ \mathbb{E}|XY| \leq (\mathbb{E}|X|^p)^{1/p} (\mathbb{E}|Y|^{q})^{1/q} \] for \(p,q \in (1,\infty)\) with \(1/p + 1/q = 1\), provided the expectations on the right hand side are finite.
Use inequality \(ab \leq a^p/p + b^q/q\) for any \(a,b>0\) and any \(p,q\geq 1\) such that \(1/p + 1/q = 1\). See page 80 of Billingsley.
Cauchy-Schwarz: Show that \[ \operatorname{Cov}(X,Y) \leq \sqrt{\operatorname{Var}X} \sqrt{\operatorname{Var}Y}, \] provided the variances on the right hand side are finite.
Minkowski’s: Show that \[ (\mathbb{E}|X+Y|^p)^{1/p} \leq (\mathbb{E}|X|^p)^{1/p} + (\mathbb{E}|Y|^p)^{1/p} \] provided the expectations on the right hand side are finite.
For a definition of joint moments and the joint moment generating function for the components of a random vector, see Definition 6.2.12 on page 198 of A&L.
Independence
“Events” in a probability space: Recall that in a probability space \((\Omega, \mathcal{F}, P)\) we tend to call subsets of \(\mathcal{F}\) events and interpret \(P(A)\) for \(A \in \mathcal{F}\) as the probability of the event \(A\).
Independence among finitely many events: Let \((\Omega,\mathcal{F},P)\) be a probability space and let \(A_1,\dots,A_k \in \mathcal{F}\), \(1 \leq k < \infty\). Then
- \(A_1,\dots,A_k\) are independent with respect to \(P\) if \[ P(\bigcap_{j=1}^l A_{i_j}) = \prod_{j=1}^l P(A_{i_j}) \] for each subset of indices \(\{i_1,\dots,i_l\} \subset \{1,\dots,n\}\), \(2 \leq l \leq k\).
- \(A_1,\dots,A_k\) are pairwise independent with respect to \(P\) if \[ P(A_i \cap A_j) = P(A_i) P(A_j) \] for all \(1 \leq i < j \leq k\).
Independence of arbitrarily indexed events: For a nonempty set \(I\), a collection of events \(\{A_\alpha\}_{\alpha \in I}\) in a probability space \((\Omega,\mathcal{F},P)\) is independent with respect to \(P\) if for every finite set of indices \(\{\alpha_1,\dots,\alpha_k\} \subset I\) the events \(A_{\alpha_1},\dots,A_{\alpha_k}\) are independent with respect to \(P\).
Independence of finitely many collections of events: A set of collections \(\mathcal{C}_1,\dots,\mathcal{C}_k\) of events on a probability space \((\Omega,\mathcal{F},P)\) is independent with respect to \(P\) if every collection of events \(A_1,\dots,A_k\) with \(A_j \in \mathcal{C}_j\) for \(j = 1,\dots,k\) is independent with respect to \(P\).
Independence of arbitrarily indexed collections of events: For nonempty set \(I\), an arbitrarily indexed family \(\{\mathcal{C}_\alpha\}_{\alpha \in I}\) of collections of events on a probablity space \((\Omega,\mathcal{F},P)\) is independent with respect to \(P\) if every collection of events \(\{A_\alpha\}_{\alpha \in I}\) with \(A_\alpha \in \mathcal{C}_\alpha\) for \(\alpha \in I\) is independent with respect to \(P\).
Note that we need language for a set of sets and for a set of sets of sets. In these notes we have most of the time called a set of sets a collection of sets or a class of sets. For a set of sets of sets, we have used, in the above, a family of collections of sets. This language is arbitrary. Billingsley (on page 55, for example), introduces a “collection of classes of sets”.
Independence of finitely many random variables: A finite collection \(X_1,\dots,X_k\) of random variables on a probability space \((\Omega,\mathcal{F},P)\) is independent with respect to \(P\) if the collections of events \(\sigma \langle X_1\rangle,\dots,\sigma \langle X_k \rangle\) are independent with respect to \(P\).
Independence of an arbitrarily indexed family of random variables: For a nonempty set \(I\), a family \(\{X_\alpha\}_{\alpha \in I}\) of random variables on a probability space \((\Omega,\mathcal{F},P)\) is independent with respect to \(P\) if the family of \(\sigma\)-algebras \(\{\sigma \langle X_\alpha\rangle\}_{\alpha \in I}\) is independent with respect to \(P\).
Independence of \(\sigma\)-algebras generated by independent \(\pi\)-systems: Let \((\Omega,\mathcal{F},P)\) be a probability space and let \(\mathcal{C}_1,\dots,\mathcal{C}_k \subset \mathcal{F}\) be \(\pi\)-systems which are independent with respect to \(P\). Show that \(\sigma \langle \mathcal{C}_1\rangle,\dots, \sigma \langle \mathcal{C}_k \rangle\) are independent with respect to \(P\).
See Proposition 7.1.1 on page 220 of A&L or Theorem 4.2 on page 55 of Billingsley. This can be extended to the case of an arbitrarily indexed family of \(\pi\)-systems: See Corollary 1 on page 56 of Billingsley. Just proof for \(k=2\).
Factorization rule: Show that for a nonempty set \(I\) the family \(\{X_\alpha\}_{\alpha \in I}\) of random variables is independent if and only if for every finite subset of indices \(\{\alpha_1,\dots,\alpha_k\} \subset I\) the joint CDF \(F_{\alpha_1,\dots,\alpha_k}\) of \((X_{\alpha_1},\dots,X_{\alpha_k})\) is equal to the product of the marginal CDFs, that is \[ F_{\alpha_1,\dots,\alpha_k}(x_1,\dots,x_k) = \prod_{j=1}^kF_{\alpha_j}(x_j) \] for all \(x_1,\dots,x_k \in \mathbb{R}\).
Follow Corollary 7.1.2 of A&L. Prove by showing \(\{X^{-1}((-\infty,x]), x\in \mathbb{R}\}\) is a \(\pi\)-class and invoking the previous result.
Expectation of a product: Let \(X_1,\dots,X_k\), \(2 \leq k < \infty\), be random variables on a probability space \((\Omega,\mathcal{F},P)\).
Show that \(X_1,\dots,X_k\) are independent if and only if \[ \mathbb{E}\prod_{j=1}^k h_j(X_j) = \prod_{j=1}^k \mathbb{E}h_j(X_j) \] for all bounded Borel measurable functions \(h_j:\mathbb{R}\to \mathbb{R}\), \(j=1,\dots,k\).
Show that if \(X_1\) and \(X_2\) are independent and \(\mathbb{E}|X_1| <\infty\) and \(\mathbb{E}|X_2| < \infty\) then \(\mathbb{E}|X_1 X_2| < \infty\) and \[ \mathbb{E}X_1 X_2 = \mathbb{E}X_1 \mathbb{E}X_2. \]
Show that if \(X_1\) and \(X_2\) are independent and \(A \in \sigma \langle X_1\rangle\) then \[ \int_A X_1 X_2 dP = \int_A X_1 dP \cdot \mathbb{E}X_2. \]
See Prop 7.1.3 on page 221 of A&L as well as (21.19) on page 277 of Billingsley.
Kolmogorov’s inequality for partial sums: Let \(X_1,\dots,X_n\) be random variables on a probability space \((\Omega,\mathcal{F},P)\) have zero mean and finite variance and define the partial sums \(S_k = \sum_{i=1}^k X_i\) for \(k=1,\dots,n\). Show that for each \(\varepsilon > 0\) we have \[ P(\max_{1 \leq k \leq n} |S_k| \geq \varepsilon) \leq \frac{1}{\varepsilon^2}\operatorname{Var}S_n. \]
See Theorem 22.4 on page 287 of Billingsley.
Borel-Cantelli, Kolmogorov zero-one laws
Borel-Cantelli Lemmas: Let \((\Omega,\mathcal{F},P)\) be a probability space and let \(\{A_n\}_{n\geq 1} \subset \mathcal{F}\) be sequence of events.
- Show that \(\sum_{n\geq 1} P(A_n) < \infty\) implies \(P(\limsup_{n\to \infty} A_n) = 0\).
- Show that if \(\{A_n\}_{n\geq 1}\) are pairwise independent, then \(\sum_{n\geq 1} P(A_n) = \infty\) implies \(P(\limsup_{n\to \infty} A_n) = 1\).
See Theorem 7.2.2 on page 223 of A&L. This is known as a “zero-one” law, because under pairwise independence of \(\{A_n\}_{n\geq 1}\) it states that \(P(\limsup_{n\to \infty} A_n)\) is equal to zero or one according to whether \(\sum_{n\geq 1} P(A_n)\) is finite or infinite. Compare to Theorems 4.3 and 4.4 on pages 59 and 60 of Billingsley (the latter makes the stronger assumption of independence instead of assuming only pairwise independence).
A corollary to Borel-Cantelli: Let \(\{X_n\}_{n \geq 1}\) be a sequence of random variables on a probability space \((\Omega,\mathcal{F},P)\).
Show that \(\sum_{n\geq 1}P(|X_n| > \epsilon) < \infty\) for every \(\epsilon > 0\) implies \(P(\lim_{n\to \infty} X_n = 0) = 1\).
Suppose \(\{X_n\}_{n\geq 1}\) are pairwise independent. Show that \(P(\lim_{n\to \infty} X_n = 0) = 1\) implies \(\sum_{n\geq 1}P(|X_n| > \epsilon) < \infty\) for every \(\epsilon > 0\).
See Proposition 7.2.3 on page 224 of A&L.
Tail \(\sigma\)-algebra of a collection of events: Given a probability space \((\Omega,\mathcal{F},P)\) the tail \(\sigma\)-algebra of a collection of events \(\{A_n\}_{n \geq 1} \subset \mathcal{F}\) is defined as \[ \mathcal{T}\equiv \bigcap_{n \geq 1} \sigma \langle \{A_k: k \geq n\}\rangle. \] Events \(A \in \mathcal{T}\) are called tail events.
Limit supremum and limit infimum as tail events: Show that \(\limsup_{n \to \infty} A_n\) and \(\liminf_{n \to \infty}A_n\) belong to \(\mathcal{T}\).
Kolmogorov 0-1 law for sequences of independent events: Let \(\{A_n\}_{n \geq 1}\) be a sequence of independent events in the probability space \((\Omega,\mathcal{F},P)\). Show that \[ A \in \mathcal{T}\equiv \bigcap_{n\geq 1} \sigma \langle \{A_k: k \geq n\} \rangle \] implies \(P(A)\in \{0,1\}\).
Kolmogorov 0-1 law for sequences of independent random variables: Let \(\{X_n\}_{n \geq 1}\) be a sequence of independent random variables on a probability space \((\Omega,\mathcal{F},P)\). Show that \[ A \in \mathcal{T}\equiv \bigcap_{n\geq 1} \sigma \langle \{X_j : j \geq n\} \rangle \] implies \(P(A)\in \{0,1\}\). Here \(\mathcal{T}\) is called the tail \(\sigma\)-algebra of \(\{X_n\}_{n \geq 1}\). In the above \(\sigma \langle \{X_j: j \geq n\}\rangle = \sigma \langle \{X_j^{-1}(A): A \in \mathcal{B}(\mathbb{R}), j \geq n\}\rangle\) for each \(n\geq 1\).
Extended real-valued random variable: Given a probability space \((\Omega,\mathcal{F},P)\), a function \(X:\Omega \to \bar{\mathbb{R}}\) which is \(\langle \mathcal{F},\mathcal{B}(\bar{\mathbb{R}})\rangle\)-measurable is called an extended real-valued random variable. Here, \(\bar{\mathbb{R}} \equiv [-\infty,\infty]\) and \(\mathcal{B}(\bar{\mathbb{R}}) \equiv \sigma\langle\mathcal{B}(\mathbb{R}) \cup\{-\infty\} \cup \{\infty\}\rangle\). See Definition 7.2.3 on page 226 of A&L.
Degeneracy of tail random variables: Let \(\{X_n\}_{n \geq 1}\) be a sequence of independent random variables having tail \(\sigma\)-algebra \(\mathcal{T}\) and let \(X\) be a \(\langle \mathcal{T},\mathcal{B}(\bar{\mathbb{R}}) \rangle\)-measurable random variable. Show that there exists \(c \in \bar{\mathbb{R}}\) such that \(P(X = c) = 1\).
A random variable equal to a single value with probability one is called degenerate. Note that since \(\liminf_{n\to \infty} X_n\) and \(\limsup_{n \to \infty} X_n\) are tail random variables they are degenerate. See Corollary 7.2.5 of A&L. Cf. Problem 22.1 on page 294 of Billingsley.