13  The bootstrap

The bootstrap is a method for estimating sampling distributions.

We usually want to estimate sampling distributions of pivotal quantities—that is, quantities of which the limiting distribution (as the sample size grows) is known.

Even if one knows the asymptotic distribution of a pivotal quantity, one may need a very large sample size before one can make use of it. For example, the central limit theorem together with Slutzky’s theorem gives \[ \sqrt{n}(\bar X_n - \mu)/S_n \overset{\text{d}}{\longrightarrow}\mathcal{N}(0,1) \] as \(n \to \infty\) under mild conditions, where \(\bar X_n\) is the sample mean and \(S_n^2\) is the sample variance. This result tells us that the interval with endpoints \(\bar X_n \pm z_{\alpha/2} S_n/\sqrt{n}\)1 will contain \(\mu\) with probability tending to \(1-\alpha\) as \(n \to \infty\). For a given finite sample size, however, the coverage probability of this interval may be drastically more or less than \(1 - \alpha\), depending on the distribution from which the sample is drawn.

Enter the bootstrap.

The bootstrap can be used to estimate, say, the upper and lower \(\alpha/2\) quantiles of the sampling distribution of \(\sqrt{n}(\bar X_n - \mu)/S_n\), which can then be used to construct a confidence interval with coverage probability tending to \(1-\alpha\) as the sample size grows. It can be shown (under some settings) that such a bootstrap interval will perform better (achieve closer-to-nominal coverage) than the interval based on the asymptotic distribution. The ability of the bootstrap to provide a better approximation to a sampling distribution then does the asymptotic limiting distribution is called the “second-order correctness” of the bootstrap.

Apart from possessing the much-lauded second-order correctness property, the bootstrap may in some situations offer the most—or the only—practical way to make inferences, for example when the statistic or pivotal quantity has an asymptotic distribution which is intractable (e.g. such that one cannot compute its quantiles), so that using the asymptotic distribution to conduct inference is not even feasible.

All of this and more we will discuss, beginning with the simple case of the bootstrap for the sample mean.

  1. Note that the \(t\)-based interval \(\bar X_n \pm t_{n-1,\alpha/2} S_n/\sqrt{n}\) is tailored to the case in which the sample is drawn from a Normal distribution, in which it has coverage probability exactly equal to \(1-\alpha\). If the sample is not drawn from a Normal distribution, there no principled reason to use the \(t\)-based interval.↩︎