11 Pinsker’s Theorem for Sobolev ellipsoids
$$
$$
We will continue in the infinite-dimensional Normal means model
\[ Z_j = \theta_j + \sigma \xi_j, \quad j = 1,2,\dots, \] where \(\boldsymbol{\theta}= (\theta_1,\theta_2,\dots)\) is unknown, \(\xi_1,\xi_2,\dots,\) are independent \(\mathcal{N}(0,1)\) random variables, and \(\sigma > 0\). In this section, we wish to identify the linear minimax risk for estimating \(\boldsymbol{\theta}\) in a Sobolev ellipsoid.
In the previous chapter we considered the same infinite-dimensional Normal means model and presented the linear minimax rate for estimating \(\boldsymbol{\theta}\) on a general ellipsoid. Now we will apply the result, which was Theorem 10.1, to a Sobolev ellipsoid, which, we recall, is defined as \[ \Theta_{\text{Sob}}(\beta,L) = \Big\{(\theta_1,\theta_2,\dots) \in \mathbb{R}: \quad \sum_{j=1}^\infty \theta^2_j < \infty \text{ and } \sum_{j=1}^\infty a_j^2 \theta_j^2 \leq L^2/\pi^{2\beta}\Big\}, \] with \[ a_j = \left\{\begin{array}{ll} j^\beta,& j \text{ even}\\ (j-1)^\beta,& j \text{ odd}. \end{array}\right. \tag{11.1}\]
To apply Theorem 10.1 to estimation on the Sobolev ellipsoid one first finds the solution to \[ \eta^{-1}\sigma^2\sum_{i=1}^\infty a_j (1 - \eta a_j)_+ = L^2/\pi^{2\beta} \] (cf. Equation 10.1) over \(\eta>0\) when \(a_1,a_2,\dots\) are from Equation 11.1. One can show (See page 145 of Tsybakov (2008)) that the solution satisfies \[ \eta = \Big(\frac{\beta\pi^{2\beta}}{L(\beta+1)(2\beta + 1)}\Big)^{\frac{\beta}{2\beta + 1}}\sigma^{\frac{2\beta}{2\beta + 1}}(1 + o(1)) \quad \text{ as } \sigma \to 0. \tag{11.2}\]
Secondly, after defining the Pinsker weights \(\ell_j = (1 - \eta a_j)_+\) for \(j=1,2,\dots\) under the above value of \(\eta\), one finds (See page 145 of Tsybakov (2008)) \[ \sigma^2 \sum_{j=1}\ell_j = C\sigma^{\frac{4\beta}{2\beta + 1}}(1 + o(1))\quad \text{ as } \sigma \to 0, \] (cf. Equation 10.2) where \[ C = L^{\frac{2}{2\beta + 1}}\Big(\frac{\beta}{\pi(\beta+1)}\Big)^{\frac{2\beta}{2\beta+1}}(2\beta + 1)^{\frac{1}{2\beta + 1}}. \tag{11.3}\] Putting the above results together, we can state the following result:
Theorem 11.1 (Linear minimax risk on a Sobolev ellipsoid) Under the infinite dimensional Normal means model we have \[ M_{\text{Lin}}(\Theta_{\text{Sob}}(\beta,L)) = \sup_{\boldsymbol{\theta}\in \Theta_{\text{Sob}}(\beta,L)}R(\hat{\boldsymbol{\theta}}_\boldsymbol{\ell},\boldsymbol{\theta}) = C \sigma^{\frac{4\beta}{2\beta + 1}}(1 + o(1)) \tag{11.4}\] as \(\sigma \to 0\), where \(\boldsymbol{\ell}= (\ell_1,\ell_2,\dots)\) is defined with the Pinsker weights \(\ell_j = (1 - \eta a_j)_+\), with \(\eta\) as in Equation 11.2 and \(a_1,a_2,\dots\) as in Equation 11.1 and the constant \(C\) is defined in Equation 11.3.
Note that in this section and in the previous section we have only reported the minimax risk over all linear estimators of \(\boldsymbol{\theta}\). Since our focus was on linear estimators only, we could derive the minimax risk somewhat directly—that is without the use of any Bayesian arguments such as those used in Chapter 8.
It turns out that finding the minimax rate over all possible estimators \(\hat{\boldsymbol{\theta}}\) of \(\boldsymbol{\theta}\) in the Normal means model when \(\boldsymbol{\theta}\) lies in a Sobolev ellipsoid requires considerably more work. The details are given in Section 3.3 of Tsybakov (2008). Pinsker’s theorem establishes that the minimax risk over all possible estimators in this setting is the same as the linear minimax risk which we have presented.
Even so, we have, in our foray into minimax theory, at last seen something resembling a nonparametric rate: If we make the infinite dimensional Normal means model finite-dimensional with dimension \(n\) and replace the error term variance \(\sigma^2\) with \(\sigma^2/n\), the linear minimax risk presented in Theorem 11.1 will take the form \[ \tilde C n^{-\frac{2\beta}{2\beta + 1}}(1 + o(1))\quad \text{ as } n \to \infty, \] which we can connect to the nonparametric rates of convergence we encountered in previous chapters on nonparametric regression and density estimation.