Lecture 9. E) Rao-Blackwell
Rao-Blackwell Theorem
The Rao-Blackwell theorem allows us to take an existing estimator, and create a more efficient one. In order to do this, one requires a sufficient statistic.
The theorem states the following:
Let [math]\widehat{\theta}\in W_{u}[/math] and let [math]T[/math] be a sufficient statistic for [math]\theta[/math].
Then,
- [math]\widetilde{\theta}=E\left(\left.\widehat{\theta}\right|T\right)\in W_{u}[/math]
- [math]Var_{\theta}\left(\widetilde{\theta}\right)\leq Var_{\theta}\left(\widehat{\theta}\right),\,\forall\theta\in\Theta[/math]
The new estimator [math]\widetilde{\theta}[/math] is the expected value of a previous one, [math]\widehat{\theta}[/math], conditioning on statistic [math]T[/math]. As we will see, the conditioning preserves the mean (so that if [math]\widehat{\theta}[/math] is unbiased, so is [math]\widetilde{\theta}[/math]), and reduces variance.
Let us first open up the formula for the new estimator:
[math]\begin{aligned} \widetilde{\theta}\left(x\right)=E\left(\left.\widehat{\theta}\right|T\right) & =\int_{-\infty}^{\infty}\widehat{\theta}\left(x\right)f_{\left.X\right|\theta,T}\left(x\right)dx\\ & =\int_{-\infty}^{\infty}\widehat{\theta}\left(x\right)f_{\left.X\right|T}\left(x\right)dx\end{aligned}[/math]
where the second equality follows from the fact that [math]T[/math] is a sufficient statistic. This clarifies why we require a sufficient statistic [math]T[/math] to apply the Rao-Blackwell theorem: If this was not the case, the expectation [math]E\left(\left.\widehat{\theta}\right|T\right)[/math] would have produced a function of [math]\theta[/math], which cannot be an estimator by definition.
We now prove the theorem:
- [math]E_{\theta}\left(\widetilde{\theta}\right)=E_{\theta}\left(E\left(\left.\widehat{\theta}\right|T\right)\right)\underset{L.I.E.}{\underbrace{=}}E_{\theta}\left(\widehat{\theta}\right)\underset{\widehat{\theta}\in W_{u}}{\underbrace{=}}\theta.[/math]
- [math]Var_{\theta}\left(\widetilde{\theta}\right)=Var_{\theta}\left(E\left(\left.\widehat{\theta}\right|T\right)\right)\underset{C.V.I.}{\underbrace{=}}Var_{\theta}\left(\widehat{\theta}\right)-E_{\theta}\left(Var\left(\left.\widehat{\theta}\right|T\right)\right)[/math]. Because [math]E_{\theta}\left(Var\left(\left.\widehat{\theta}\right|T\right)\right)\geq0[/math], [math]Var_{\theta}\left(E\left(\left.\widehat{\theta}\right|T\right)\right)\leq Var_{\theta}\left(\widehat{\theta}\right)[/math].
The operation of producing an estimator via the conditional expectation on a sufficient statistic is often called Rao-Blackwellization.
