# Point Estimation (cont.)

## Example: Uniform

Suppose $X_{i}\overset{iid}{\sim}U\left(0,\theta\right)$ where $\theta\gt 0$ is unknown.

The likelihood function equals $L\left(\left.\theta\right|x_{1}..x_{n}\right)=\Pi_{i=1}^{n}f\left(\left.x_{i}\right|\theta\right)=\Pi_{i=1}^{n}\frac{1}{\theta}1\left(0\leq x_{i}\leq\theta\right)$

Since $x_{i}$’s are draws from $U\left(0,\theta\right)$, the $0\leq x_{i}$ constraint will always be satisfied. However, since we are uncertain about the true value of $\theta$, the upper constraint may be binding.

This yields the following likelihood: $\Pi_{i=1}^{n}\frac{1}{\theta}1\left(0\leq x_{i}\leq\theta\right)=\Pi_{i=1}^{n}\frac{1}{\theta}1\left(x_{i}\leq\theta\right)=\frac{1}{\theta^{n}}1\left(x_{\left(n\right)}\leq\theta\right)$

Notice that $L\left(\left.\cdot\right|x_{1}..x_{n}\right)$ is not differentiable at $\theta=x_{\left(n\right)}$. We separate the problem:

• $L\left(\left.\cdot\right|x_{1}..x_{n}\right)=0$ if $\theta\lt x_{\left(n\right)}$; this reveals the impossibility that a value is generated above $\theta$.
• $L\left(\left.\cdot\right|x_{1}..x_{n}\right)=\frac{1}{\theta^{n}}$ if $\theta\geq x_{\left(n\right)}$; it is decreasing in $\theta$, so constraint is active, and $\widehat{\theta}_{ML}=x_{\left(n\right)}$.

Notice that the maximum likelihood estimator is different from the method of moments, $\widehat{\theta}_{ML}=x_{\left(n\right)}$ while $\widehat{\theta}_{MM}=2\overline{x}$.

Unlike the method of moments, we cannot obtain an estimator st $x_{i}\gt \widehat{\theta}_{ML}$.

However, as we will discuss later, there are some bad news.

The fact that we can never obtain $\widehat{\theta}_{ML}\gt \theta_{0}$, where $\theta_{0}$ is the true value of parameter $\theta$, means that the maximum likelihood estimator is likely to systematically underestimate the true parameter value.

# Evaluating Estimators

A good estimator of $\theta$ is close to $\theta$ in some probabilistic sense. For reasons of convenience, the leading criterion is the mean squared error:

The mean squared error (MSE) of an estimator $\widehat{\theta}$ of $\theta\in\Theta\subseteq\mathbb{R}$ is a function (of $\widehat{\theta}$) given by

$MSE_{\theta}\left(\widehat{\theta}\right)=E_{\theta}\left[\left(\theta-\widehat{\theta}\right)^{2}\right]$

Where from here on, we use notation $E_{\theta}\left[\cdot\right]=E_{\theta}\left[\left.\cdot\right|\theta\right]$, that is, the subscript indicates the variable to be conditioned on (before, it used to mean the variable of integration). So,

$MSE_{\theta}\left(\widehat{\theta}\right)=E_{\theta}\left[\left(\theta-\widehat{\theta}\right)^{2}\right]=E\left[\left.\left(\theta-\widehat{\theta}\right)^{2}\right|\theta\right].$

The interpretation is that MSE gives us the expected quadratic difference between our estimator and a specific value of $\theta$, which we usually assume to be some true value.

MSE is mostly popular due to its tractability. When $\theta$ is a vector of parameters, we employ the vector version instead:

$MSE_{\theta}\left(\widehat{\theta}\right)=E_{\theta}\left[\left(\theta-\widehat{\theta}\right).\left(\theta-\widehat{\theta}\right)^{'}\right].$

The vector version of the MSE produces a matrix. Two compare 2 estimator vectors, we compare these matrices. Namely, we say an MSE is lower than another if the difference of the matrices is positive semi-definite (i.e., $z'.M.z\geq0,\,\forall z\neq0$). We will confine ourselves to the scalar case most of the time.

If $MSE_{\theta}\left(\widehat{\theta}_{1}\right)\gt MSE_{\theta}\left(\widehat{\theta}_{2}\right)$ for all values of $\theta$, we are tempted to say that $\widehat{\theta}_{2}$ is better, since it is on average closer to $\theta$, whatever value it has. However, we may feel different if $\widehat{\theta}_{2}$ systematically underestimates (or overestimates) $\theta$, where as $\widehat{\theta}_{1}$ is on average correct.

In order to take this into account, we introduce the concept of bias:

$Bias_{\theta}\left(\widehat{\theta}\right)=E_{\theta}\left(\theta-\widehat{\theta}\right)$

Whenever $E_{\theta}\left(\theta-\widehat{\theta}\right)=0$ - or equivalently, $E_{\theta}\left(\widehat{\theta}\right)=\theta$ - we say estimator $\widehat{\theta}$ is unbiased.

What follows is fundamental result about the decomposition of the MSE:

$MSE_{\theta}\left(\widehat{\theta}\right)=Var_{\theta}\left(\widehat{\theta}\right)+Bias_{\theta}\left(\widehat{\theta}\right)^{2}$

This means that, for estimators with a given MSE, for example, there is a tradeoff between bias and variance.

The proof of the result is obtained by adding and subtracting $E_{\theta}\left(\widehat{\theta}\right)$:

\begin{aligned} MSE_{\theta}\left(\widehat{\theta}\right)=E_{\theta}\left[\left(\theta-\widehat{\theta}\right)^{2}\right] & =E_{\theta}\left[\left(\theta-\widehat{\theta}+E_{\theta}\left(\widehat{\theta}\right)-E_{\theta}\left(\widehat{\theta}\right)\right)^{2}\right]\\ & =E_{\theta}\left[\left(\widehat{\theta}-E_{\theta}\left(\widehat{\theta}\right)\right)^{2}\right]+\underset{=\left(\theta-E_{\theta}\left(\widehat{\theta}\right)\right)^{2}}{\underbrace{E_{\theta}\left[\left(E_{\theta}\left(\widehat{\theta}\right)-\theta\right)^{2}\right]}}+\underset{=0}{\underbrace{...}}\\ & =Var_{\theta}\left(\widehat{\theta}\right)+Bias_{\theta}\left(\widehat{\theta}\right)^{2}\end{aligned}

We now define an efficient estimator:

Let $W$ be a collection of estimators of $\theta\in\Theta$. An estimator $\widehat{\theta}$ is efficient relative to $W$ if

$MSE_{\theta}\left(\widehat{\theta}\right)\leq MSE_{\theta}\left(w\right),\,\forall\theta\in\Theta,\,\forall w\in W$.

In order to find a “best” estimator, we have to restrict $W$ in some way (otherwise, we can often find many estimators with equal MSE, by exploiting the bias/variance tradeoff).

# Minimum Variance Estimators

We usually focus our attention on unbiased estimators. Those that, one average, produce the correct result.

The collection of unbiased estimators is

$W_{u}=\left\{ w:\,Bias_{\theta}\left(w\right)=0,\,Var_{\theta}\left(w\right)\lt \infty,\,\forall\theta\in\Theta\right\}$.

So, if $\widehat{\theta}\in W_{u}$, then $MSE_{\theta}\left(\widehat{\theta}\right)=Var_{\theta}\left(\widehat{\theta}\right).$

We can now define a type of minimum variance estimator:

An estimator $\widehat{\theta}\in W_{u}$ of $\theta$ is a uniform minimum-variance unbiased (UMVU) estimator of $\theta$ if it is efficient relative to $W_{u}$.

The minimum-variance unbiased part of UMVU should be clear. Of the unbiased estimators, $\widehat{\theta}$ is “MVU” if it achieves the lowest variance and is unbiased. The “uniform” part simply means that $\widehat{\theta}$ is unbiased and minimum variance for all values that $\theta$ may hold. It is MVU if $\theta=4$, and if $\theta=-3$, etc.

It is often possible to identify UMVU estimators. The tool to do this is the Rao-Blackwell theorem. Before we do so, we need to introduce an additional concept.

# Sufficient Statistics

Let $X_{1}..X_{n}$ be a random sample from a distribution with pmf/pdf $f\left(\left.\cdot\right|\theta\right)$, where $\theta\in\Theta$ is unknown.

A statistic $T=T\left(X_{1}..X_{n}\right)$ is a sufficient statistic for parameter $\theta$ if the conditional pmf/pdf of $\left(X_{1}..X_{n}\right)$ given $T$ does not depend on $\theta$, i.e.,

$f\left(\left.X\right|\theta,T\right)=f\left(\left.X\right|T\right).$

The reason we are interested in sufficient statistics will be clear once we present the Rao-Blackwell theorem. However, it is worth thinking a bit about the meaning of sufficient statistics first.

Sufficient statistics are the portion of the data that is useful to calculate the maximum likelihood estimator of $\theta$.

## Intuitive Example: Uniform

In order to estimate $\theta$ via maximum likelihood, one writes down the likelihood function of the sample:

$f_{X_{1}..X_{n}}\left(\left.x\right|\theta\right)=\frac{1}{\theta^{n}}1\left(x_{1}\leq\theta\wedge x_{2}\leq\theta\wedge...\wedge x_{n}\leq\theta\right).$

We can rewrite the pdf of the sample as:

$f_{X_{1}..X_{n}}\left(\left.x\right|\theta\right)=\frac{1}{\theta^{n}}1\left(\max_{i=1..n}x_{i}\leq\theta\right).$

From the expression above, it is clear that the MLE only depends on the maximum observation and not the whole sample.

Hence, $\max_{i=1..n}X_{i}$ is a sufficient statistic of $\theta$.

## Remark: What does $f\left(\left.X\right|\theta,T\right)$ mean?

At this point, you have have worked with conditional pdfs. For example, you may have worked with a pdf $f_{X|Y}$, where $X$ and $Y$ are random variables.

However, the expression $f\left(\left.X\right|\theta,T\right)$ means something slightly different. The reason is that $T=T\left(X_{1}..X_{n}\right)$ is a function of the random variables $X_{1}..X_{n},$ i.e., it's a function of the sample.

The simplest way to see this is to consider the following joint pmf:

$f_{X_{1},X_{2}}$
$X_{1}$
0 1 2
$X_{2}$ 0 5% 7% 8%
1 20% 5% 5%
2 15% 25% 10%

and suppose we would like to calculate the pmf

$f_{X_{1},X_{2}|X_{1}+X_{2}=1}.$

Clearly, this pmf is obtained by dividing the probability of observing each combination of $X_{1},X_{2}$ - for the cases where $X_{1}+X_{2}=1$ - by $P\left(X_{1}+X_{2}=1\right)$, whereas the remaining elements will be equal to zero. This yields the conditional pmf:

$f_{X_{1},X_{2}|X_{1}+X_{2}=1}$
$X_{1}$
0 1 2
$X_{2}$ 0 0 26% 0
1 74% 0 0
2 0 0 0

where $7\% \div (20\%+7\%) = 26\%$ and $20\% \div (20\%+7\%) = 74\%.$ In this example, $T\left(X_{1}..X_{n}\right)=X_{1}+X_{2}.$

It follows that

$f_{X|T}\left(x|t\right)=\frac{f_X\left(x\right)}{f_T\left(t\right)}.1\left(T\left(x\right)=t\right).$

This result is valid for both pmfs and pdfs.

The result is a bit different from the one we are used to, when we condition $X$ on $Y$, for example. The reason is that in this case we are not conditioning $X$ on a different random variable. Rather, we are interested in the pdf of a random vector $X$, conditional on it respecting some equality $T\left(X\right)=t.$

<< Minimum Variance Estimators
Rao-Blackwell Theorem >>

# Rao-Blackwell Theorem

The Rao-Blackwell theorem allows us to take an existing estimator, and create a more efficient one. In order to do this, one requires a sufficient statistic.

The theorem states the following:

Let $\widehat{\theta}\in W_{u}$ and let $T$ be a sufficient statistic for $\theta$.

Then,

• $\widetilde{\theta}=E\left(\left.\widehat{\theta}\right|T\right)\in W_{u}$
• $Var_{\theta}\left(\widetilde{\theta}\right)\leq Var_{\theta}\left(\widehat{\theta}\right),\,\forall\theta\in\Theta$

The new estimator $\widetilde{\theta}$ is the expected value of a previous one, $\widehat{\theta}$, conditioning on statistic $T$. As we will see, the conditioning preserves the mean (so that if $\widehat{\theta}$ is unbiased, so is $\widetilde{\theta}$), and reduces variance.

Let us first open up the formula for the new estimator:

\begin{aligned} \widetilde{\theta}\left(x\right)=E\left(\left.\widehat{\theta}\right|T\right) & =\int_{-\infty}^{\infty}\widehat{\theta}\left(x\right)f_{\left.X\right|\theta,T}\left(x\right)dx\\ & =\int_{-\infty}^{\infty}\widehat{\theta}\left(x\right)f_{\left.X\right|T}\left(x\right)dx\end{aligned}

where the second equality follows from the fact that $T$ is a sufficient statistic. This clarifies why we require a sufficient statistic $T$ to apply the Rao-Blackwell theorem: If this was not the case, the expectation $E\left(\left.\widehat{\theta}\right|T\right)$ would have produced a function of $\theta$, which cannot be an estimator by definition.

We now prove the theorem:

• $E_{\theta}\left(\widetilde{\theta}\right)=E_{\theta}\left(E\left(\left.\widehat{\theta}\right|T\right)\right)\underset{L.I.E.}{\underbrace{=}}E_{\theta}\left(\widehat{\theta}\right)\underset{\widehat{\theta}\in W_{u}}{\underbrace{=}}\theta.$
• $Var_{\theta}\left(\widetilde{\theta}\right)=Var_{\theta}\left(E\left(\left.\widehat{\theta}\right|T\right)\right)\underset{C.V.I.}{\underbrace{=}}Var_{\theta}\left(\widehat{\theta}\right)-E_{\theta}\left(Var\left(\left.\widehat{\theta}\right|T\right)\right)$. Because $E_{\theta}\left(Var\left(\left.\widehat{\theta}\right|T\right)\right)\geq0$, $Var_{\theta}\left(E\left(\left.\widehat{\theta}\right|T\right)\right)\leq Var_{\theta}\left(\widehat{\theta}\right)$.

The operation of producing an estimator via the conditional expectation on a sufficient statistic is often called Rao-Blackwellization.

# Factorization Theorem

As we saw in the example of the Normal distribution, it can be tedious to find a sufficient statistic for a parameter. Luckily, the factorization theorem makes it easy, provided the pmf/pdf of the sample is available:

Let $X_{1}..X_{n}$ be a random sample from a distribution with pmf/pdf $f\left(\left.\cdot\right|\theta\right)$, where $\theta\in\Theta$ is unknown.

A statistic $T=T\left(X_{1}..X_{n}\right)$ is sufficient for $\theta$ if and only if there exist functions $g\left(\cdot\right)$ and $h\left(\cdot\right)$ s.t.

$\Pi_{i=1}^{n}f\left(\left.x_{i}\right|\theta\right)=g\left(\left.T\left(x_{1},...,x_{n}\right)\right|\theta\right).h\left(x_{1},...,x_{n}\right),$

for every $\left(x_{1}..x_{n}\right)\in\mathbb{R}^{n}$ and every $\theta\in\Theta.$

One way to understand the result above, is that if we were to maximize the likelihood above, only the first factor would be relevant, since the second equals a constant that is independent of the parameters.

## Example: Uniform

Suppose $X_{i}\sim U\left(0,\theta\right)$ such that the joint pdf equals

$\Pi_{i=1}^{n}f\left(\left.x_{i}\right|\theta\right)=\underset{g\left(\left.x_{\left(n\right)}\right|\theta\right)}{\underbrace{\frac{1}{\theta^{n}}.1\left(x_{\left(n\right)}\leq\theta\right)}}.\underset{h\left(x_{1}..x_{n}\right)}{\underbrace{1\left(x_{\left(1\right)}\geq0\right)}}$

From the factorization theorem, $x_{\left(n\right)}$ is a sufficient statistic for $\theta$. Again, notice that the maximization of the likelihood function w.r.t. $\theta$ will only depend on $x_{\left(n\right)}$, since $h\left(x_{1}..x_{n}\right)$ is a constant that will not affect the estimator.