Lecture 14. F) Delta Method
Delta Method
The delta method allows us to approximate the distribution of the transformation of a random variable, as long as [math]n[/math] is large.
The delta method states that if a sequence of random variables [math]X_{n}[/math] satisfies [math]\sqrt{n}\left(X_{n}-\mu\right)\overset{d}{\rightarrow}N\left(0,\sigma^{2}\right)[/math] then for any function [math]g\left(\cdot\right)[/math] continuously differentiable in a neighborhood of [math]\mu[/math] with derivative [math]g^{'}\left(\mu\right)[/math],
[math]\sqrt{n}\left(g\left(X_{n}\right)-g\left(\mu\right)\right)\overset{d}{\rightarrow}N\left(0,g^{'}\left(\mu\right)^{2}.\sigma^{2}\right)[/math]
Proof
The proof involves expanding [math]g\left(X_{n}\right)[/math] around [math]\mu[/math]:
[math]g\left(X_{n}\right)=g\left(\mu\right)+\left(X_{n}-\mu\right)g^{'}\left(\mu\right)+R\left(X_{n}\right)[/math].
We can rewrite this version of the Taylor expansion using the mean value theorem, which yields the alternative version
[math]g\left(X_{n}\right)=g\left(\mu\right)+\left(X_{n}-\mu\right)g^{'}\left(\tilde{X}_{n}\right),\,for\,some\,\tilde{X}_{n}\in\left(X_{n},\mu\right)[/math].
Moving [math]g\left(\mu\right)[/math] to the lhs yields [math]g\left(X_{n}\right)-g\left(\mu\right)=\left(X_{n}-\mu\right)g^{'}\left(\tilde{X}_{n}\right)[/math]
Because [math]\tilde{X}_{n}[/math] lies between [math]X_{n}[/math] and [math]\mu[/math], [math]\tilde{X}_{n}\overset{p}{\rightarrow}\mu[/math] if [math]X_{n}\overset{p}{\rightarrow}\mu[/math].
Remembering that
[math]\sqrt{n}\left(X_{n}-\mu\right)\overset{d}{\rightarrow}N\left(0,\sigma^{2}\right)[/math],
then,
[math]\underset{\overset{d}{\rightarrow}N\left(0,\sigma^{2}\right)}{\underbrace{\sqrt{n}\left(X_{n}-\mu\right)}}.\underset{\overset{p}{\rightarrow}g^{'}\left(\mu\right)}{\underbrace{g^{'}\left(\tilde{X}_{n}\right)}}\Rightarrow\sqrt{n}\left(g\left(X_{n}\right)-g\left(\mu\right)\right)\overset{d}{\rightarrow}N\left(0,g^{'}\left(\mu\right)^{2}.\sigma^{2}\right)[/math]
To obtain the result above, we first used the facts that [math]\sqrt{n}\left(X_{n}-\mu\right)\overset{d}{\rightarrow}N\left(0,\sigma^{2}\right)[/math] and that [math]g^{'}\left(\tilde{X}_{n}\right)\overset{p}{\rightarrow}g^{'}\left(\mu\right)[/math]. By Slutsky’s theorem, the product converges in distribution to the distribution of [math]g^{'}\left(\mu\right).\sigma.Z[/math], where [math]Z=N\left(0,1\right)[/math]. Finally, the last result follows from the fact that [math]\sqrt{n}\left(X_{n}-\mu\right)g^{'}\left(\tilde{X}_{n}\right)=\sqrt{n}\left(g\left(X_{n}\right)-g\left(\mu\right)\right)[/math].
This result is often useful to calculate the distribution of a transformation of an estimator. For example, while estimating a model, it may be convenient to constrain a parameter estimate of [math]\theta[/math] to be positive. One way to accomplish this is to estimate some parameter [math]\widehat{\beta}[/math], s.t. [math]\widehat{\theta}=\exp\left(\widehat{\beta}\right)[/math]. For any value of [math]\widehat{\beta}[/math], we will obtain a positive estimate of [math]\widehat{\theta}[/math] by applying the exponential. If we know the distribution of [math]\sqrt{n}\left(\widehat{\beta}-E\left(\widehat{\beta}\right)\right)[/math] has a [math]N\left(0,\sigma^{2}\right)[/math] distribution, then the delta method can be used to produce the distribution of [math]\widehat{\theta}[/math]. To be clear, in this case the sequence of random variables [math]X_{n}[/math] would be data, such that [math]\widehat{\beta}_{n}=\widehat{\beta}\left(X_{n}\right)[/math], for example.
