Lecture 3. C) Moment Generating Function
Contents
Moment Generating Function
The moment generating function (m.g.f.) is a function that can be used to calculate moments of a r.v. in a different way than the one we defined before. We’ll talk about its use and intuition after we define it.
The m.g.f. of a r.v. [math]X[/math] is the function [math]M_{X}:\mathbb{R}\rightarrow\mathbb{R}_{+}[/math] given by
[math]M_{X}\left(t\right)=E\left(\exp\left(tX\right)\right),t\in\mathbb{R}[/math].
First, notice that the m.g.f. is a function of [math]t[/math], not [math]X[/math]. [math]X[/math] is “integrated out” by the expectation operator. (After we calculate the integral over [math]X[/math], it disappears.)
How does the m.g.f. operate? To see this, first, open up the expectation to obtain
[math]\int_{-\infty}^{\infty}\exp\left(tx\right)f_{X}\left(x\right)dx[/math].
Now, we can take advantage of the fact that the derivative of [math]\exp\left(tx\right)[/math] w.r.t. [math]t[/math] is [math]x\exp\left(tx\right)[/math]. Taking the first derivative of [math]M_{X}\left(t\right)[/math] w.r.t. [math]t[/math] yields [math]\frac{d}{dt}M_{X}\left(t\right)=\frac{d}{dt}\int_{-\infty}^{\infty}\exp\left(tx\right)f_{X}\left(x\right)dx=\int_{-\infty}^{\infty}\frac{\partial}{\partial t}\exp\left(tx\right)f_{X}\left(x\right)dx=\int_{-\infty}^{\infty}x\exp\left(tx\right)f_{X}\left(x\right)dx[/math].
Granted, it’s not clear how this step may have helped. But evaluate this expression at [math]t=0[/math], and we obtain
[math]\int_{-\infty}^{\infty}x\underset{=1}{\underbrace{\exp\left(0x\right)}}f_{X}\left(x\right)dx=\int_{-\infty}^{\infty}xf_{X}\left(x\right)dx[/math], which is the formula for [math]E\left(X\right)[/math].
Suppose now that, rather than one derivative, we took [math]k[/math]. In this case, we will obtain the equality
[math]\frac{d}{dt^{k}}M_{X}\left(t\right)=\int_{-\infty}^{\infty}x^{k}\exp\left(tx\right)f_{X}\left(x\right)dx[/math]. If we evaluate this function at [math]t=0[/math], we obtain the important equality
[math]\left.\frac{d}{dt^{k}}M_{X}\left(t\right)\right|_{t=0}=E\left(X^{k}\right)[/math].
We have established the main result of the m.g.f.: by taking [math]k[/math] derivatives, and then evaluating it at zero, we obtain the "k-th" moment of [math]X[/math]. One of the uses of the m.g.f. (which we will use later), is that if two random variables have the same m.g.f. (i.e., [math]M_{X}\left(t\right)=M_{Y}\left(t\right))[/math], then they have the same distribution, i.e., [math]F_{X}\left(x\right)=F_{Y}\left(x\right)[/math]. Certain conditions are necessary. For starters, notice that we have already assumed that we can differente under the integral sign.
Facts about the MGF
- The m.g.f. only applies if the moments of the r.v. exists.
- A r.v. may have moments, and yet the m.g.f. may yield infinity (notice that we’re taking an integral of an exponential, and so the integral will diverge if the pdf does not approximate zero fast enough). The typical example for this is the log-normal distribution.
- Our proof relied on being able to move the derivative inside of the integral. We have covered the conditions for doing so in the previous lecture.
- If there exists a neighborhood near [math]t=0[/math] where the m.g.f. is finite, i.e. if [math]\exists h\gt 0:M_{X}\left(t\right)\lt \infty,t\in\left(-h,h\right)[/math], then it is possible to show that the m.g.f. uniquely identifies the c.d.f. of [math]X[/math]. This means that if [math]M_{X}\left(t\right)=M_{Y}\left(t\right)[/math] and this condition is met, then [math]F_{X}\left(x\right)=F_{Y}\left(x\right),\forall x\in\mathbb{R}[/math]. In other words, the m.g.f. can be used to identify the distribution of a random variable uniquely, provided it is finite in a neighborhood of zero. The proof is not trivial.
- If the m.g.f. is finite in a neighborhood of zero, then all of the moments of the distribution exist.
- When the m.g.f. does not exist around a neighborhood of zero, we can always use the characteristic function to characterize a distribution. This function is given by [math]C_{X}:\mathbb{R}\rightarrow\mathbb{C}[/math], where [math]C_{X}\left(t\right)=E\left(\exp\left(itX\right)\right)=E\left(\text{cos}\left(tX\right)+i.sin\left(tX\right)\right)[/math].
Standard Normal Distribution
Let's calculate the mgf of the standard normal distribution. When
[math]X\sim N\left(0,1\right)[/math], then [math]M_{X}\left(t\right)=\int_{-\infty}^{\infty}\exp\left(tx\right)\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{x^{2}}{2}\right)dx[/math].
We can rearrange the integrand:
[math]\int_{-\infty}^{\infty}\exp\left(tx\right)\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{x^{2}}{2}\right)dx=\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\left(x-t\right)^{2}}{2}\right)\exp\left(\frac{t^{2}}{2}\right)dx=\exp\left(\frac{t^{2}}{2}\right)\int_{-\infty}^{\infty}\underset{\text{pdf}\,N\left(t,1\right)}{\underbrace{\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\left(x-t\right)^{2}}{2}\right)}}dx.[/math]
The first equality follows from completing the square, and the second equality uses the fact that the integral is w.r.t. [math]x[/math], not [math]t[/math]. Note also that the integrand of the last expression is the pdf of the [math]Normal(t,1)[/math] distribution, and so its integral equals 1. Hence, we obtain [math]M_{X}\left(t\right)=\exp\left(\frac{t^{2}}{2}\right)[/math]. We can test that it produces the mean and variance of [math]N\left(0,1\right)[/math].
MGF of Affine Transformations
A useful fact about the m.g.f.: If [math]X[/math] has m.g.f. [math]M_{X}\left(t\right)[/math], then [math]M_{aX+b}\left(t\right)=\exp\left(bt\right)M_{X}\left(at\right)[/math]. This is easily proved by expanding [math]E\left(\exp\left(aX+b\right)t\right)[/math]. This implies the following for the general normal distribution:
Suppose [math]X\sim N\left(\mu,\sigma^{2}\right)[/math], such that [math]X=\mu+\sigma Z[/math] where [math]Z\sim N\left(0,1\right).[/math]
Then,
[math]M_{X}\left(t\right)=M_{\mu+\sigma Z}\left(t\right)=\exp\left(\mu t\right)M_{Z}\left(\sigma t\right)=\exp\left(\mu t\right)\exp\left(\frac{1}{2}\sigma^{2}t^{2}\right)=\exp\left(\mu t+\frac{1}{2}\sigma^{2}t^{2}\right).[/math]
We can now use this m.g.f. to calculate the moments of the general normal distribution [math]N\left(\mu,\sigma^{2}\right).[/math]
