# Difference between revisions of "Lecture 17. D) Bootstrapping"

m |
(→Bootstrapping) |
||

Line 4: | Line 4: | ||

= Bootstrapping = | = Bootstrapping = | ||

− | The origin of the term “bootstrapping” may relate to someone pulling themselves up by their own boot straps/laces. In a sense, it means making do with little or nothing. Here is the idea: Suppose you would like to conduct an hypothesis test, but were unaware of the test distribution. Even if it converges to a normal, who knows what its asymptotic variance may be ( | + | The origin of the term “bootstrapping” may relate to someone pulling themselves up by their own boot straps/laces. In a sense, it means making do with little or nothing. Here is the idea: Suppose you would like to conduct an hypothesis test, but were unaware of the test distribution. Even if it converges to a normal, who knows what its asymptotic variance may be (e.g., what is the variance of the test statistic when <math display="inline">n</math> tends to infinity)? |

Consider the following approach: If one has enough data, then the distribution in the sample is representative of the distribution of the population. So, one may pretend that the sample itself is the population, and draw from that sample as if one was drawing from the population. | Consider the following approach: If one has enough data, then the distribution in the sample is representative of the distribution of the population. So, one may pretend that the sample itself is the population, and draw from that sample as if one was drawing from the population. | ||

Line 10: | Line 10: | ||

The bootstrap technique can be applied to MLE in the following way. Given a sample of size <math display="inline">N</math>, <math display="inline">\left\{ y_{i},x_{i}\right\} _{i=1}^{N}</math>: | The bootstrap technique can be applied to MLE in the following way. Given a sample of size <math display="inline">N</math>, <math display="inline">\left\{ y_{i},x_{i}\right\} _{i=1}^{N}</math>: | ||

− | * Estimate | + | * Estimate <math display="inline">\widehat{\theta}_{ML}=\text{argmax}_{\theta}\,f\left(\left.y\right|X,\theta\right)</math> as usual. |

− | + | * Calculate the test statistic of interest, <math display="inline">T</math>. (We could use the LRT, for example; notice that we do not know its distribution, nor the appropriate critical value). | |

− | <math display=" | + | * Then, resample (with replacement) <math display="inline">\left\{ y_{i},x_{i}\right\}</math> to get a new (bootstrap) sample <math display="inline">\left\{ y_{j}^{b},x_{j}^{b}\right\} _{j=1}^{N}</math>. Do this <math display="inline">B</math> times, such that each sample can be indexed by <math display="inline">b\in\left\{ 1,..,B\right\} .</math> |

− | + | * For each bootstrap sample, estimate <math display="block">\widehat{\theta}_{b}=\text{argmax}_{\theta}\,f\left(\left.y\right|X,\theta\right).</math> | |

− | as usual. | ||

− | |||

− | * Calculate the test statistic of interest, <math display="inline">T</math>. (We could use the LRT, for example; notice that we do not know its distribution, nor the | ||

− | * Then, resample (with replacement) | ||

− | * For each bootstrap sample, estimate | ||

− | |||

− | <math display="block">\widehat{\theta}_{b}=\text{argmax}_{\theta}\,f\left(\left.y\right|X,\theta\right)</math> | ||

− | |||

* Calculate the test statistic of interest for each estimation, <math display="inline">T^{b}</math>. | * Calculate the test statistic of interest for each estimation, <math display="inline">T^{b}</math>. | ||

While we do not know the distribution of the test statistic, we can approximate it, since we drew it many times from our own sample. Moreover, we can now build confidence intervals for the test statistic (we just need to pick <math display="inline">\underline{t},\overline{t}</math> s.t. 95% of the test statistics <math display="inline">T^{b}</math> fall in the interval), and in the case of the LRT, we can reject the null hypothesis if <math display="inline">T</math> is higher than at least 95% of the <math display="inline">T^{b}</math> tests we drew. Notice that such a test commits a type 1 error with 5% probability, as is often conducted. | While we do not know the distribution of the test statistic, we can approximate it, since we drew it many times from our own sample. Moreover, we can now build confidence intervals for the test statistic (we just need to pick <math display="inline">\underline{t},\overline{t}</math> s.t. 95% of the test statistics <math display="inline">T^{b}</math> fall in the interval), and in the case of the LRT, we can reject the null hypothesis if <math display="inline">T</math> is higher than at least 95% of the <math display="inline">T^{b}</math> tests we drew. Notice that such a test commits a type 1 error with 5% probability, as is often conducted. |

## Latest revision as of 18:46, 2 December 2019

# Bootstrapping

The origin of the term “bootstrapping” may relate to someone pulling themselves up by their own boot straps/laces. In a sense, it means making do with little or nothing. Here is the idea: Suppose you would like to conduct an hypothesis test, but were unaware of the test distribution. Even if it converges to a normal, who knows what its asymptotic variance may be (e.g., what is the variance of the test statistic when [math]n[/math] tends to infinity)?

Consider the following approach: If one has enough data, then the distribution in the sample is representative of the distribution of the population. So, one may pretend that the sample itself is the population, and draw from that sample as if one was drawing from the population.

The bootstrap technique can be applied to MLE in the following way. Given a sample of size [math]N[/math], [math]\left\{ y_{i},x_{i}\right\} _{i=1}^{N}[/math]:

- Estimate [math]\widehat{\theta}_{ML}=\text{argmax}_{\theta}\,f\left(\left.y\right|X,\theta\right)[/math] as usual.
- Calculate the test statistic of interest, [math]T[/math]. (We could use the LRT, for example; notice that we do not know its distribution, nor the appropriate critical value).
- Then, resample (with replacement) [math]\left\{ y_{i},x_{i}\right\}[/math] to get a new (bootstrap) sample [math]\left\{ y_{j}^{b},x_{j}^{b}\right\} _{j=1}^{N}[/math]. Do this [math]B[/math] times, such that each sample can be indexed by [math]b\in\left\{ 1,..,B\right\} .[/math]
- For each bootstrap sample, estimate [math]\widehat{\theta}_{b}=\text{argmax}_{\theta}\,f\left(\left.y\right|X,\theta\right).[/math]
- Calculate the test statistic of interest for each estimation, [math]T^{b}[/math].

While we do not know the distribution of the test statistic, we can approximate it, since we drew it many times from our own sample. Moreover, we can now build confidence intervals for the test statistic (we just need to pick [math]\underline{t},\overline{t}[/math] s.t. 95% of the test statistics [math]T^{b}[/math] fall in the interval), and in the case of the LRT, we can reject the null hypothesis if [math]T[/math] is higher than at least 95% of the [math]T^{b}[/math] tests we drew. Notice that such a test commits a type 1 error with 5% probability, as is often conducted.