Lecture 11. D) Testing Errors
Type 1 and Type 2 Errors
The critical value is a fundamental aspect of hypothesis testing. In the previous normal example, we chose a critical value of [math]2[/math]. Was this a good idea? Surely the probability of selecting the right hypothesis is not 100% either way.
For any critical value, there will be cases where we will say that [math]\mu=0[/math] when in reality [math]\mu\gt 0[/math], and the converse will also happen. If we were always able to pick the right hypothesis, then there would be no uncertainty.
Let us first organize the possible cases of “hit and miss” in the following table:
| Truth\Decision | [math]H_{0}[/math] | [math]H_{1}[/math] |
| [math]H_{0}[/math] | [math]\unicode{x2714}[/math] | Type 1 error |
| [math]H_{1}[/math] | Type 2 error | [math]\unicode{x2714}[/math] |
The main diagonal is simple to memorize: If [math]H_{0}[/math] is true and we opt for [math]H_{0}[/math] (or if [math]H_{1}[/math] is true and... you get the point), then no errors were made. If we decide for [math]H_{1}[/math] when [math]H_{0}[/math] is true, then we have committed a type 1 error. If we decide for [math]H_{0}[/math] when [math]H_{1}[/math] is true, then we committed a type 2 error.
Example: Normal
Let’s use the normal example from before. After all, we need to get used to these relatively artificial error names.
In our example,
[math]H_{0}:\mu=0[/math] and [math]H_{1}:\mu\gt 0[/math].
- If we reject [math]\mu=0[/math] when it was true, then we commit a type 1 error.
- If we accept [math]\mu=0[/math] and it was false, then we commit a type 2 error.
So, provided an error was made, type 1 error happens when rejecting the null, type 2 error occurs when accepting the null. Rehearse this: Type 1 error, reject the null.... Type 1 error, reject the null...
Often, we select the critical value so that [math]P_{\theta_{0}}\left(\text{type 1 error}\right)\leq5\%[/math]. One interpretation typical in sciences is that we would like to be conservative, and only rarely reject the null hypothesis erroneously.
The 5% threshold used above is arbitrary. For example, in the natural sciences where experiments can be reproduced with high precision, we may use [math]p=0.003[/math] or even lower.
We will talk about the probability of committing a type 2 error in the next lecture. As a preview, we will minimize that probability, constrained by the fact that the probability of a type 1 error cannot surpass 5% (or some other established level).
An incredibly useful too to analyze this problem further is the power function (and then some graphs).
A Quick Note on Notation
Statements like [math]P_{\theta_{0}}\left(\text{type 1 error}\right)\leq5\%[/math] will be frequent from here on.
When using the subscript notation [math]P_{\theta_{0}}\left(\cdot\right)[/math], it may appear that we mean [math]P\left(\cdot|\theta=\theta_{0}\right)[/math]. The issue with this statement is that we do not consider [math]\theta[/math] to be a random variable (rather, it is the true value of the parameter, a constant), so it does not make sense to condition on it. Hence, the use of the subscript notation.
