Quality of Tests

• Every hypothesis test has 4 possible outcomes — 2 are correct, 2 are errors (Type I and Type II).
• Type I error = rejecting H₀ when it was actually true (a "false alarm").
• Type II error = not rejecting H₀ when it was actually false (a "missed detection").
• P(Type I error) is found assuming H₀ is true, and equals (or is close to) the significance level.
• P(Type II error) requires you to be told the actual/true value of the parameter — you can't find it just from H₀.
• Size of a test = P(Type I error). A smaller size is better.
• Power of a test = P(correctly rejecting H₀) = 1 − P(Type II error). A bigger power is better.
• Reducing one error's probability tends to increase the other — the only way to shrink both is a bigger sample size.
• A power function is the power written algebraically in terms of the parameter (p or λ), instead of needing an actual numeric value.

The four outcomes of a hypothesis test

Think about it like a courtroom. The defendant is either actually guilty or actually innocent (that's reality — H₀ is either true or false). The jury then reaches a verdict — guilty (reject H₀) or not guilty (don't reject H₀). Put those two things together and you get four possible combinations:

Two of these are good news (the jury reached the correct verdict). Two are bad news — the errors:

A useful mental trick: Type I is the error you make by being too trigger-happy (rejecting when you shouldn't). Type II is the error you make by being too cautious (not rejecting when you should have).

Finding P(Type I error)

This connects directly back to how the critical region was built in the first place — it was constructed using the significance level, α%, under the assumption that H₀ was true. So:

• Continuous distributions