Library › Quality of Tests

Further Mathematics

Quality of Tests

Revise Quality of Tests for Further Mathematics — revision notes and instant AI marking. Free to start.

📖 Revision notes · preview

Edexcel A Level Further Maths — Further Statistics 1

Quality of Tests

Every hypothesis test can go wrong in two different ways — and "size" and "power" are just the numbers that tell you how good your test is at avoiding those mistakes.

Summary — The Big Picture

Every hypothesis test has 4 possible outcomes — 2 are correct, 2 are errors (Type I and Type II).
Type I error = rejecting H₀ when it was actually true (a "false alarm").
Type II error = not rejecting H₀ when it was actually false (a "missed detection").
P(Type I error) is found assuming H₀ is true, and equals (or is close to) the significance level.
P(Type II error) requires you to be told the actual/true value of the parameter — you can't find it just from H₀.
Size of a test = P(Type I error). A smaller size is better.
Power of a test = P(correctly rejecting H₀) = 1 − P(Type II error). A bigger power is better.
Reducing one error's probability tends to increase the other — the only way to shrink both is a bigger sample size.
A power function is the power written algebraically in terms of the parameter (p or λ), instead of needing an actual numeric value.

1. Type I & Type II Errors

The four outcomes of a hypothesis test

Think about it like a courtroom. The defendant is either actually guilty or actually innocent (that's reality — H₀ is either true or false). The jury then reaches a verdict — guilty (reject H₀) or not guilty (don't reject H₀). Put those two things together and you get four possible combinations:

	Reject H₀	Accept (don't reject) H₀
H₀ True	Type I error	✓ No error
H₀ False	✓ No error	Type II error

Two of these are good news (the jury reached the correct verdict). Two are bad news — the errors:

Type I Error — "False Positive" H₀ was actually true, but your test gave you enough evidence to reject it anyway.
Courtroom version: an innocent person is found guilty.
Real-life version: a coin is actually fair, but your 20 flips happened to come up so heavy on heads that you (wrongly) concluded it was biased.

Type II Error — "False Negative" H₀ was actually false, but your test failed to reject it — you didn't have enough evidence to spot that something had changed.
Courtroom version: a guilty person is found innocent (walks free).
Real-life version: a coin really is biased towards heads, but your sample of flips just happened to look "normal enough" that you didn't detect the bias.

A useful mental trick: Type I is the error you make by being too trigger-happy (rejecting when you shouldn't). Type II is the error you make by being too cautious (not rejecting when you should have).

Finding P(Type I error)

Formula P(Type I error) = P(being in the critical region | H₀ is true)

In plain English: assume H₀ is true, then work out the probability that your test statistic still happens to land in the critical (rejection) region.

This connects directly back to how the critical region was built in the first place — it was constructed using the significance level, α%, under the assumption that H₀ was true. So:

Continuous distributions (Normal, χ²): P(Type I error) is exactly α%. You can often just write this down — no calculation needed, because continuous critical regions can be built to hit the significance level exactly.
Discrete distributions (Binomial, Poisson, Geometric): you usually can't hit α% exactly, because probabilities jump in discrete chunks. So the critical region is chosen to get as close as possible to α% while still being a valid critical region (≤ α%). P(Type I error) is then this actual significance level — and you do need to calculate it.

Why the discrete case is different Imagine you want a 5% critical region for a Binomial distribution. There might be no value of X where P(X in region) is exactly 0.05 — it might jump from 0.07 to 0.03 depending on where you draw the line. Statisticians always choose the region so the actual probability stays at or below α (you never want to accidentally reject H₀ more often than you said you would).

Finding P(Type II error)

Formula P(Type II error) = P(not being in the critical region | actual population parameter)

Here's the crucial difference: you must be told the actual (true) value of the parameter to calculate this. You can't work it out just from H₀ being "false" — false could mean anything!

For example, if H₀ assumed p = ½, saying "H₀ is false" only tells you p ≠ ½ — that's not specific enough to calculate a probability. You need something concrete, like being told the actual value is p = ⅓. Only then can you calculate the probability of your test failing to detect that change.

The golden rule of P(Type II error) Type I error calculations use the hypothesised parameter (from H₀).
Type II error calculations use the actual, true parameter (given in the question).
Mixing these up is the single most common mistake on this topic!

Reducing the errors

The trade-off

Reduce the significance level (α) → P(Type I error) goes down, but P(Type II error) goes up.
Increase the significance level (α) → P(Type II error) goes down, but P(Type I error) goes up.
The only way to reduce both at the same time is to increase the sample size.

Think of it like a smoke alarm's sensitivity dial. Turn the sensitivity up (easier to trigger = smaller α needed to "not trigger", i.e. stricter test) and you'll catch every real fire (fewer Type II errors) — but you'll also get woken up by burnt toast a lot more often (more Type I errors). Turn it down and you avoid false alarms, but you risk missing a real fire. A bigger, better sensor (bigger sample) is the only way to improve both at once.

Practice Question 1

Lucy can hit the target 70% of the time when she throws an axe with her right hand. She claims that the proportion, p, of her throws that hit the target is higher than 70% when she uses her left hand. Lucy uses the hypotheses H₀: p = 0.7 and H₁: p > 0.7 to test her claim. Lucy makes 100 throws and will reject the null hypothesis if the axe hits the target more than 77 times.

(a) Find the probability of a Type I error.
(b) Given that Lucy actually hits the target 80% of the time with her left hand, find the probability of a Type II error.

Practice Question 2

A researcher tests, at the 5% significance level, whether a die is biased towards rolling a six, using 60 rolls of the die. Explain why P(Type I error) can be stated as "exactly 5%" if the test used a Normal approximation, but not if the test used the exact Binomial distribution.

2. Size of a Test

What is "size"?

The size of a test is just another name for something you've already met: it's the probability of rejecting H₀ when H₀ was actually true.

Formula Size = P(in critical region | H₀ is true)

This is identical to P(Type I error). They are the same calculation with a different name.

A "better" test has a smaller size — you want the probability of wrongly rejecting a true H₀ to be as low as possible. This links directly to the significance level:

Continuous distributions: Size = significance level α (exactly). Often needs no calculation.
Discrete distributions: Size = actual significance level (≤ α), the closest achievable value while remaining a valid critical region.

The one thing to remember Size = P(Type I error). Full stop. If a question asks for "the size of the test," you calculate it in exactly the same way you calculated P(Type I error) earlier — using the H₀ value of the parameter.

Worked Example

A student wants to test, at a 10% significance level, whether a coin is biased towards heads by counting the number of heads in 20 flips of the coin. Calculate the size of this test.

Let X = number of heads. X ~ B(20, p), H₀: p = ½, H₁: p > ½, α = 0.1

Under H₀: X ~ B(20, ½). We need the critical region — the smallest set of extreme values with probability ≤ 0.1.

P(X ≥ 13) = 0.1316 > 0.1 (too big, X ≥ 13 not extreme enough alone)

P(X ≥ 14) = 0.0577 < 0.1 ✓

So the critical region is X ≥ 14.

The actual significance level (= size) is the actual probability of landing in this region: P(X ≥ 14) = 0.5766...

Size = 0.577 (3 s.f.)

Wait, 0.577 for a "10% test"? Yes — that number looks huge compared to the 10% we asked for! Don't panic; this is a classic feature of small discrete samples. There simply isn't a value of X between "X ≥ 13" (prob 0.132) and "X ≥ 14" (prob 0.058) — no cut-off gives anything close to 0.10, so with such a small sample the achievable size can look very different from the target α. This is exactly why we always check the actual critical region achieved rather than assuming discrete tests hit their target neatly.

Practice Question

A test is conducted at the 5% significance level using X ~ B(15, p), H₀: p = 0.3, H₁: p > 0.3. Given that the critical region is found to be X ≥ 8, and P(X ≥ 8 | p = 0.3) = 0.0348, state the size of the test.

3. Power of a Test

What is "power"?

If size measures how good a test is at avoiding false alarms, power measures how good a test is at actually catching real changes. It's the flip side of the same coin.

Formula Power = P(in critical region | H₀ is false)

In words: given that something really has changed (H₀ is genuinely false), what's the probability your test correctly picks up on it and rejects H₀?

Just like with P(Type II error), you need to be given the actual value of the parameter to calculate power — "H₀ is false" alone isn't specific enough.

Key Relationship Power = 1 − P(Type II error)

This makes total sense once you see it: Power is "H₀ false AND correctly rejected" (a good outcome). Type II error is "H₀ false AND NOT rejected" (a bad outcome). Since these two outcomes cover every possibility when H₀ is false, their probabilities must add to 1.

A better test has higher power — you want to maximise the chance of correctly detecting a real change. Ideally, you want power > 0.5, meaning your test is more likely than not to reach the correct conclusion when H₀ really is false.

Worked Example

Let X ~ Po(λ). A hypothesis test is conducted at the 5% significance level in which H₀: λ = 8 and H₁: λ < 8. It is later discovered that λ = 6. Find the power of the test.

Step 1 — find the critical region, assuming H₀ is true (λ = 8):

X ~ Po(8): P(X ≤ 4) = 0.0996 > 0.05; P(X ≤ 3) = 0.0424 < 0.05

So critical region is X ≤ 3.

Step 2 — find the probability of landing in this region, using the true λ = 6:

X ~ Po(6): P(X ≤ 3) = 0.1512

This probability — P(in critical region | true parameter) — is the power.

Power = 0.151 (3 s.f.)

Notice this power is low! A power of only 0.151 means this test only has a 15% chance of correctly spotting that λ really dropped to 6 — it's much more likely (85%) to make a Type II error and miss the change entirely. This is exactly why, in the workflow, you always want to check power isn't too small: a "significant" test with tiny power is not very trustworthy at detecting real effects.

Practice Question

A test uses X ~ B(25, p), H₀: p = 0.4, H₁: p < 0.4, at the 10% significance level. The critical region is found to be X ≤ 6. Given that the true value is actually p = 0.25, and P(X ≤ 6 | p = 0.25) = 0.5611, find the power of the test and state what this tells you about the test's ability to detect the change.

4. Power Functions

What is a power function?

So far, every power calculation has needed a specific numeric "actual" value of the parameter (e.g. p = 0.25, or λ = 6). But what if you want to know the power for any possible true value, without being told a specific one? That's what a power function gives you — power written as an algebraic expression in terms of p (or λ), rather than a single number.

Formula Power function = P(in critical region | population parameter is p)

Or for Poisson: P(in critical region | population parameter is λ). You keep the parameter as a letter instead of substituting in a number.

In real life you'd almost never actually know the true parameter (if you did, why would you be running a hypothesis test on it?). Power functions sidestep that problem entirely — they describe how powerful your test would be across every possible true value, which you can then explore by substituting in different values, or by plotting a graph.

Worked Example — Building a power function

If the critical region is X ≤ 2 for a binomial hypothesis test with n = 50, find the power function.

The power function is P(in critical region | parameter is p), so let X ~ B(50, p):

P(X ≤ 2 | p) = (50C0)p⁰(1−p)⁵⁰ + (50C1)p¹(1−p)⁴⁹ + (50C2)p²(1−p)⁴⁸

= (1−p)⁵⁰ + 50p(1−p)⁴⁹ + 1225p²(1−p)⁴⁸

Factorise out (1−p)⁴⁸ and collect like terms:

Power function = (1 − p)⁴⁸ (1 + 48p + 1176p²)

What can you actually DO with a power function?

Plot it against p (or λ) to see visually where power is highest.
Substitute in different values of p (or λ) to compare two or more different hypothesis tests — the better test is the one with higher power at the values you care about.
Check where power > 0.5 — the region of true parameter values where your test is more likely than not to reach the correct conclusion (and less likely to give a Type II error).

Worked Example — Using a power function

Residents suspect the number of accidents on a main road has decreased. They test, at a 5% significance level, H₀: λ = 9 and H₁: λ < 9.

(a) Show that the power function is (1/6)e⁻ᵂ(a + bλ + cλ² + dλ³), where a, b, c, d are integers to be found.

Step 1 — Find the critical region, assuming λ = 9. X ~ Po(9): P(X ≤ 4) = 0.0550 > 0.05; P(X ≤ 3) = 0.0212 < 0.05. So critical region is X ≤ 3.

Step 2 — Now treat λ as unknown (H₀ might be false). The power function is P(X ≤ 3 | λ), for X ~ Po(λ):

P(X ≤ 3) = P(X=0)+P(X=1)+P(X=2)+P(X=3) = e⁻ᵂ + e⁻ᵂλ + e⁻ᵂλ²/2 + e⁻ᵂλ³/6

= e⁻ᵂ(1 + λ + λ²/2 + λ³/6) = (1/6)e⁻ᵂ(6 + 6λ + 3λ² + λ³)

Comparing to the given form: a = 6, b = 6, c = 3, d = 1

(b) Find the largest integer value of λ for which P(Type II error) < 20%.

Recall Power = 1 − P(Type II error), so we need Power > 0.8.

Substitute integer values of λ into the power function:

λ	Power	P(Type II error)
1	0.9810	0.0189
2	0.8571	0.1428
3	0.6472	0.3527

We need the largest λ where P(Type II error) < 0.2 — that's satisfied at λ = 2 (0.1428 < 0.2) but fails at λ = 3 (0.3527 > 0.2).

λ = 2

Why does power shrink as λ gets closer to 9? Makes total sense if you think about it: λ = 1 is a HUGE drop from the hypothesised λ = 9, so it's very easy for the test to spot (power close to 1). But λ = 3 is a much smaller, subtler drop — harder to distinguish from "just random noise around 9," so the test is far less likely to catch it (power drops). The further the true value is from H₀, the easier it is to detect, and the higher the power.

Practice Question

Explain, using the idea of a power function, why a hypothesis test is generally better at detecting a large change in a parameter than a small change, even when the sample size and significance level stay the same.

What to Memorise

Type I Error

Rejecting H₀ when it was actually true. "False positive." P(Type I error) = P(in critical region | H₀ true).

Type II Error

Not rejecting H₀ when it was actually false. "False negative." P(Type II error) = P(not in critical region | actual parameter).

Size

P(Type I error). For continuous distributions = α exactly. For discrete = actual significance level ≤ α. Smaller size is better.

Power

P(in critical region | H₀ false, given actual parameter) = 1 − P(Type II error). Bigger power is better; ideally > 0.5.

Power Function

Power written algebraically in terms of p or λ, instead of a specific number — lets you explore power across all possible true values.

The Trade-off

Reducing α lowers P(Type I error) but raises P(Type II error), and vice versa. Only increasing sample size reduces both.

Which parameter to use

Type I error / Size → use the H₀ (hypothesised) parameter. Type II error / Power → use the actual/true (given) parameter.

Continuous vs Discrete

Continuous (Normal, χ²): P(Type I error) = α exactly, no calculation needed. Discrete (Binomial, Poisson, Geometric): must calculate the actual achieved probability, which is ≤ α.

Concepts Checklist

I can describe the four possible outcomes of a hypothesis test using a 2×2 table. I can define a Type I error in my own words and give a real-world example. I can define a Type II error in my own words and give a real-world example. I know P(Type I error) uses the H₀ (hypothesised) parameter value. I know P(Type II error) requires being given the actual/true parameter value. I understand why continuous distributions give P(Type I error) = α exactly, but discrete ones don't. I can calculate P(Type I error) and P(Type II error) for a Binomial or Poisson hypothesis test. I understand the trade-off between reducing Type I vs Type II errors, and why sample size is the only fix for both. I know that Size = P(Type I error), exactly the same calculation. I know that Power = 1 − P(Type II error), and can explain why. I can calculate the power of a test given the true parameter value. I understand what a power function is and why it's more general than a single power value. I can derive a power function algebraically for a Binomial or Poisson test. I can use a power function to find the largest/smallest integer parameter value satisfying a power or Type II error condition. I understand why power is higher when the true value is further from the H₀ value.

Exam Tips

Trap #1 — Using the wrong parameter By far the most common mistake: students calculate P(Type II error) or power using the H₀ value of p or λ instead of the actual/given value. Always double check — did the question give you a "true" or "actual" value? That's the one to substitute into your distribution, NOT the H₀ value.

Trap #2 — Forgetting to find the critical region first To calculate P(Type II error) or power, you always need the critical region first, and that critical region is always built using the H₀ value. Only after you have the critical region do you switch to the true parameter value to calculate the probability of landing in it.

Trap #3 — Assuming discrete tests hit α exactly Never assume P(Type I error) = α for Binomial, Poisson, or Geometric tests. Always calculate the actual probability of the critical region you found — it will be ≤ α, but rarely exactly equal.

Trap #4 — Confusing "size" and "power" language Size is about H₀ being true (bad outcome = rejecting it). Power is about H₀ being false (good outcome = rejecting it). If a question says "the test correctly detected a real change," that's about power. If it says "the test wrongly signalled a change that wasn't there," that's about size/Type I error.

Trap #5 — Power function algebra errors When deriving a power function, examiners want to see clean factorisation (e.g. factoring out (1−p)ⁿ⁻ᵏ or e⁻λ) rather than a messy unsimplified expression. Always double-check your final expression matches the form given in the question (e.g. matching a·λ⁰ + b·λ¹ + ... to identify integer coefficients) — this is usually how marks are awarded.

What examiners are really testing This topic isn't really about the calculations (which are often just Binomial/Poisson probability lookups) — it's about whether you understand which conditional probability you're being asked for, and which parameter value belongs in it. Read every question slowly, identify whether it wants P(Type I), P(Type II), size, or power, and note down "H₀ value" or "actual value" before you touch your calculator.

What's inside

📖 Revision notes ✦ AI flashcards ✓ Instant AI marking

Read the full Quality of Tests notes free

That's the preview — create a free account to read the rest, plus flashcards and practice questions with instant AI marking. No credit card.

Unlock the full notes free →