Errors and power basics - Statistics AP Study Notes

Hypothesis testing involves making decisions about population parameters based on sample data, but these decisions can contain errors.

Type I Error (α): Rejecting a true null hypothesis H₀. This is a false positive — concluding there's an effect when none exists. The significance level α (commonly 0.05 or 0.01) represents P(Type I Error) = P(reject H₀ | H₀ is true). Lower α values reduce Type I errors but increase Type II errors.

Type II Error (β): Failing to reject a false null hypothesis. This is a false negative — missing a real effect. We denote β = P(Type II Error) = P(fail to reject H₀ | H₀ is false). Unlike α, β isn't directly controlled and depends on the true parameter value, sample size, and α level.

Power of a test: The probability of correctly rejecting a false H₀. Power = 1 - β = P(reject H₀ | H₀ is false). Higher power means better ability to detect real effects. Typical target power is 0.80 (80%).

Key relationships:

• As α ↑, β ↓ (and power ↑)
• As sample size n ↑, β ↓ (and power ↑)
• As the true parameter moves further from H₀, power ↑

Memory Aid: "Type I = Innocent convicted (wrongly reject truth); Type II = Guilty freed (fail to catch the lie)"

For proportions specifically, these errors relate to decisions about population proportion p based on sample proportion p̂. The standard error SE = √[p₀(1-p₀)/n] under H₀ determines test boundaries.

Medical Testing Analogy: Consider COVID-19 rapid tests. A Type I error occurs when the test shows positive (reject H₀: "no infection") for a healthy person — a false positive causing unnecessary isolation. A Type II error means testing negative when infected — a false negative allowing disease spread. The test's power is its ability to correctly identify infected individuals.

Quality Control Example: A factory producing light bulbs claims 95% meet standards (H₀: p = 0.95). Inspectors sample bulbs:

• Type I Error: Concluding the batch is substandard when it's actually fine → rejecting good batches, wasting money
• Type II Error: Approving a genuinely substandard batch → defective products reach customers
• Power: The inspector's ability to catch truly defective batches

Increasing α from 0.01 to 0.05 means inspectors reject marginal batches more readily (catching more bad batches = higher power) but also risk rejecting more good batches (more Type I errors).

Drug Trial Context: Testing if a new drug improves recovery rates beyond the standard 60% (H₀: p = 0.60):

• Type I Error (α = 0.05): Approving an ineffective drug 5% of the time → wasted resources, false hope
• Type II Error: Missing an effective treatment → patients denied benefits
• Increasing power: Use larger trials (n ↑) or look for drugs with substantially better results (effect size ↑)

Real-world trade-off: Legal systems prefer Type I errors (convicting innocents) to be rare, accepting more Type II errors (freeing guilty parties). This "innocent until proven guilty" principle uses low α values.