The Multiple-Testing Problem

The problem, in one number

Recall what p < 0.05 means: if the null hypothesis is true, you will still get a "significant" result 5% of the time, purely by chance. That is the false-positive rate baked into the threshold. It is acceptable when you run one test. But genomics never runs one test.

A typical RNA-seq experiment tests every gene for differential expression, around 20,000 tests in one analysis. Suppose, in the worst case, that not a single gene truly changed. How many would still cross p < 0.05 by chance alone?

20,000 × 0.05 = 1,000 false positives. A thousand genes flagged "significant" from a dataset with zero real signal.

If you hand in that gene list, almost all of it could be noise, and you would have no way to tell the real hits from the garbage. That is the multiple-testing (or multiple-comparisons) problem. GWAS is even more extreme: millions of variants tested at once.

Two different things you might want to control

The fix depends on what kind of error you are willing to tolerate. There are two formal targets:

Target	What it controls	Use when
Family-wise error rate (FWER)	The probability of making even one false positive across all tests	A single false positive is costly: clinical decisions, confirmatory studies
False discovery rate (FDR)	The expected proportion of false positives among the hits you call significant	Exploratory genomics: you accept some false hits as long as most of your list is real

The distinction matters enormously. Controlling FWER is strict: it tries to avoid any false alarm at all, which makes it conservative and prone to missing real effects. Controlling FDR is more permissive and is the right tradeoff for discovery-stage genomics, where you want a gene list that is "mostly true" and will be validated later anyway.

Fix 1: Bonferroni (controls FWER)

The simplest correction. To keep the family-wise error rate at 0.05 across m tests, only call a result significant if its p-value is below 0.05 / m. For 20,000 genes, that threshold becomes 0.05 / 20,000 = 0.0000025. A gene now needs an extraordinarily small p-value to survive.

Bonferroni is correct, simple, and bulletproof, but very conservative: by guarding against even a single false positive, it throws away many genuine effects (false negatives). In GWAS the famous "genome-wide significance" threshold of p < 5×10⁻⁸ is a Bonferroni-style correction reflecting roughly a million independent common variants tested across the genome (it is a fixed community standard, not recalculated per study). For a 20,000-gene RNA-seq study, Bonferroni is usually too strict and you lose real biology.

Fix 2: Benjamini-Hochberg (controls FDR)

This is the genomics workhorse, the method behind the padj (DESeq2) and FDR (edgeR, limma) columns you will actually use. The Benjamini-Hochberg (BH) procedure (1995) controls the false discovery rate instead of the family-wise error rate, so it is far more powerful while still keeping your hit list honest.

The algorithm is surprisingly simple:

1. Sort all m p-values from smallest to largest. Give each a rank i (1, 2, 3, …).
2. For each, compute the "BH critical value" (i / m) × Q, where Q is your chosen FDR (say 0.05).
3. Find the largest rank i where the p-value is still ≤ its critical value.
4. Call that p-value and everything ranked below it significant.

The "adjusted p-value" (q-value-like number) reported in your results is this procedure expressed per gene, so you can simply filter padj < 0.05 and trust that your list is controlled at 5% FDR.

Implement Benjamini-Hochberg yourself

Below is a small set of 10 p-values. The cell first counts how many pass a naive 0.05 cutoff, then applies BH correction and counts how many survive. Watch how BH is stricter than the raw cutoff but far less brutal than Bonferroni would be. Press Run (first run loads Python in your browser).

Try editing the list: add twenty entries of 0.9 (null genes) and rerun. The naive count barely moves, but it should: those extra tests are exactly the multiple-testing burden that BH and Bonferroni account for and a raw cutoff ignores.

A free diagnostic: the p-value histogram

Before trusting any large set of p-values, plot their histogram. Under the null (nothing real), p-values are uniformly distributed, a flat histogram. Real signal shows up as a spike near zero on top of that flat background. Two warning shapes:

• A spike near 1, or a hill in the middle: something is wrong with your test or model (often a mis-specified or overly conservative test). FDR correction on these is meaningless.
• Perfectly flat with no spike at zero: you likely have no real signal, so a long "significant" list after correction should make you suspicious.

This one plot catches more analysis bugs than almost anything else, and you will learn to read it in the next lesson on honest visualization.

What to actually do

1. For genome-wide discovery (RNA-seq, methylation, screens): report results at an FDR threshold using Benjamini-Hochberg. Filter on padj / adjusted p-value, never the raw p-value.
2. When a single false positive is genuinely costly, use Bonferroni or another FWER method and accept the loss of power.
3. Decide the threshold and the correction method before looking at results, and report how many tests were performed.
4. Always glance at the p-value histogram first.

Check your understanding

Next in Track 5

Plots that tell the truth: visualization best practices →