Stats You Can't Avoid: p-values, Distributions, and Effect Sizes

Why a biologist has to care about statistics

Biology is noisy. Two genetically identical cells given the identical treatment will not produce identical measurements. So when you see gene X go up 1.4-fold between treated and control, the real question is never "did it change?" (the numbers are always at least slightly different). The question is: could a difference this big have happened just by chance, given how noisy the measurements are? Statistics is the toolkit for answering that question honestly. Skip it, and you will confidently report noise as discovery.

Distributions: the shape of your data

A distribution describes how often each value occurs. It is the single most important concept here, because every statistical test assumes your data follow some distribution, and choosing the wrong one gives wrong answers. Three you will meet constantly:

• Normal (Gaussian, the "bell curve"). Symmetric, described by a mean and a standard deviation. Heights, measurement errors, and many log-transformed quantities are roughly normal. Many classic tests (the t-test) assume it.
• Binomial. Counts of successes out of a fixed number of yes/no trials, like how many of 50 sequencing reads carry the alternate allele. The foundation of variant calling.
• Negative binomial. The workhorse for RNA-seq counts. Read counts per gene are non-negative integers and are "overdispersed" (their variance is larger than a simple Poisson would predict because of biological variability between samples). This is exactly why DESeq2 and edgeR model counts with the negative binomial rather than assuming a normal bell curve, and why you should never run a plain t-test on raw counts.

The null hypothesis and the logic of a test

A hypothesis test does something slightly counterintuitive: it starts by assuming nothing interesting is happening. That assumption is the null hypothesis (often written H₀), for example "this gene's expression is the same in treated and control." You then ask: if the null were true, how surprising is the data I actually observed? If the data would be very unlikely under the null, you have grounds to reject the null in favor of the alternative ("the gene did change").

The number that quantifies "how surprising" is the p-value.

What a p-value actually is

Here is the precise definition, worth memorizing word for word:

A p-value is the probability of observing data at least as extreme as what you saw, assuming the null hypothesis is true.

A small p-value (say 0.001) means: if there were truly no effect, data this extreme would happen only 1 time in 1000. That is surprising enough that most fields agree to "reject the null." The conventional, and entirely arbitrary, threshold is 0.05, a convention from Ronald Fisher in the 1920s, not a law of nature.

See it for yourself: simulate the null

The clearest way to understand a p-value is to build the null world and look. Below, we assume a coin is fair (the null), flip it 100 times in thousands of imaginary experiments, and ask how often we would see a result as extreme as 64 heads. That fraction is the p-value. Press Run (the first run loads Python in your browser and takes a moment).

Change observed_heads to 52 and rerun: the p-value shoots up, because 52 heads is not surprising for a fair coin at all. Change it to 70 and it nearly vanishes. You are watching the definition of a p-value in action.

Effect size: the question the p-value ignores

A p-value tells you whether an effect is detectable. It says nothing about whether the effect is big enough to matter. With a huge sample, a trivially small difference can be wildly "significant"; with a tiny sample, a large, real effect can miss the 0.05 cutoff. That is why you must always report an effect size alongside the p-value.

Effect size	What it measures	Where you'll see it
Fold change / log2 fold change	How many times larger one group's value is than another's	RNA-seq, the x-axis of every volcano plot
Cohen's d	Difference between two means measured in standard deviations	Comparing continuous measurements between two groups
Odds ratio / relative risk	How much an exposure changes the odds of an outcome	GWAS, epidemiology, case-control studies
Correlation coefficient (r)	Strength and direction of a linear relationship	Comparing two continuous variables

This is why a volcano plot has two axes: effect size (log₂ fold change) on x and statistical significance (−log₁₀ p) on y. The genes you care about are large on both: a real, big change that is also unlikely to be noise. A gene with a tiny fold change but a microscopic p-value is statistically significant and biologically boring.

Sample size, power, and why small studies mislead

Statistical power is the probability that your study will detect a real effect if one exists. It rises with sample size and with effect size, and falls when the data are noisy. Underpowered studies (too few samples, common in biology where samples are expensive) have two nasty properties: they miss real effects (false negatives), and the "significant" hits they do report are inflated and often wrong in direction or magnitude, the "winner's curse." This is a major driver of the reproducibility crisis. Three biological replicates is a starting point for RNA-seq, not a guarantee of adequate power.

The mental checklist for any result

1. What is the null hypothesis? If you can't state it, you can't interpret the p-value.
2. What is the effect size, and is it biologically meaningful? Not just the p-value.
3. Is the test appropriate for the data's distribution? Counts are not normal; use count-aware methods.
4. How many tests were run? One p-value of 0.05 is one thing; 20,000 of them is the subject of the next lesson.

Check your understanding

Next in Track 5

The multiple-testing problem →