Sample QC and PCA for RNA-seq

Before you start

You can run a basic DESeq2 analysis; see From Raw Counts to Differential Expression, and some basic R helps (R basics).
Any term new? The Glossary has it.

Learning objectives

By the end of this lesson you will be able to: explain why you transform counts before PCA, read a PCA plot to spot batch effects and outliers, describe what PC1 and PC2 represent, and use a sample-to-sample distance heatmap as a second check.

Check the samples before you test the genes

Differential expression assumes your samples are what their labels say they are. Sample quality control is the step where you confirm that. A few minutes of plotting can reveal an outlier sample, a batch effect, or two labels that got swapped at the bench, any of which would quietly corrupt every gene-level result downstream. The two workhorse checks are a PCA plot and a sample-to-sample distance heatmap.

1Transform the counts first

Raw counts are unsuitable for PCA because their variance grows with the mean: highly expressed genes show big swings just because they are big, and they would dominate the plot. DESeq2 fixes this with a variance stabilizing transform (vst), or rlog for small datasets. After the transform, a gene's spread no longer depends so strongly on how highly it is expressed, so PCA reflects genuine differences between samples rather than just the loudest genes.

vst vs rlogWhich transform

Both stabilize variance. vst is fast and the usual choice; rlog can behave better on very small or very uneven datasets but is slower. Either way, the transform is for QC and visualization only, the differential test still runs on raw counts.

2Read the PCA plot

PCA finds the directions in which your samples differ most and projects them onto a 2D plane. PC1 is the direction of largest variance and PC2 the next largest. The hope is simple: samples should cluster by biological condition. If instead they cluster by processing day or sequencing lane, you are looking at a batch effect. If one sample sits far from everything else, it is an outlier. And if a sample lands firmly in the wrong group, suspect a swapped label.

Good sign: the two conditions form separate clusters along PC1. Warning sign: the amber point sits far from both groups, a likely outlier to investigate before testing.

When the plot looks like the one above, with conditions separated and one stray point, you investigate the stray sample rather than deleting it on sight. Sometimes it is a genuine biological extreme; sometimes it is failed library prep or a mislabel. The point of QC is to ask the question early.

3The distance heatmap as a second check

A sample-to-sample distance heatmap computes how different every pair of samples is across all genes and draws the result as a grid, usually with clustering. Replicates of the same condition should look most similar to each other (small distances, a tight block on the diagonal). If a sample is closer to the wrong group than to its own replicates, that agrees with what PCA hinted at, and two checks pointing the same way is a strong signal.

What you see	Likely meaning	What to do
Clusters split by condition	Healthy design; biology is the dominant signal	Proceed to differential expression
Clusters split by batch / lane / day	Batch effect dominates	Model the batch (e.g. ~ batch + dex)
One sample far from all others	Outlier or failed prep	Investigate; consider removing if technical
A sample sits firmly in the wrong group	Possible swapped label	Trace the sample back to the metadata

One-line summary

Transform counts with vst (or rlog), run PCA, and hope samples cluster by condition. Clustering by batch means a batch effect; a lone far point means an outlier; a sample in the wrong group means a possible swapped label. Confirm with a distance heatmap.

A worked example

In DESeq2 the whole check is a few lines. You build the dataset, transform with vst, then call plotPCA, coloring points by the treatment column (dex in the airway dataset).

library(DESeq2)

# dds built from RAW counts, airway design ~ cell + dex
dds <- DESeqDataSetFromMatrix(countData = counts_raw,
                              colData   = sample_info,
                              design    = ~ cell + dex)

# variance stabilizing transform for QC / visualization only
vsd <- vst(dds, blind = TRUE)

# PCA colored by the treatment column; look for two clusters
plotPCA(vsd, intgroup = "dex")

# Second check: sample-to-sample distance heatmap
sampleDists <- dist(t(assay(vsd)))      # distances across all genes
mat <- as.matrix(sampleDists)
pheatmap::pheatmap(mat,
                   clustering_distance_rows = sampleDists,
                   clustering_distance_cols = sampleDists)

Check your understanding

Why transform counts with vst (or rlog) before running PCA?

Right. Raw count variance grows with the mean; the transform evens this out so PCA reflects real sample differences, not just the loudest genes.

On a PCA plot, what does PC1 represent?

Correct. PC1 is the axis along which samples differ most; PC2 captures the next largest amount of variance.

Your samples cluster by sequencing lane instead of by biological condition. What does this indicate?

Exactly. Clustering by a technical factor like lane signals a batch effect; record it and model it (for example ~ batch + dex).

One sample sits far from every other point on the PCA. What is the right first move?

Right. A lone far point may be a real extreme or a technical failure; the QC step exists so you ask that question early rather than later.

What does a sample-to-sample distance heatmap add on top of PCA?

Correct. The heatmap is a complementary check; when it agrees with PCA about an outlier or batch split, the signal is much stronger.

Next in Track 3