How the right statistical test is chosen

Four questions decide it, and none of them require you to know statistics first.

1. What kind of outcome? Numerical or categorical.
2. How many groups? One, two, or more than two.
3. Paired or independent? Same people twice, or different people.
4. Do the assumptions hold? Chiefly whether the numbers are roughly normally distributed, and whether the groups vary similarly.

The answers land on one test. Chavery asks these questions in plain language, checks the assumptions against your actual data, and names the alternative when one fails.

Why it matters

Test selection is where most dissertations go wrong, and it is almost always a mistake of classification rather than mathematics — treating paired data as independent, or an ordinal scale as numerical. Getting the four questions right gets the test right.

An example

Numerical outcome, two groups, independent, normally distributed → independent t-test.

Numerical outcome, two groups, independent, not normal → Mann–Whitney U.

Numerical outcome, same people twice, differences normal → paired t-test.

Numerical outcome, same people twice, not normal → Wilcoxon signed-rank.

Numerical outcome, three or more independent groups, normal → one-way ANOVA, then a post-hoc test.

Categorical outcome, two groups → chi-square, or Fisher's exact when the expected counts are small.

Common mistakes

• Choosing the test that gives the significant result.
• Running a t-test on three groups by comparing them two at a time, which inflates the false-positive rate.
• Assuming normality instead of checking it.

Read next

• Normality, and what to do when it fails — What the assumption actually means, how it is checked, and why failing it is not a disaster.
• Paired or independent — the question that changes the test — Are your two sets of numbers from the same people, or different people?
• Types of variable, and why the type decides the test — Numerical or categorical; and if categorical, ordered or not.