[Editor’s note: Today, we introduce a new regular feature on the blog: Allison Martell will write “Margin of Error” once a month, looking at numbers and statistics in the news. Visit her website or follow her on Twitter.]

What's the real value of a university education? The answer's more complicated than you might think. Illustration by Graham F. Scott.

Welcome to Margin of Error. Each month, I will be picking apart some number or statistical problem in the news or on my mind. I will draw on my own modest knowledge of econometrics — the statistics of economics, lately being applied to all kinds of problems — and interview the occasional expert. If you spot a questionable number that you’d like me to pick apart, send a tip to marginoferror (at) this (dot) org.

Thanks to midterms, university students across the country are too busy to read this. That’s probably just as well, because I’m here to share a story that might upset them — the degrees they are cramming for probably aren’t worth as much as they think.

It’s no secret that people with university degrees, on average, make more money. If you’re interested in how much more, you might look up some census data, take the average wages for someone with a degree, and subtract the average wages of someone without a degree.

But if you are following along on the back of an envelope — aren’t you proactive! — I hope that you won’t assume that you’ve figured out the effect of a university degree on income. It doesn’t matter how many times we say that correlation does not imply causation — reporters everywhere keep falling down the rabbit hole. And it’s hard to blame them, with oversimplified press releases like this coming out of Statistics Canada.

To understand what is wrong with this simple calculation, it helps to imagine an ideal experiment that could tell us how much going to university raises one’s income. Let’s say we could take a large group of high school graduates, and randomly assign half of them to complete university. Travelling several decades into the future, we would ask them about their income, and compare the two groups.

The difference between doing this and using census data is the random assignment. In real life, the choice to attend university is far from random. It might have to do with your ability to pay tuition, parents’ education, work ethic, or intelligence. If students who are likely to make more money anyway are also more likely to go to university, then we cannot tell what we are measuring – the initial advantage that secures a place in university, or the impact of the university education itself.

Luckily, there are some alternatives to this impossible experiment. Using statistics, we can control for just about anything that we can observe. Tons of research has been done on education and income. Close to home, a paper published in August in the Canadian Journal of Economics, by Vincenzo Caponi and Miana Plesca, focused on data from Statistic Canada’s 1994 General Social Survey. (The official version is gated, but a draft is available for free.) This paper looks at 3,274 people, all high school grads aged 17-65, not students, and employed in 1994.

If we simply look at averages, in this survey, men with university degrees made 59 per cent more per hour than men with only high school diplomas. For women, the gap was 50 per cent. But for all the reasons we’ve discussed, the authors do not do this — they use control variables. With basic controls — immigration status, province, marital status, education, work experience, etc. — the university wage premium drops to about 42 per cent for both genders.

Then the authors use a fancy procedure called propensity score matching. This lets them compare workers with different levels of education that nonetheless have a similar background, taking into account, among other things, parents’ education and number of siblings. With matching, the university wage premium drops further, to about 35 per cent for men and 39 per cent for women.

Unfortunately, there is a bigger problem. What if university students posses some inherent intelligence and motivation — let’s call it ability — that is difficult to measure? Caponi and Plesca argue that ability is inherited, so they control for it by using parents’ education. Most studies from the US control for ability by using test scores. But it is hard to believe we could ever control for, for example, someone’s drive to land a high-paying job. If the parts of ability that are impossible to measure also help you get into university, than we’re still overestimating the impact of university.

That isn’t to say that a degree is useless. First of all, the premium is still large, even with controls, so we can guess that in a perfect experiment it would not disappear. University might also benefit students in ways that do not affect income. Increasing the number of university graduates in Canada might be good for our culture or economic growth.

But there’s something to be learned here, whether you are studying for midterms or not. All sorts of things can cause two numbers to be correlated. When you hear an argument about causality, you should always think about what has been left out.