Data Literacy: Testing for Statistically Significant Difference Between Estimates

It’s time for another data literacy blog post, and this month we’ll be discussing how to work with the concept of statistical significance. But to be clear from the outset, what we’ll be talking about is understanding statistical significance as a data user, rather than as a statistician: interpreting published figures, not calculating or publishing your own figures.

For starters, let’s establish what we mean by “statistical significance” for the purposes of this discussion. What we’ll be talking about is determining whether the difference between two estimates is statistically significant. This means, basically, whether the two estimates can be viewed as really different, or if the difference between them may just be due to some chance or statistical reason[1].

The U.S. Census Bureau offers free training presentations on a variety of topics, and one of them, “Using ACS Estimates and Margins of Error,” introduces the following formula as a basic way to test whether two estimates are statistically different from each other[2]:

|Est1 – Est2|
(√(MOEest12+MOEest22))

It’s much easier than it looks. Don’t worry – we’ll break it down and provide examples. But to make a long story short, if the number produced by this formula is greater than one, the difference between the two estimates is significant. If it’s less than one, it’s not[3].

To demonstrate how this formula works, we’re going to walk through two related examples: the estimated percentage of the population over 25 years of age with a bachelor’s degree in the years 2014 and 2015, and the estimated percentage of the population over 25 years of age with a graduate or professional degree in the years 2014 and 2015. (In both cases, these estimated percentages are from Champaign County. All the estimates we’re about to discuss are from the U.S. Census Bureau’s American Community Survey 1-Year Estimates datasets. Something else to keep in mind is that all ACS estimates are at a 90% confidence interval – meaning that it is 90% certain that the actual value lies within the range denoted by the margin of error.[4])

|Est1 – Est2|
(√(MOEest12+MOEest22))

|21.8 – 22.0|
(√(1.92+2.12))

|-0.2|
(√(3.61+4.41))

_0.2_
(√(8.02))

_0.2_
2.83196

= 0.07062

As we see here, the resulting figure is less than one, so the two estimates are not statistically different from one another. What this means is that, even though the two years’ estimates appear different, the difference is small enough and the margins of error are large enough that we can’t be sure that this apparent increase reflects an actual increase.

This really becomes relevant when we’re trying to describe trends: we can’t describe the years 2014-2015 as part of an increasing trend in the percentage of Champaign County residents aged 25 and over with bachelor’s degrees. All we can say is that, between 2014 and 2015, the percentage of the given population with a bachelor’s degree remained roughly the same.

Let’s take a look at a related statistic: the percentage of Champaign County residents aged 25 and over with a graduate or professional degree.

|Est1 – Est2|
(√(MOEest12+MOEest22))

|20.5 – 23.9|
(√(2.02+2.02))

|-3.4|
(√(4.0+4.0))

_3.4_
(√(8.0))

_3.4_
2.82843

= 1.20216

Unlike the result from the first example, 1.20216 is greater than one: this means that the difference between the two years’ estimates is statistically different. In other words, the increase from 20.5% to 23.9% isn’t due to statistical error or chance; it reflects an actual, albeit small, increase in the real percentage of the given population with a graduate or professional degree in Champaign County. If the differences between these and other previous or subsequent years are also significant, we can start discussing concrete trends.

With what types of data is this formula useful?

As we said earlier, this formula should only be used to test the statistical significance of the difference between estimates. Exact counts are exact counts – they don’t have margins of error, and you don’t need to test whether the pile of seven hats you counted today is really greater than the pile of five hats you counted yesterday. The question just isn’t relevant. (For more information about the difference between estimates and counts, check out our August data literacy post, about margins of error, and our December data literacy post, about the U.S. Census Bureau’s American Community Survey.)

Obviously, you’ll need margins of error to work with in order to carry out these calculations. Ideally, these will be available with the estimates in question.

What about the shortcut method?

You may also be aware of a bit of a shortcut: using the given margin of error for each estimate to calculate a possible range for that estimate, then comparing the ranges of two estimates to see if they overlap, with the assumption that estimates with overlapping ranges are not statistically different from each other, and estimates with non-overlapping ranges are.

We’re sorry to have to tell you this, but it doesn’t work like that. Let’s go through those two examples again with this method.

If the 2014 percentage of the population aged 25 and older with a bachelor’s degree in Champaign County is 21.8 +/- 1.9, we have a range of 19.9-23.7. Similarly, the same estimate from 2015, 22.0 +/- 2.1, has a range of 19.9-24.1. Clearly, these ranges overlap, and the difference between the estimates is not statistically significant.

If the 2014 percentage of the population aged 25 and older with a graduate or professional degree in Champaign County is 20.5 +/- 2.0, we have a range of 18.5-22.5. The same estimate from 2015, 23.9 +/- 2.0, has a range of 21.9-25.9. These ranges also overlap – but as we demonstrated above, the difference between the estimates is statistically significant.

What we can learn from this is that the assumption that all estimates with overlapping ranges are not statistically different from each other is pretty flawed.

The shortcut method does have its uses: if you’re asked a question about this in person, or if you otherwise really do not have time to do the full calculation, it can allow you to provide a guess. If the ranges don’t overlap, you can say that the estimates are probably, but not definitely, statistically different. If the ranges do overlap, the best thing to say is that it’s possible that the estimates are not statistically different – but that it’s also possible that they are. Be upfront about the fact that all you can do is hazard a guess. Even better, offer to follow up with the person who asked the question later, and get back to them with a solid answer after you have time to run the calculation. It’s better to give a correct answer a day later than a wrong answer immediately.

What does all of this mean about the data visualizations elsewhere on the site?

If you’ve spent time looking at different pages around this site, especially in the Indicators section, you’ll notice that many of the pages include charts and graphs plotting the year-to-year data. Some of these year-to-year data points are not significantly different from each other. In those cases, graphs may look like they show an upward or downward trend, but with a quick test of the data, it’s clear that, like in Example 1 above, we can’t actually assign that trend to those years.

Does that mean that these graphs are meaningless? No – no more so than the tables of data associated with them. Each data point is the estimate for its period of time. That much is solid. The only thing we can’t get concrete about is exactly what’s going on between one data point and the next.

Every graph on this website has an associated table of data with it, and all estimates include their MOEs, so you can run the formula for yourself and check the statistical difference between any years you’re interested in. And where we can’t assign a year-to-year trend, we’ve usually noted this in the analysis.

Thanks for sticking with us through 1,500 words of statistics! We hope this post cleared things up a little, and gave you one more skill for being an informed, savvy data user.

[1] U.S. Census Bureau, Sirius Fuller. (April 6, 2016). “Using ACS Estimates and Margins of Error.” (Retrieved 14 March 2017).

[2] Ibid.

[3] Ibid.

[4] U.S. Census Bureau; Census Glossary; “Confidence Interval (American Community Survey”; (Retrieved 23 March 2017).

Table sources: U.S. Census Bureau; American Community Survey, 2015 American Community Survey 1-Year Estimates, Table S1501; generated by CCRPC staff; using American FactFinder; <http://factfinder2.census.gov>; (19 September 2016). U.S. Census Bureau; American Community Survey; 2014 American Community Survey 1-Year Estimates, Table S1501; generated by CCRPC staff; using American FactFinder; (16 March 2016).

Related Documents

Leave a Reply

Please be respectful. All fields are required, and all comments are subject to moderation. URLs are not allowed.