What Makes an Indicator?: Our Criteria

This website hosts an updated set of community indicators to describe Champaign County, track and analyze trend data, and engage community members. The categories and indicators on this site are based on the visioning process and goals laid out in Our Future. Here., and we’d like to take this opportunity to present the criteria we used in selecting each indicator in the new set.

Is it illustrative?

An indicator, basically, does what it says on the box: it indicates. It’s a small piece of information that tells a larger story.

In biology, there are indicator species. The presence or absence of an indicator species tells you not just about that species, but about the whole ecosystem. That’s because that species requires certain conditions to survive and thrive. Maybe it can only live in water within a narrow range of temperatures, or it might need a certain level of flow. It could be sensitive to a certain contaminant. It definitely needs a sufficient food supply, and it can’t be overwhelmed by predators. What all of this means is that, if you survey a stream and find this species thriving, you now also know about the temperature, flow, and water quality of that stream. If you survey the stream and find none of that organism, you likely won’t know exactly why, but you have some educated guesses to start from.

And that’s what we wanted out of an indicator: a small, illustrative detail that we can reliably extrapolate context from. Actually, we want many small details that we can reliably extrapolate lots of context from; that’s why indicators generally come in sets. The first criterion is whether the indicator is illustrative, or whether it tells you more than just a number. If that answer is yes, we can move on to the next question.

Is it quantifiable?

This is where a lot of great ideas can peter out. Some of the most important aspects of a community, some of what residents value most, are things that cannot be counted.

A good rule of thumb is that abstract concepts, like beauty, or community pride, are out. This doesn’t mean they’re unimportant, only that they don’t work for this particular project. Yes, it is important to many people that the community is beautiful, and yes, we can count certain things in the community that are beautiful, but beauty is too subjective to be a workable indicator. We can count acreage of restored prairie, or the number of parks, pieces of public art, or buildings by renowned architects, but we can’t aggregate all of that and defensibly say “Okay, that gives us an 8.2 out of 10 on beauty. That’s better than last year’s 7.8, but still short of our goal of 8.5.” If we did that and published our conclusion, we would probably spend a lot more time explaining – and defending – our methodology than encouraging action to beautify the community.

The same is true of community pride. We could count things like the number of t-shirts sold by the local Chambers of Commerce, attendance at local festivals, or even positive mentions on social media, but these would be imperfect measures at best. The alternative is qualitative research, like surveys and interviews, which are time-consuming and expensive to do once, let alone the many times it would take to establish trend data. Furthermore, we run into the same subjectivity problem as with beauty. We posit that it would be difficult for anyone to rate their level of community pride on a scale of one to 10, and even more difficult for us to conclusively interpret that data. For example, how you define your 8/10 in community pride might look a lot like how your neighbors define their 6/10 and 9/10.

Does that mean that beauty, community pride, and other abstracts are unimportant? Absolutely not. Does it mean that they’re not practical for this use? Yes. And so we moved forward with what’s quantifiable.

Is it timely? Is there recent data? Is there trend data?

When developing the indicator set, we didn’t just want to know about here, we also wanted to know about now. But at the same time, we needed to moderate our expectations on how recent our indicator data should be. Timeliness is limited by data collection, which is resource-intensive in both time and funding. So, unfortunately, we can’t expect to always have data that’s current as of last month, last fiscal quarter, or even last year. In developing this set, we tried to select indicators with new data released at least every five years. In this set, we have indicators at both ends of the spectrum: our water use per capita indicator, in the Sustainability category, is drawn from a United States Geological Survey dataset released every five years, while many of our other indicators, whose source is the U.S. Census Bureau’s American Community Survey 1-Year Estimates, will have new data annually.

Having data available from previous years was also important. The ability to analyze and react to trends is one of the major goals of this project, and that requires both new and old data. It’s nice to know that there were 256 mussels in a surveyed stream in 2014, but on its own, it’s not that informative. If you don’t know that there were 113 mussels in the same stream in 2012, and 87 in 2010, you can’t know whether 256 is a lot or not very many at all. It’s even better to know all that and that a few decades ago, there were over 500. This amount of data is not always available, but it’s always helpful. New datasets tell us how things are, while trend data helps us understand how things are going.

Is it aspirational?

We wanted a significant number of our indicators to be aspirational: to be associated with a goal or a trend of improvement. The direction of this goal (why more is better and fewer is worse, or vice versa) should be both intuitive and easy to agree upon. For each indicator, we needed to be able to expect setting goals and tracking progress to be relatively simple. This goes back to the importance of indicators being quantifiable. Let’s get away from mussels for awhile and talk education. One of the indicators we chose for that category was high school graduation rates in Champaign County. It’s a good indicator: the data is available, annual, local, and easy to fit into context. It’s also easy to understand as an aspiration. It’s pretty intuitive to consider high school graduation rates and know that higher is better and lower is worse, and that a higher rate this year than five years ago shows progress. Furthermore, it’s something everyone can agree on: it would be hard to argue that a higher high school graduation rate is a bad thing. There can, and should, be healthy discussion on how that higher rate could be achieved, who is most impacted by having or not having a diploma, or why a district’s rates are what they are. But that single fact, that one piece of data in and of itself, is a pretty clear “more is better” case. We needed the indicators to aptly reflect the goals and priorities of the community.

In some cases, the indicators are less directional and more simply descriptive, and that’s okay, too. Not every indicator in the set is aspirational, and, of course, once you get into the nitty-gritty of analysis, things almost always become more complicated than they seemed at the level of a simple overview. (For example, median age is obviously non-aspirational, and some indicators, like median income, defy birds’-eye-view discussion.) But we put this indicator set together not only to define and measure progress, but also to inform and describe.

Does it reflect our community’s goals and priorities?

This is where it can get complicated, and this is where Our Future. Here. came back in. While there are a lot of common indicators in indicator sets from around the country and around the world, no two communities’ indicator sets will be identical, because no two communities are identical. The indicator set should be informed by the goals, priorities, and values of the community it’s designed for.

In the case of Champaign County, the indicator set was informed by the goals set forth in Our Future. Here., and shaped by the four key criteria we outlined above: illustrative, quantifiable, timely, and aspirational. Our Future. Here. wanted Champaign County to be collaborative, prosperous, sustainable, supportive, and enriching. These goals, and their subgoals, were foundational in the development of our eight new categories and the indicators in each one.

We’re excited about these new indicators. We’re excited for upcoming data releases, so we can establish and analyze trends; for the Regional Dashboard and Information Clearinghouse, where we present even more data and resources; and for this blog, where we’ll keep you posted on site updates, go over how to read and interpret the data on this site and elsewhere, and keep data fun. We plan to post a new blog entry on the first Friday of every month. So check out the rest of the site, and stay tuned for updates!

Related Documents

Leave a Reply

Please be respectful. All fields are required, and all comments are subject to moderation. URLs are not allowed.