5 Reasons Not to Monkey Around with Citizen Surveys

-By Tom Miller-

Most local governments start out with the right idea to do a community assessment. They want to get a finger on the pulse of their community by conducting a survey of their citizens, and they want to spend taxpayer dollars wisely.

However, instead of contracting the work out to professionals, some choose to develop a survey “in-house,” run it by a few colleagues, and post it online for the public to respond, using a free program to keep costs down. Seems simple enough. But unfortunately, the ultimate consequences of cutting the corners can be devastating. Too often, this approach leads governments to mistaken conclusions and misguided action. Here’s why:

1. It’s tough to ask the right questions.

Let’s say you’re conducting a general community survey. Where do you start? Do you ask citizens to rate each department of local government? Individual services provided by the jurisdiction? Which services should you include? It’s difficult to get a full picture of a community without overwhelming respondents with too many questions or asking questions residents can’t answer. For policy surveys, it can be challenging to decide what the survey should focus on in order to get the most useful and actionable data. For example, if residents rate community safety as highly important, it’s not clear whether more resources are needed in that area. Decision-makers also need to know how residents rate their sense of safety in their community and whether they would support additional funding for safety services.

National Research Center, Inc. (NRC) has developed a comprehensive bank of questions jurisdictions can draw from as they develop their survey. These questions tap topics that are relevant for both citizens and local governments. Each question is word-smithed to make results easy to unpack and put into action. Choosing from questions prepared by the pros can save governments a lot of time and headache. Predesigned queries also make for easier comparison to other communities that have asked the same questions.

2. Questions must be asked the right way.

As any good lawyer knows, the way you ask a question has a great deal of influence on how a person answers. Asking citizens whether they would support a tax increase to fund recreation projects is quite different from asking whether they would support or oppose a sales tax increase of .05 percent for one year to fund a new community swimming pool. Asking a clear, balanced question and being specific about the action you propose packs a bigger predictive punch than overly vague (or overly technical) language.

The response options citizens can choose between on the survey matter are similarly significant. Are response options slanted toward the positive? Are residents able to say “don’t know” when they don’t have an opinion or enough information to weigh in? The beauty of working with a professional survey firm is that questions with different wording and response options have been tried, tested, trashed or revised enough times that we know what works and what doesn’t. By collaborating with folks who have experience constructing surveys, you’re more likely to ask citizens the right questions the right way.

3. Survey responses need to represent the truth about your community.

Even among residents with Internet access, some will opt to take your survey and some won’t. In our experience, homeowners, women, residents who identify as White, and middle-aged residents are more likely to respond to surveys than are other members of the community. In other words, the “sample” of voices you get on a survey posted to your jurisdiction’s website or advertised in your community newsletter is not going to represent your community accurately. Like a funhouse mirror, the opinions of certain groups will appear bigger or smaller than they truly are. In a worst-case scenario, you may end up pouring resources into a project supported only by a handful of residents who were overrepresented in the sample. In our work, we use the best methods to make sure the characteristics of the entire community are reflected as accurately as possible. By selecting respondents at random, “oversampling” certain groups, and adjusting results, we ensure that sample characteristics match more closely with Census data.

4. Statistical adjustments and analyses require an experienced professional.

It’s easy to get lost in the numbers when results of a citizen survey come in. What if a resident chooses multiple responses on a question begging a single answer? How do you make sense of hundreds of written responses to open-ended questions? How do you correct for duplicate responses? Data can be messy, and haphazard efforts to “clean it up” can lead to disaster. Even with a pristine dataset, it can be difficult to know how to make sense of the results. Often, our clients want to know more than what’s on the surface. They want to know if there were differences of opinion between residents of different geographic areas, how their community changed over time, and how their results stack up against other jurisdictions. Digging deeper into the data takes the hand of a survey pro who knows how to deal with messy results, conduct the right analyses to answer tough questions, and present the results in a way that’s easy to understand and put into action.

5. It’s always better to have an outside perspective.

Keeping a survey locked up in-house is often a recipe for groupthink or, conversely, an intractable disagreement. From developing the right survey questions to interpreting results and finding feasible, creative ways to use them, involving an outside professional can be a big help. You are the best expert on your community, but an expert may raise important points about improving survey methods or making sense of complicated data that aren’t readily apparent.

The truth is, developing and administering a scientific survey of your residents that yields reliable, actionable data is not as easy as "monkey see, monkey do." Our job is to bring an experienced, supportive, and knowledgeable voice to the process of conducting a community survey. Our professional services and guidance allows local governments to get the real story about their residents.  We'll take care of the survey, so you can focus on applying the results to improve your community.

Related Articles

• Top Ten Reasons to Conduct The National Citizen Survey™
• How Citizen Survey Data is Analyzed
• Does Copyright Apply to Surveys?