Seeking the Gold Standard of Survey Accuracy (Part II)

See part one here.

As I looked through jurisdictions’ surveys conducted by my own organization and interviewed Scott Keeter, director of survey research at PEW Research Center, I uncovered a number of examples that offer reasonable proof of the accuracy of surveys that have no link to voting outcomes.

Mr. Keeter pointed to a series of studies by PEW that demonstrated close concordance of national survey results based on reports of about one thousand respondents on their demographic characteristics, health status, receipt of government benefits and more – with results from tens of thousands of Americans collected in U.S. government surveys.

PEW findings also showed close connection between the characteristics of those who responded to surveys (these days only 9 percent, on average) and those who did not respond when comparing party affiliation, financial status and voting behavior, giving real confidence that those who do respond are largely like those who do not.

In our own research on hundreds of unique places across the country we found these examples of survey accuracy (to list only a few). They are examples of concurrent validity because they show how survey respondent opinions are linked to existing conditions to which we would expect them to be linked, if the opinions were accurate.

• In places without buses, residents give the lowest ratings to public transportation.
• Where the bus runs every five minutes until two a.m., is free and safe, residents give the highest rating for public transportation.
• Once a city builds a library or recreation center, ratings of these services spike (and sustain).
• Residents of poor cities in Southern India with turbid water that runs intermittently only from stand pipes on the street give ratings for water service lower than in any U.S. city.
• Ratings of snow removal plummet after an impossibly heavy winter storm.
• In a city with smelly open drainage swales, residents give low ratings to storm drainage.
• During The Great Recession in the U.S., residents from communities across the country gave lower ratings to their personal economic outlook than they gave before or after the recession.
• Where roads are mostly dirt, residents give lower ratings to streets than where roads are paved.
• Where communities have invested in the construction of road connections, residents rate traffic flow better than in communities where no such investments occur.
• In a community where the primary water source is polluted by mine tailings residents give very low ratings to water quality.
• Long time residents of communities favor population growth less than do newcomers.
• Wealthier residents of communities feel safer than do poorer residents.
• Renters generally are more supportive of property tax increases than are property owners.

When you receive the results of a survey, take a moment to assess whether resident opinions square with your understanding of the community. Do residents in sections of the city without recreation facilities give lower ratings to recreation opportunities than do residents near to parks and recreation centers? Do low recycling ratings fairly reflect the fact that the community offers no curbside recycling? Are public trust ratings in the dump, as expected, because the mayor was just sent to prison? Are quality of life ratings really high the year you receive a national award as one of the top 10 best places to live?

Even though response rates are low and falling, there remains plenty of evidence to support confidence in high quality local government survey results, and you should add a review of your own data, if doubt lingers.