Paper surveys were once the gold standard in survey research. Things are changing, but just like the satisfying static of a vinyl record, old formats like paper still offer value that newer methods cannot.
“We’ve tried a couple of different approaches and we’ve learned some things,” Polco Director of Survey Research Jade Arocha said.
Let’s take a look at past methodologies to show you how we arrived at where we are today, as well as where we plan on going in the future.
Death of the Dial Tone
The phone came along 5000 years after the invention of the written word. But the landline did not have the same lasting power as print. The survey industry shifted away from phone calls years ago for a number of reasons.
Community surveying requires that residents live in the place being analyzed. Area codes are no longer a reliable indicator of where a person lives since people can move states with their cell phones. Landlines are obsolete. And since most unknown phone numbers come from passionate strangers who desperately want you to extend your car warranty, people don’t answer their phones anyway.
The Problem With Paper Surveys
Unlike an area code, a home address does reveal where a person lives, which is why mail is suitable for surveying.
In thepast, Polco’s National Research Center sent each respondent from a random sample a postcard that notified the recipient that a survey packet was on its way. Then the packet would arrive, followed up with a second survey packet.
“That second wave increases response rate, but all of that paper, all of that postage, is expensive. It’s not environmentally sustainable,” Arocha said. “Then governments have to pay for the data entry team to enter all of those responses. It’s why paper-based surveying is becoming slightly outdated.”
So, why do we still use paper?
404: Online Issues
Polco started offering an online option years ago. All mail materials have a weblink for people who prefer a digital method. But we found that this method could not fully replace mail-in surveys.
We A/B tested the three-part mailing method (postcard, survey, then another survey) against an online-only option. For the online-only group, the randomly selected person would receive two postcards with weblinks to the assessment (they didn’t receive any paper survey packet in the mail).
Both groups were tested within the same community. “It’s not an apples to apples comparison unless you are trying both methods in the same jurisdiction,” Arocha said. Some communities may have better access to the internet, have an older demographic, contain more homeowners, etcetera, all of which can play a part in how many people are willing to interact online.
“It was really hard to predict how many people would respond to each method,” Arocha said. “It was overcomplicating the approach.”
The web-only version created another problem. We list a phone number on the mailing materials for people who have questions. For the online-only group, local governments were inundated with tech support inquiries, which wasn’t practical.
The major benefit is the online-only cost is significantly lower—no paper, no postage, no paid data entry team. But the response rate is also lower. In some places it was much, much lower. This is an issue when government leaders want a representative sample of their community.
Now we recommend a hybrid approach: one postcard with the weblink, followed by one paper survey packet.
“It’s the best of both worlds,” Arocha said. “It allows people to respond online, it lowers costs, it is more environmentally sustainable, because it is one packet instead of two, and it enhances equity for people who don't have internet access.”
It’s important to note that the paper vs. online-only methodology question only applies when representative random sampling is the goal. There are plenty of online-only surveys that serve a valuable function. Online often reaches younger people, which has long been a challenge in the survey industry.
At Polco, we encourage governments to form an online subscriber base, so organizations have a built-in pool of residents willing to offer feedback quickly. Digital methods also completely eliminate the need for a data entry team since answers are automatically collected online.
But the survey industry isn't ready to move completely online just yet. Plus, using both provides us more information to analyze. Polco’s goal is to compare both online and paper data sets to find out how disparate they are, and present clearer, more robust insights to local governments.
“All signs point to online surveying absolutely being the wave of the future,” Arocha said. “But we can confidently say we are not at the point where we can eliminate paper surveys. They certainly serve an important function and probably will continue to for a while.”
Learn more about Polco’s weighted, randomized surveys that allow you to compare your community with municipalities around the nation. Polco’s flagship products give government leaders a clear and representative view of their residents’ opinions and priorities.