Research Findings: Traditional Mailed Surveys vs. Digital Data Collection

Although traditional mailed surveys have long been considered the gold standard for community surveying, our increasingly digital landscape (and ever-rising costs of mailed survey implementations) encouraged us to explore alternatives. Conducting surveys exclusively online is undoubtedly more attractive: it allows you to see responses sooner and in real time, reduces both costs and paper waste, and helps build a panel of respondents for future engagement opportunities. But can you adopt a digital-only data collection methodology without compromising the validity of your survey results? Our data scientists at National Research Center (Polco’s in-house research team) set out to answer that very question.

In short, the research we conducted supports transitioning to digital data collection, in which residents receive a physical, mailed postcard invitation but complete the survey online. Along with all the benefits listed above, our study showed remarkably few drawbacks with this methodology. While response rates were lower, we found that online-only data collection had no discernible effect on the respondent’s demographics or responses to survey questions when compared to traditional survey mailings. 

For those reasons, our standard methodology for The National Community Survey has shifted accordingly; rather than continuing to send hard copy surveys to randomly selected households, we now (by default) will mail postcards to randomly selected household directing recipients to take the survey online. Read on for more details about the research that lead us to that decision. 

Our Research

Methodology

From 2020 to 2022, Polco conducted 92 implementations of The National Community Survey (The NCS) with a split sample methodology. In each of the 92 communities, a portion of the households randomly selected to participate in the survey were contacted using traditional mailed survey outreach with three contacts: 

• The first contact was a prenotification postcard informing the household that they were invited to participate in the survey. This postcard noted that recipients could either wait to receive a mailed hard copy survey or go online immediately to complete the survey. 
• The second contact was a mailed survey packet, including a cover letter (that also had a link to the online survey), the 5-page printed questionnaire, and a postage-paid return envelope. 
• The third contact was a similar survey packet, containing a reminder cover letter (again with a link to the online survey), the 5-page questionnaire, and envelope. 

The other portion of the randomly selected households was never mailed a hard copy of the 5-page questionnaire; instead, they were contacted twice with mailed outreach inviting recipients to respond online:

• The first contact was a half-page postcard containing a short URL to the online survey.
• The second contact was a similar half-page reminder postcard. 

Polco’s National Research Center analyzed the results, focusing on three important questions about comparisons between the two groups: 

1. Were the response rates different?  
2. Were the respondent characteristics different?  
3. Were the opinions and behaviors of the respondents different? 

The split sample research methodology meant that we could safely assume that any differences observed between the two groups were due to the outreach methodology, and not due to differences in communities surveyed or time of year the surveys were administered.

Results

Response rates

As shown in the table below, the average response rate to the traditional mailed survey outreach (with the three contacts explained above) was 20%, with a range from 8% to 34%. For the mailed invitation to online survey (with two contacts), the average response rate was 12%, with a range from 4% to 26%.

This was not unexpected: We had anticipated a lower response rate for the latter group, as this group was only contacted two times compared to three times. Further, some additional effort was required by those recipients to complete the survey—after receiving the invitation, they had to find a computer, laptop, tablet, or phone and then type in the short URL or use a QR code to access the survey, rather than simply grabbing a pen to complete the hard copy questionnaire. We also knew that some recipients may be more comfortable using technology to complete the survey than others. 

Respondent demographics

In addition to response rate, we examined the demographic characteristics of the respondents to each survey outreach method. Because respondents from both groups came from the same communities, we also compared the demographics of the respondents to the demographics of all adult residents in the surveyed communities.

For this analysis, we took the average of the demographic profile of respondents to each survey implementation, so that each community would have an equal weight, and surveys with more respondents would not have more weight than surveys with fewer respondents. As shown in the table below (and as we typically see in most survey efforts), respondents to either survey method were more likely to be older, female, non-Hispanic White, homeowners, and living in single-family homes than the general population.

However, the differences between respondents in the two outreach methods were minimal and not statistically significant. Differences between the groups were no greater than +/- 3 percentage points. In general, the margin of error around results for these surveys are +/- 5 percentage points, and these results were well within that level of precision.

Survey responses

We also evaluated whether the opinions of those responding via each outreach method were similar. As with the demographics, we found both groups exhibited very similar results (with no statistically significant differences) in respondents’ ratings of community characteristics and government services. The table below shows the average ratings for a brief assortment of items from The NCS, although all items were examined. 

We also analyzed respondents’ self-reported participation in various civic and social activities. Again, the responses were similar between the two groups, with the largest observed differences of about +/- 3 percentage points (as shown in the next table). And again, these differences were not statistically significant.

Considerations

While The NCS was being administered with the split sample methodology, some households receiving the mailed invitation to the online survey voiced concern about having limited internet access or not being comfortable taking the survey online. To address these concerns, we strongly recommend including a phone number and email address on the survey invitations specifically for technical questions and assistance. For Polco-led implementations of The NCS, Polco will provide this technical support to residents, to alleviate any burden on government staff.

If you’d like to offer printed surveys to accommodate those with difficulty accessing the survey online, you could consider making hard copy surveys available for residents upon request, with staff individually mailing and manually entering the paper responses online afterward.

Finally, for those that prefer the traditional mailed survey approach, Polco will continue to offer and facilitate that mailing upon request; however, we will pass on the additional costs associated with that outreach. It’s important to note that at the time of this experiment, the traditional mailed survey recipients received two survey packets. Polco’s current traditional mailed survey methodology now only includes one mailed survey packet, for a total of two contacts (an initial postcard followed by a survey packet) for each recipient. This does result in a slightly lower response rate than discussed above—about 4 percentage points lower—but still 4 percentage points higher than the digital data collection method. 

We’d be happy to talk through these considerations, and any questions you may have about methodology, as you plan your next survey project.