What’s the Difference Between Probability and Nonprobability Sampling?

A quick guide to understanding two key types of surveys and how they affect your data. 

Any government organization conducting a survey should know the difference between two main kinds of surveying: probability and nonprobability.

A probability survey allows everyone in the community an equal chance of receiving a survey.

A nonprobability survey allows anyone the chance to participate in a survey, which can lead to an abundance or lack of responses from certain groups.

But what do the two different types mean for you and how do they affect your data? Here is a simple guide to understanding these two survey types and the benefits of each. 

Why Probability Surveys Are Important

The key difference is that probability surveys achieve greater representativeness. That means the results will better reflect your community as a whole, including often unheard groups.

Probability surveying is also considered the gold standard in survey science due to its rigorous methodology. 

"With probability surveying, each person has an equal chance of being selected to take the survey," said Erin Caldwell, Survey Research Principal at Polco.

For example, say there are 1000 residents in a community and you are sending out 100 surveys. Through random selection, each person equally has a 1 in 10 chance of being picked to take the survey. 

But there's more to this equation. Research shows certain types of people are more likely to respond to surveys than others.

Who do you think is more likely to fill out a community survey: a 21-year-old student who rents an apartment or a 65-year-old retired homeowner?

If you guessed the retired homeowner, you are right. Young people are notoriously difficult to reach through surveys. Renters are also less likely to respond. Yet the average American city is comprised of 20 to 25% people aged 18 to 35 years old. The U.S. Census shows 36% of people rent. It’s important to get their feedback for data that accurately reflects your community.

Because of this, researchers will send more surveys to groups that are less likely to respond to increase the chances of getting feedback among those groups. This strategy, called oversampling, captures more voices from underheard groups.

This is a huge benefit to probability surveying and is often what local governments are seeking more than anything else.

To recap, here are the pros of probability surveys.

Benefits of Probability Surveys

1. Gold standard, quality data – You will get the best community data through probability surveys. When local governments use data state plans or grant applications, they must feel confident in their data. The best way to achieve that confidence is through probability sampling.
   
   
2.  Greater representativeness – Getting feedback from underheard populations is essential to many local governments but so difficult to achieve. Probability sampling is one of the most effective ways local governments can get this information.
   
   

So Why Conduct Nonprobability Surveys at All?

Unlike probability surveys, nonprobability surveys are open to everyone. (At Polco, we refer to these as open participation surveys.) Often, nonprobability surveys are online-only and promoted through a government's communication channels and social media.

As mentioned, certain types of people are more likely to respond to surveys. This is the downside of nonprobability sampling; sometimes results do not represent an entire community.

You may wonder, "What’s the point of a nonprobability survey if it won’t reflect what my community thinks? Isn't that the point of surveying?"

There are many benefits to open participation surveys that still make them worthwhile.

Benefits of Nonprobability Surveys

1. Supplement probability data – Responses on nonprobability surveys can strengthen the data from probability surveys. They often confirm what you see on the probability survey and boost confidence in your data.
   
   Polco does offer auto-weighting, a survey technique that scientifically adjusts data to be more representative. Polco users can see their auto-weighted data directly next to their raw data. 
   
   
2. Easier, faster, and often cheaper to conduct – Because nonprobability surveys are often conducted online, you can receive results instantly. Fast feedback is helpful to local governments that want to get a quick pulse on any topic in their community.
   
   
3. Give everyone a chance to participate – Some people who didn’t receive a probability survey may feel left out. "If I get the survey, my neighbor may be sort of upset that they didn't also get one because they want to give their opinion, too," Caldwell said. "So the nice thing about open participation surveys is it gives everyone a chance to participate."
   
   
4. Capture low-hanging fruit and direct more outreach to underheard groups – If you know you can instantly get feedback from a certain demographic with a nonprobability survey, you can then direct more time and resources to get resident feedback from underheard groups.

Leveraging both types of surveys' strengths can deliver a comprehensive and well-informed understanding of your community's needs while giving everyone a seat at the table. A thoughtful combination of probability and nonprobability surveys can empower local governments to make impactful decisions based on quality data.

Interested in Conducting a Community Survey? 

With Polco, you have access to a range of surveys and tools to easily engage your residents for different local government needs. An in-house team of data and survey scientists conduct probability surveys for you, so you know your results are accurate, reliable, and reflect your city's entire demographics. You can also access a library of pre-built nonprobabilty surveys to gain quick feedback on any important topic. To learn more about how Polco can help you engage your community, please reach out to one of our experts. 

• Five Ways To Become a Data-driven Government
• Is Your Community Survey Data Representative?
• Top Ten Common Community Survey Mistakes to Avoid