Social desirability bias, a tendency of survey respondents to answer questions in a manner that will be viewed more favorably by others, is well-documented, but the latest findings from PEW Research comparing Web and phone survey results show that this tendency can result in double-digit discrepancies. The reporting differences that arise due to the mode of survey administration are known as mode effects; the impact of these effects differ with the questions being asked.
The divide between phone and Web surveys is, in some places, staggering: phone respondents report being satisfied with family 18 percentage points higher than those responding via a Web survey and 14% more of phone respondents said that “there is a lot of discrimination against gays and lesbians” than did their Web counterparts, who had no pressure to make the socially “correct” response. Another notable difference is the reporting of negative opinions about political figures in one’s opposite political party. Even here, respondents on phone surveys show a polite tendency to complain less about “the enemy.” Republicans reported more positive opinions about Democrats in phone surveys than in Web surveys and Democrats followed suit, reporting more positive opinions about Republicans in phone compared to Web surveys.
National Research Center’s own studies show that even with questions asking residents to evaluate community services or quality of life, responses by phone average 7% to 15% more positive than do responses about the same community characteristics asked by Web or mail.
This raises several questions about the reliability of surveys conducted over the phone. It is hard to gauge true behavior or sentiment on important issues via phone. Lower-income Americans are 12 points more likely on the Web than on the phone to report that finding enough money for food is an issue. Phone interviews result in respondents downplaying issues that are crucial to know for service planning or social programs. This well documented self-censorship exhibited during phone interview surveys leaves self-administered surveys (Web or mail) as the best bet.