Being careful about survey reports

Benito Teehankee
Managing for Society, Manila Times
April 12, 2022

H.G. Wells, the famed British author of The Time Machine and The War of the Worlds, predicted in 1903 that statistical thinking would one day be as necessary for efficient citizenship as the ability to read and write. I used to think this was a far-fetched idea. Why would ordinary people ever need statistical thinking? Isn’t this more necessary for government officials and business people? Why should ordinary people care about averages, probabilities, and margins of error?

The campaign season for the presidential elections proves how correct Wells was. Different survey outfits have bombarded the public on what proportion of the potential voters prefer this or that candidate.   All these reports use a lot of statistics. But unfortunately, the media interviews on the latest Pulse Asia survey held from March 17 to 21 on candidate preferences show misunderstandings about the logic of survey statistics.

In TV interviews of Pulse Asia’s Ronald Holmes by ANC and CNN, there were misconceptions about survey statistics that the public should be careful to avoid. Let’s start with the basics, focusing on the survey results for presidential candidates Marcos and Robredo. The survey posed the question: “Of the people on this list, whom would you vote for as PRESIDENT OF THE PHILIPPINES if the May 2022 elections were held today and they were the candidates?”  The survey reported that, among the 2,400 adult respondents, 56 percent selected Marcos, and 24 percent selected Robredo. The reported error margin is ± 2% at the 95% confidence. Error margins give the range of estimates we should use to account for errors due to random sampling.

What does this mean? 

Holmes explained that they used multi-stage random sampling. This sampling method allows Pulse Asia to use the percentages above as reasonable estimates of the percentages going for Marcos and Robredo, respectively, among all adults. However, such estimates are not exact; hence the error margin of ± 2%. The proper way to read the estimate would be to mention the range of the estimates, taking the error margins into account. So, the estimates would be Marcos, 54 to 58 percent, and Robredo, 22 to 26 percent.   By always using ranges when interpreting survey estimates, people can avoid assuming a level of exactness that isn’t there.    Unfortunately, both ANC and CNN displayed the percentage estimates without showing the range of estimates or the error margins.  

What about comparing percentages? There is a tendency to use error margins to comment on whether the difference in the estimated percentages signal a real difference in the population percentages. So, a difference of less than two percentage points in the above situation would be referred to as a “statistical tie” or “there is no statistically significant difference in the percentages”.    I have also heard interpretations where the error margin is used to evaluate percentage changes across surveys. However, using the error margin in this way would not be correct since the reported error margin applies to individual percentages and not to differences between percentages.  

It will take another column to explain the more appropriate interpretation of error margins. Still, I would advise against the use of the term “significant” in any public discussion of statistical estimates. The public quickly misinterprets the word to mean “important”, which is furthest from its statistical meaning. Estimates are just that, estimates. Their importance is not a statistical matter but depends more on their practical and ethical consequence.

Dr. Benito Teehankee is the Jose E. Cuisia Professor of Business Ethics and Head of the Business for Human Development Network at De La Salle University. Email: benito.teehankee@dlsu.edu.ph