How public opinion polls can mislead

In the introduction we stressed the increased importance of critical thinking and doing your homework in the new chaotic world of information overload and bias.  We suggested asking the question: "compared to what" when you see information suggesting a trend is either good or bad.

Critical thinking also requires an understanding of how opinion polls can and often do mislead through non-neutral language, and by sequencing questions in a way that leads to a particular answer. Any complex and controversial issue has pros and cons that must be weighed.  If the pollster simply lines up questions that point out all the pros in advance of the key question, respondents naturally feel they should be consistent with the answers they've just given. If all the cons were pointed out first, the outcome will be different. 

Let's take a look at how this might work:

Approach A

1. Would you say that terrorist groups like al 'Qaeda, who answer to no government and want to destroy civil societies, pose a risk to Americans?  (No risk at all__  Very little risk__   Some risk__  Significant risk__  Very serious risk__  )
2. Do you think it is likely al 'Qaeda will try and carry out another attack like 9/11? (No chance at all__  Very little chance__  Some chance__ Likely__ Highly Likely__ )
3. The Geneva Convention outlines standards for treating enemy prisoners who have followed certain rules such as wearing uniforms while in battle. Do you agree Al 'Qaeda combatants captured in battle should be entitled the same rights as those who have followed the Geneva Convention rules?   Strongly disagree__ Disagree__ Neither Agree nor Disagree__ Agree___  Strongly Agree__
4. Do you agree that al 'Qaeda prisoners captured on the battlefield should have the same rights to a taxpayer funded trial as American citizens, or is a military trial sufficient?  (Strongly disagree__ Disagree__ Neither Agree nor Disagree__ Agree___ Strongly Agree__)

 Headline:   "Poll indicates 85% of Americans disagree with Supreme Court ruling giving enemy combatants right to trial."

 Approach B 

1. Do you think it is right to keep someone in jail for years without any chance to argue his innocence?
2. Do you agree that denying al 'Qaeda prisoners in Guantanamo the right to a civilian trial fuels anti-American hatred around the world?
3. Do you agree that anti-American hatred puts the country at greater risk?
4. Do you agree that the U.S. should stand as a high example of the rule of law and apply it even when it is not expedient?
5. Do you agree al 'Qaeda prisoners captured on the battlefield should have the same rights to a taxpayer funded trial as American citizens?

Headline:  "Poll indicates 78% agree with Supreme Court ruling granting right to American trial for al 'Qaeda combatants."

For polls to be valid and useful, great pains must be taken to ensure neutral language and objective question sequencing. The necessary rigor is often not taken by organizations whose objective is to sell news or a particular narrative to their audience of choice.