From the census to political parties, social issues and business analysis, taking a survey is a tried-and-true method for finding out what people think.
Or is it?
“My wife chuckles when those calls come in,” said Dave Dillman, who teaches marketing and is the Distributive Education Clubs of America adviser for students interested in business careers at Durango High School. “I turn a 10-minute survey into a 30-minute conversation. A certain number are really legitimate and unbiased, others are push-pull, starting with the answer they want, and then writing the question to get it.”
In recent months, La Plata County residents have been relentlessly polled for political campaigns and surveyed for large capital projects on the horizon such as the Durango-La Plata County Airport Master Plan and the science, theatre, education, art and music facility known as the STEAM Park.
What should a respondent beware of when taking a survey, and how much should we rely on the results?
The quality of a survey’s results is dependent on several factors, said Richard Miller, executive director of the Office of Institutional Research, Planning and Assessment at Fort Lewis College.
“The first thing you look for is reliability, ‘Can I expect to get the same results twice?’” he said. “You can determine that by either asking a new group or mixing up the response items in a different order.”
The second, he said, is validity.
“Am I really measuring what I think I’m measuring?” Miller said. “A lot of political questions lead by using words people really like or words people really dislike, which makes you question the validity of the results.”
The third is representativeness.
“You can say you surveyed 100 people, but if they’re all members of the National Rifle Association, the response pattern represents the NRA, not the broader population,” he said. “In elections, they often use surveys not to observe but to promote.”
‘Don’t hang up’
Miller’s single biggest suggestion to people who receive a call for a survey?
“Don’t hang up if it’s a pro-life organization and you’re a supporter of Planned Parenthood,” he said. “If you do, you’ve just given up your right for your opinion to be represented in their survey.”
No matter what the survey is studying, you have to pay attention to how the question is asked.
“The more adjectives you see, the more likely the question is to be leading,” Miller said. “It’s possible to be positive without necessarily being leading.”
Dillman spends time with his students teaching them how to write surveys and conduct market research to help local businesses solve problems.
“‘Where do you eat breakfast?’ is so open, they may get 50 responses if they ask 100 people,” he said. “Students will often times not realize there is bias contained in that first version of the question. After I show such a question to the whole class and discuss bias, they will then realize that the question assumes the person eats breakfast. Someone who is less fortunate and doesn’t have the time or money to eat breakfast may ‘make up’ the answer, not wanting to admit he or she cannot afford or does not take the time to eat a healthy breakfast.”
Short and sweet
Miller reviewed some questionnaires that are, or have been, making the rounds in Durango in recent months.
“I’m not going to rip them apart,” he said, “because I always give them the benefit of the doubt that they honestly want the information.”
But he did have some thoughts on how they could be better. The survey being done for the feasibility study regarding the STEAM Park, for example, at four pages is probably too long to get anyone who’s not strongly for or against the project to take it.
“If it’s longer, it’s less likely to represent a broad population,” he said. “If I don’t really care, I’m not going to take the time to fill it out. A two-page survey, most people will do. Both the Republicans and Democrats have really long surveys when they’re working on their platforms. You have to be a dedicated party member to fill them out, which is why they can be so polarizing.”
The STEAM Park survey also has a number of open-comment sections, which take time. And one question asks people to rank the benefits the STEAM Park would bring to the area, with one being the most important benefit.
“People are trained to think on a scale of one to five, with five being the best,” Miller said. “They do specify one is the best, but people don’t always read carefully, so they might be confused, which could affect the results.”
The city of Durango’s Virtual City Hall, which launched on the city’s website in August, informs residents about issues the city is currently facing and asks their opinions. Recent topics have included the possibility of charging $1 to ride the trolley, different options for improving the sustainability of the Recycling Center and the plan to go to voters to renew the half-cent sales tax for recreational projects.
The Virtual City Hall is meant to be more like a community meeting than a survey, said Sherri Dugdale, assistant to the city manager, who runs the site.
“One of the goals of this City Council is to increase civic involvement,” she said. “This is another way for people to tell us what they’re thinking.”
But results of Virtual City Hall questionnaires are presented in survey-result style, which can confuse people as to how reliable the numbers are. And the survey allows people to look at the results without answering the questions first.
“No one likes to be in the minority,” Miller said. “That can skew the results.”
He took a look at the Durango-La Plata County Airport Master Plan questions on Virtual City Hall, which asks for opinions on current service to determine the support for a possible bond issue for enlarging the facility to deal with growing need.
“This was written by someone who’s really excited about the project,” Miller said with a smile. “Some of these are several questions in one, and some are things I wouldn’t expect the general public to know, like asking how much they agree with: ‘The past eight years have seen strong growth in the region, and it will continue to grow at least a moderate pace in the future.’”
How much weight should City Council give the results of the Virtual City Hall airport survey?
“With only 23 responses, it’s definitely not representative,” Miller said. “It’s not an ‘Amen, brother, I’m there.’ I wouldn’t use it to make decisions. If they want to make it better, have them come see us.”
abutler@durangoherald.com
To learn more
Visit www.durangosteampark.org or www.durangogov.org/STEAM to take the STEAM Park survey. Virtual City Hall can be accessed at durangogov.org.
Wondering if you can ever trust a poll’s result again?
Richard Miller, executive director of the Office of Institutional Research, Planning and Assessment at Fort Lewis College, says polls by these organizations follow good survey practices and poll a true random sampling:
Cornell University Survey Research Institute
National Opinion Research Center
Pew Research Center
Gallup Inc.
Nielsen Media Research
Quinnipiac University Poll
