By Angela Burtch, Vice President
5.6 Minute Read
As market research professionals for nearly four decades, we’ve seen it all. Especially the dirty little secrets of poor questionnaire design. Like what, you ask? Well, how about trying to stuff 10 pounds of questions into a five-pound questionnaire? Some call it scope-creep. In the software industry it is called bloatware. We call it a sin.
So, how do you draft a sinless questionnaire that yields valid and meaningful insights? Some of the best practices are highlighted in our previous blog entitled, The Blueprints of a Proper Questionnaire:
- Identifying research objectives and the specific questions that need to be answered
- Identifying the appropriate targets for the questionnaire. Remember GIGO (Garbage In, Garbage Out).
- Developing a framework for and flow of the questionnaire
- Finalizing a well designed, field-ready survey that addresses those objectives and questions.
Ah, but temptation lies within the questionnaire. That’s where we see lots of transgressions—among them, the 7 deadly sins that you should always avoid, and ideally never commit when designing, writing or fielding a questionnaire.
1. Greed: This very often occurs when two questions are asked at once, making it difficult for a respondent to just give one answer. An obvious example of this: What brand do you think leads for quality winemaking and value? These are two different questions. Who is the leader for quality winemaking? And who is the leader for value? There could be two different answers.
Don’t hem in your respondents by forcing them to make one choice. The results will be misleading, if not useless.
2. Pride: We all like to be the George Gallup of questionnaire construction. Maybe some are. But experience says that every questionnaire should be outlined, designed and written. Then edited. Then pre-tested (sometimes first among peers or colleagues). Then edited. Then launched to a small sub-set of your target audience (soft launch). Then possibly reedited. Then fully deployed.
So much time and money is invested into the research process that these final QA steps must not be overlooked. Failure to do so can result in questions that are not clear to respondents, or the logic of the questionnaire is incorrect.
3. Gluttony: Not every study needs thousands of respondents. Sometimes a couple of hundred will meet your needs. Make sure you aren’t spending money and incurring increased field time with an unnecessarily large sample size. Bigger is not always better.
Considerations: How many data segments need to be observed? What level of confidence do you want for statistical significance (well, that’s another blog)?
4. Sloth: This iniquity can often be seen in tracking studies. You’ve been asking the same questions for wave after wave after wave. The market may have changed and your learning needs may be different but you continue using the same questions every wave, because “that’s just how you’ve always done it.”
Yes, tracking data is critical to measure over time, but sometimes you need to tweak the questions to make them more current. An example may be in the hospitality industry where competitor brand lists may change significantly in just a year.
Try to look at the questionnaire with fresh eyes every wave and look for ways to improve or perhaps expand questioning. Don’t be lazy!
5. Lust: Be choosy about your partners. This is especially true regarding respondent sample. Not all sample providers source their panels the same way.
Avoid “river sampling”. That’s when respondents click on a survey banner or a social media ad. Then they’re asked several screening questions and will get channeled to a pertinent survey based on their answers. Like fishing in a river stream—you don’t know who you’ll catch and what their demographic/psychographic profile is.
6. Envy: A vice of desire. Do not copy other research company’s questions. Thou shall not steal!
7. Wrath: Suffice it to say, you could incur the anger of respondents with agonizingly long surveys and/or questions or response-sets that are poorly worded or irrelevant. Make the user experience as friendly as possible. Common wrath-worthy design flaws include asking several questions where each respondent has to rate a long list of attributes (more than 12)—essentially sending them to survey purgatory.
These are just the 7 major sins that we’ve identified. Have you seen any that we missed? If so, email us at firstname.lastname@example.org and let us know.
As you can, see proper survey construction involves much more than creating questions. In these unprecedented times, valid and reliable information is more critical than ever to ensuring company success. We can help. We understand sinless survey design.
Merrill Research, Experience You Can Count On