Questionnaire - Merrill Research

5.6 Minute Read

As market research professionals for nearly four decades, we’ve seen it all. Especially the dirty little secrets of poor questionnaire design. Like what, you ask? Well, how about trying to stuff 10 pounds of questions into a five-pound questionnaire? Some call it scope-creep. In the software industry it is called bloatware. We call it a sin.

So, how do you draft a sinless questionnaire that yields valid and meaningful insights? Some of the best practices are highlighted in our previous blog entitled, The Blueprints of a Proper Questionnaire:

Identifying research objectives and the specific questions that need to be answered
Identifying the appropriate targets for the questionnaire. Remember GIGO (Garbage In, Garbage Out).
Developing a framework for and flow of the questionnaire
Finalizing a well designed, field-ready survey that addresses those objectives and questions.

Ah, but temptation lies within the questionnaire. That’s where we see lots of transgressions—among them, the 7 deadly sins that you should always avoid, and ideally never commit when designing, writing or fielding a questionnaire.

1. Greed: This very often occurs when two questions are asked at once, making it difficult for a respondent to just give one answer. An obvious example of this: What brand do you think leads for quality winemaking and value? These are two different questions. Who is the leader for quality winemaking? And who is the leader for value? There could be two different answers.

Don’t hem in your respondents by forcing them to make one choice. The results will be misleading, if not useless.

2. Pride: We all like to be the George Gallup of questionnaire construction. Maybe some are. But experience says that every questionnaire should be outlined, designed and written. Then edited. Then pre-tested (sometimes first among peers or colleagues). Then edited. Then launched to a small sub-set of your target audience (soft launch). Then possibly reedited. Then fully deployed.

So much time and money is invested into the research process that these final QA steps must not be overlooked. Failure to do so can result in questions that are not clear to respondents, or the logic of the questionnaire is incorrect.

3. Gluttony: Not every study needs thousands of respondents. Sometimes a couple of hundred will meet your needs. Make sure you aren’t spending money and incurring increased field time with an unnecessarily large sample size. Bigger is not always better.

Considerations: How many data segments need to be observed? What level of confidence do you want for statistical significance (well, that’s another blog)?

4. Sloth: This iniquity can often be seen in tracking studies. You’ve been asking the same questions for wave after wave after wave. The market may have changed and your learning needs may be different but you continue using the same questions every wave, because “that’s just how you’ve always done it.”

Yes, tracking data is critical to measure over time, but sometimes you need to tweak the questions to make them more current. An example may be in the hospitality industry where competitor brand lists may change significantly in just a year.

Try to look at the questionnaire with fresh eyes every wave and look for ways to improve or perhaps expand questioning. Don’t be lazy!

5. Lust: Be choosy about your partners. This is especially true regarding respondent sample. Not all sample providers source their panels the same way.

Avoid “river sampling”. That’s when respondents click on a survey banner or a social media ad. Then they’re asked several screening questions and will get channeled to a pertinent survey based on their answers. Like fishing in a river stream—you don’t know who you’ll catch and what their demographic/psychographic profile is.

6. Envy: A vice of desire. Do not copy other research company’s questions. Thou shall not steal!

7. Wrath: Suffice it to say, you could incur the anger of respondents with agonizingly long surveys and/or questions or response-sets that are poorly worded or irrelevant. Make the user experience as friendly as possible. Common wrath-worthy design flaws include asking several questions where each respondent has to rate a long list of attributes (more than 12)—essentially sending them to survey purgatory.

These are just the 7 major sins that we’ve identified. Have you seen any that we missed? If so, email us at info@jonathand149.sg-host.com and let us know.

As you can, see proper survey construction involves much more than creating questions. In these unprecedented times, valid and reliable information is more critical than ever to ensuring company success. We can help. We understand sinless survey design.

Merrill Research, Experience You Can Count On

Image Credit: Ape Lad

By David M. Schneer, Ph.D.

All too often, we hear about research efforts—DIY or otherwise—that did not yield helpful results. When digging into those efforts to identify what went wrong, we find a combination of contributing factors—perhaps the worst of which is poor questionnaire design. Nothing foreshadows the early death of a research project better than a crummy questionnaire.

And while some presume the questionnaire is the first phase of a project, nothing could be further from the truth. Every marketing research project starts with a clear definition of the marketing and research objectives.

For example, the marketing objective may be to sell more Chardonnay wine and the research objective may be to determine which of 3 new label designs will best accomplish that. For a tech company, the marketing objective may be to increase profitability for the next gen mobile phone while the research objective is to determine which subset of potential new features will maximize profitability.

Often this step is ignored or assumed, and that can lead to an unfortunate disconnect between the actionability of a study and the client’s decision-making process.

Crafting the questionnaire begins only after marketing and research objectives are aligned. You notice we chose the word “craft” instead of “develop” or “write.” That’s because questionnaire preparation is a craft—a seamless union between art and science. It is critical to have an experienced and objective researcher craft your questionnaire. Why?

We’ve developed the optimum survey methodology to answer your marketing and research objectives.
We thoughtfully craft the questionnaire so that it provides the raw data required for specific statistical analyses needed to address those objectives.
We go to great lengths to ensure the questionnaire is both valid and reliable, and respectful of the respondent experience.

So what’s the deal with reliability and validity? Outside statistical research, these two terms are often used interchangeably but they actually mean very different things. Reliability refers to consistency. In the case of a questionnaire, reliability means the degree to which measurements provide consistent outcomes. Validity, on the other hand, represents the degree to which a question or scale measures what it is intended to measure. A good way to remember this is the example of a clock. A clock measures “true” time, and does so continuously. If the clock were to show the wrong time, we would say it is invalid. If it were sometimes fast and sometimes slow, we would say it is unreliable. It is possible, however, to have a measure that is highly reliable but of poor validity (e.g., a clock that is precisely 20 minutes fast consistently).

As relates to questionnaire development, reliability is a bit easier to deal with than validity. To maximize reliability, it’s important the questionnaire includes easy-to-understand questions, clear instructions, and unambiguous scales. As for validity, the sequencing of questions is critical. Except for the very first question asked, all questions are potentially biased by the questions that appear earlier in the survey. For this reason, care must be taken not to “tip off the witness” by inadvertently educating the respondent or by creating awareness as a result of questions asked. For example. We’d not want to list brands and ask the respondent to indicate which they were aware of and later ask what their favorite brand is if determining the latter was a key study objective.

To make a questionnaire valid, care must be taken to ensure that the way a question is asked maps as closely as possible to what it is intended to measure.

At its best, a questionnaire is crafted in a perfectly clear, unambiguous manner, with carefully designed questions that yield results that are usable for their intended purpose. At it’s worst, a questionnaire ignores all these critical components, and yields invalid or unreliable results.

So, don’t try this at home, folks. We seasoned researchers live and breathe this stuff. We have the experience required to ensure survey results are valid and reliable, and provide you the sound guidance you need to drive your critical business decisions.

Merrill Research, Experience You Can Count On