The Blueprints of a Proper Questionnaire

By David M. Schneer, Ph.D.

All too often, we hear about research efforts—DIY or otherwise—that did not yield helpful results. When digging into those efforts to identify what went wrong, we find a combination of contributing factors—perhaps the worst of which is poor questionnaire design. Nothing foreshadows the early death of a research project better than a crummy questionnaire.

And while some presume the questionnaire is the first phase of a project, nothing could be further from the truth.  Every marketing research project starts with a clear definition of the marketing and research objectives.

For example, the marketing objective may be to sell more Chardonnay wine and the research objective may be to determine which of 3 new label designs will best accomplish that.  For a tech company, the marketing objective may be to increase profitability for the next gen mobile phone while the research objective is to determine which subset of potential new features will maximize profitability.

Often this step is ignored or assumed, and that can lead to an unfortunate disconnect between the actionability of a study and the client’s decision-making process.

Crafting the questionnaire begins only after marketing and research objectives are aligned.  You notice we chose the word “craft” instead of “develop” or “write.”  That’s because questionnaire preparation is a craft—a seamless union between art and science. It is critical to have an experienced and objective researcher craft your questionnaire. Why?

  • We’ve developed the optimum survey methodology to answer your marketing and research objectives.
  • We thoughtfully craft the questionnaire so that it provides the raw data required for specific statistical analyses needed to address those objectives.
  • We go to great lengths to ensure the questionnaire is both valid and reliable, and respectful of the respondent experience.

So what’s the deal with reliability and validity? Outside statistical research, these two terms are often used interchangeably but they actually mean very different things. Reliability refers to consistency.  In the case of a questionnaire, reliability means the degree to which measurements provide consistent outcomes.  Validity, on the other hand, represents the degree to which a question or scale measures what it is intended to measure. A good way to remember this is the example of a clock. A clock measures “true” time, and does so continuously.  If the clock were to show the wrong time, we would say it is invalid.  If it were sometimes fast and sometimes slow, we would say it is unreliable.  It is possible, however, to have a measure that is highly reliable but of poor validity (e.g., a clock that is precisely 20 minutes fast consistently).

As relates to questionnaire development, reliability is a bit easier to deal with than validity.  To maximize reliability, it’s important the questionnaire includes easy-to-understand questions, clear instructions, and unambiguous scales. As for validity, the sequencing of questions is critical.  Except for the very first question asked, all questions are potentially biased by the questions that appear earlier in the survey. For this reason, care must be taken not to “tip off the witness” by inadvertently educating the respondent or by creating awareness as a result of questions asked.  For example. We’d not want to list brands and ask the respondent to indicate which they were aware of and later ask what their favorite brand is if determining the latter was a key study objective.

To make a questionnaire valid, care must be taken to ensure that the way a question is asked maps as closely as possible to what it is intended to measure.

At its best, a questionnaire is crafted in a perfectly clear, unambiguous manner, with carefully designed questions that yield results that are usable for their intended purpose.  At it’s worst, a questionnaire ignores all these critical components, and yields invalid or unreliable results.

So, don’t try this at home, folks. We seasoned researchers live and breathe this stuff. We have the experience required to ensure survey results are valid and reliable, and provide you the sound guidance you need to drive your critical business decisions.

Merrill Research, Experience You Can Count On