Research

How to Lather, Rinse, and Repeat Your Way to Clean Research Results

by

By Angela Burtch, Vice President, Merrill Research

4-Minute Read

In Part 1 of this series we touched on a few tips on how to ensure the integrity of data when conducting a market research study. 

A lot has changed in the past few years and we thought it was time to add more content. More bots, frauds, and other illegitimate means are increasingly common and able to circumvent the safety nets we’ve cast to detect poor data.  Everyone is working hard together to mitigate bad survey data.  So, with this blog, we’d like to continue to offer up more suggestions that will help all of us to this end.

Lather

Before you even see a respondent in a survey, make sure your sample provider has done all the pre-conditioning to ensure the integrity of the participant.  These are just a few of the myriad factors that impact respondent quality.  Your respondents need to be squeaky clean.  Here are some questions to ask your vendor:

  • How are they sourcing their sample? Proprietary panel?  APIs? if so which ones?  Affiliate publishers?  River/intercept sampling?  Some other way?  Make sure you know the source.
  • If panels are used, what tools are used in the registration process to mitigate fraud?  Digital fingerprinting? 3rd party validation? Double opt-in validation? IP validation?

We could write an entire series of blogs just on this first lather.  Go to the sample provider’s website and see what they say they do!

Rinse

Even when the legwork is done up-front by your sample provider to vet your respondents, as researchers we are responsible for ensuring that quality checks continue post-recruitment. 

So, it’s time to rinse and get rid of anything that wasn’t taken out in the suds by your sample vendor.  How do you do that?  Make sure your questionnaire can stand up to fraud. 

It used to be that we would sprinkle a “gotcha” question or two into the respondent screening to rinse out the duds.  Not anymore.  For a typical 10-minute survey we will include all types of trap questions starting in the screening section all the way through the main questionnaire and we use elaborate decision trees to decide what flags or completely terminates a respondent. 

Here are a few tips that we highly recommend you incorporate into your next questionnaire:

✔ Red Herring questions.  Most researchers know about these.  As an example, in a branding study you could include a “fake brand” in an aided awareness list and flag or terminate respondents who select this option. Be careful with this one.  Make sure you do your due diligence and make sure that your red herring brand is not too similar to a real-life brand.  For example, if the subject matter is awareness of social media sites, a question with an artificial brand “Tweeter” could potentially be selected because of its very close similarity to “Twitter”.  Always remember, you don’t want to entrap and confuse respondents, you just want to make sure that you’re rinsing away the unengaged and potentially bad data responses.

✔ Skill testing questions. An actual legal requirement in Canada for sweepstakes. A simple mathematical problem slows down respondents and adds another level of security. Don’t ask respondents to solve a differential calculus problem (unless they are engineers!), but instead, ask a simple problem like (10 x 10) – 5. BTW – that’s 95. Give yourself a pat on the back if you got it right. This may not work with AI created bots, but it can help detect human or click-farm respondents. Stay tuned for my next blog on that..

✔ Attention check questions.  These work really well to wash out speeders. In this instance, you tell a respondent to pick a specific response option.  For example: “We want to make sure you are taking your time to read all the survey questions, for this question please select ‘Not Very Familiar’ below.” If they didn’t follow the instruction, then what else were they speeding through? Rinse them out of your hair.

✔ Look at the open ends.  If you have any open-text questions in your survey, these often provide tell-tale indicators of junk you want to flush out.  Nonsense answers (FDSKALFDAKJFSA). Obvious multiple copy and pastes.  Or out-of-context answers (e.g., you are asking about car brands, and the responses are about bicycles).  Even if you didn’t budget time or expense for coding an open end, including one just for the purpose of monitoring data quality can be a good idea.  Example: “In a few words, please type below the kind of movies you like to watch.”  Limit the character count to 50 characters so you aren’t using valuable survey taking time, but you can still gauge respondent presence in the question.

✔ Just way too positive.  If in a battery of questions, everything is “Extremely Favorable”, “Extremely Important”, etc. Consider you might have an over-endorsing respondent seeking to please the research sponsor and stay in the respondent pool.  This is somewhat similar to straight lining, which we addressed in this previous blog.

✔ Speaking of straight lining answers, make it harder to do.  Instead of long boring batteries of attributes to rate on a scale, consider mixing up your question structure.  How about a drag and drop exercise asking respondents to sort the attributes by Like, Dislike, and Neutral emojis instead of having them cruise down a page clicking numbers.

✔ Look for contradictions. Do the survey answers contradict themselves or do they make sense? Often this means deciding to ditch a respondent on a case-by-case basis. For example, if someone answers that they drink beer once a month and subsequently answers that they buy beer daily, is this valid? Cross-check with some of the other tips listed above.

And Repeat

Yes, repeat.  Repeat with every prospective sample partner you are considering.  Repeat with every questionnaire you design. Lather, rinse, repeat.

Need help with designing research mousetraps? Let Merrill Research guide you through your next research study!

Back in the Box: Qualitative Observations from In-Person Interviews in a Pandemic

by

“Never take anything for granted.“

– Benjamin Disraeli

By David M. Schneer, Ph.D./CEO

4-Minute Read

Several years ago, I conducted well over 600 in-depth interviews, focus groups, clinics, and ethnographies; in just one year.

Annualized, that’s nearly two interviews per day. And it felt that way.

It was a herculean effort involving planes, trains, and automobiles. Exhausted, I began to wonder if I would ever want to conduct another interview.

Would I miss moderating? “No way,” I thought at the time.

Fast forward to 2021

But then the pandemic stole in-person interviewing.  But no more. After conducting virtual-only interviews during the past 18 months, I am back in a traditional focus group facility moderating room—affectionately referred to by us moderators as “the box.”

With a costly product launch looming, the client wanted to observe first-hand how respondents would react to stimulus. Would they lean in toward the stimulus indicating an affinity for it, or would they turn away? Would they show signs of disgust or contempt or true smiles? The box would reveal all.

Turns out, I missed the box. You see, my perspective pivoted in a way that may not have been possible were it not for the pandemic. Nothing compares to an invigorating and enlightening conversation with someone, simply communicating together at the most basic human level—face to face, unfiltered, unfettered, asynchronous and in real-time. I am grateful to be back in the box. And the box has an infrastructure. I was reunited with the focus group facility professionals who helped make the research successful. I missed them too.

But the box is not the same—and likely never will be. To conduct these interviews, we needed to ensure that our pandemic protocols were buttoned up.

Short of wearing a hazmat suit, we are working with facilities who have the most stringent safety standards. Here’s ten steps we took to ensure these interviews were conducted safely:

  1. The research facility followed state and county mandates/guidance regarding Covid-19 safety protocols.
  2. The facility required all research participants to be fully vaccinated and must present proof of said vaccination upon arrival at the facility.
  3. Front room personnel were typically limited to the moderator and respondent.
  4. I—the moderator—was seated at least 16 feet apart from the respondent with ventilation and hand sanitizer available.
  5. The study was designed to be “contactless”—that is, only the respondent touched the stimulus, which was sanitized after each session.
  6. I am vaccinated and so is my client who is attending this research.
  7. Like a journalism pool, only one client representative observes in person.
  8. Remote viewers will observe via live stream.
  9. The facility required masks for all front room staff and participants in open spaces.
  10. And, of course, there was no sharing of the M&Ms! Community snack bowls are gone for good.

During the pandemic and going forward, we will continue to utilize video heavily. Not every respondent, plague or otherwise, is available in person and so video is a powerful alternative for remote data collection when in-person is untenable. We are fortunate to live in a time where there is such a fruitful alternative to traditional qualitative research.  The remote methods available both kept research companies alive and ensured our clients didn’t fall behind in their business-critical insights efforts.

But as a moderator and nonverbal intelligence expert, I can tell you that in-person interviewing yields significantly more information than remote methods for those who can decipher the cues.

Indeed, how sweet it is to be back in the box!

Merrill Research—Experience You Can Count On!

Need to know how your customers and prospective customers will react to a new product, communications effort, or service? Contact Merrill Research Today!

7 Deadly Sins of Questionnaire Design: Part 2

by

By Angela Burtch, Vice President

5.6 Minute Read

As market research professionals for nearly four decades, we’ve seen it all. Especially the dirty little secrets of poor questionnaire design. Like what, you ask? Well, how about trying to stuff 10 pounds of questions into a five-pound questionnaire? Some call it scope-creep. In the software industry it is called bloatware. We call it a sin.

So, how do you draft a sinless questionnaire that yields valid and meaningful insights? Some of the best practices are highlighted in our previous blog entitled, The Blueprints of a Proper Questionnaire:

  • Identifying research objectives and the specific questions that need to be answered
  • Identifying the appropriate targets for the questionnaire. Remember GIGO (Garbage In, Garbage Out).
  • Developing a framework for and flow of the questionnaire
  • Finalizing a well designed, field-ready survey that addresses those objectives and questions.

 


Ah, but temptation lies within the questionnaire. That’s where we see lots of transgressions—among them, the 7 deadly sins that you should always avoid, and ideally never commit when designing, writing or fielding a questionnaire.

1. Greed: This very often occurs when two questions are asked at once, making it difficult for a respondent to just give one answer. An obvious example of this: What brand do you think leads for quality winemaking and value? These are two different questions. Who is the leader for quality winemaking? And who is the leader for value? There could be two different answers.

Don’t hem in your respondents by forcing them to make one choice. The results will be misleading, if not useless.

2. Pride: We all like to be the George Gallup of questionnaire construction. Maybe some are. But experience says that every questionnaire should be outlined, designed and written. Then edited. Then pre-tested (sometimes first among peers or colleagues). Then edited. Then launched to a small sub-set of your target audience (soft launch). Then possibly reedited. Then fully deployed.

So much time and money is invested into the research process that these final QA steps must not be overlooked. Failure to do so can result in questions that are not clear to respondents, or the logic of the questionnaire is incorrect.

3. Gluttony: Not every study needs thousands of respondents. Sometimes a couple of hundred will meet your needs. Make sure you aren’t spending money and incurring increased field time with an unnecessarily large sample size. Bigger is not always better.

Considerations: How many data segments need to be observed? What level of confidence do you want for statistical significance (well, that’s another blog)?

4. Sloth: This iniquity can often be seen in tracking studies. You’ve been asking the same questions for wave after wave after wave. The market may have changed and your learning needs may be different but you continue using the same questions every wave, because “that’s just how you’ve always done it.”

Yes, tracking data is critical to measure over time, but sometimes you need to tweak the questions to make them more current. An example may be in the hospitality industry where competitor brand lists may change significantly in just a year.

Try to look at the questionnaire with fresh eyes every wave and look for ways to improve or perhaps expand questioning. Don’t be lazy!

5. Lust: Be choosy about your partners. This is especially true regarding respondent sample. Not all sample providers source their panels the same way.

Avoid “river sampling”. That’s when respondents click on a survey banner or a social media ad. Then they’re asked several screening questions and will get channeled to a pertinent survey based on their answers. Like fishing in a river stream—you don’t know who you’ll catch and what their demographic/psychographic profile is.

6. Envy: A vice of desire. Do not copy other research company’s questions. Thou shall not steal!

7. Wrath: Suffice it to say, you could incur the anger of respondents with agonizingly long surveys and/or questions or response-sets that are poorly worded or irrelevant. Make the user experience as friendly as possible. Common wrath-worthy design flaws include asking several questions where each respondent has to rate a long list of attributes (more than 12)—essentially sending them to survey purgatory.

These are just the 7 major sins that we’ve identified. Have you seen any that we missed? If so, email us at info@jonathand149.sg-host.com and let us know.

As you can, see proper survey construction involves much more than creating questions. In these unprecedented times, valid and reliable information is more critical than ever to ensuring company success. We can help. We understand sinless survey design.

Merrill Research, Experience You Can Count On

 

Image Credit: Ape Lad

THROW THE BUM OUT!!! What do Baseball and Quality Data Have in Common? More Than You Think!

by

By Michael Rinck – Vice President

The Oakland A’s brand of Billy Ball brought to light the use of data as a competitive advantage for baseball teams. But for any data to be useful, it needs to be clean. We’re pretty sure cleaning research data doesn’t conjure up thoughts of baseball.  Probably not right off the bat, anyway (pun intended).  However, a deeper dive into the methods we employ to clean data are not unlike the myriad strategies used in the game of baseball itself [1].  We guard against stolen bases, throw curveballs, and strike out those who are less than honest with their responses.  And yes, occasionally we consult the instant replay. How, you might ask? Allow us to throw out the first pitch and explain:

THROW THE BUM OUT

Cleaning data is the quality control art of removing bad data that simply cannot be trusted.  Just as a manager will pull his pitcher if he can’t throw strikes or thinks he can’t, don’t be afraid to toss erratic data.

You’ve invested a lot of time and money in creating a survey.  You built it, and hoped they would come (okay sorry, reaching here). And you’ve invested a lot of time in scoping out sample and making sure you are targeting the right respondents for your research objectives.  You collect the data.  Now it’s time to step up to the plate and clean it—that is, remove bad or incomplete records.

Good data cleansing is not just about eliminating data but also ensuring its consistency. Data cleaning leads to high quality data the way good pitching leads to strikeouts. When data is of excellent quality, it can be easily processed and analyzed, leading to insights that help the organization make better decisions.

If you’re a baseball fan (and even if you’re not) here are 4 easy-to-remember and important tips for cleaning data:

  1. Stealing Bases: Speeding is considered problematic survey behavior because respondents are not providing thoughtful, accurate answers. Consequently, the data they provide may be of poor quality, and in turn, may have to be discarded so survey estimates are not adversely affected. One rule is that the length of interview needs to be at least 40% of the average length of the survey. Thus, if the average length is 10 minutes, we might cut off a speeder at 4 minutes. Of course, there are exceptions to the rule. For example, we need to adjust time requirements for skip patterns, open-ends, loading time for images, etc.
  2. A Curve Ball for Straight-Liners: We can see if a respondent straight-lines a series of rating scale questions. For example, if there are 20 statements being rated, we will check to see how many of the statements were given the same code.  We can run a distribution of the count of codes to calculate what percentage of straight-lining we will use to remove a respondent (i.e., terminate a respondent out if they straight-lined (same code) 18 of the 20 statements).
  3. Three Strikes and You’re Out: For example, in the screener we may ask: What brands have you consumed in the past 3 months?  And later in the survey we may have a follow up question detailing when exactly did they last consume that brand (with a scale, Last Week/Last Month/2-3 Months ago/6-12 months ago/Over a year ago)?  If the respondent answers longer than past 3 months, we will flag that respondent.  Depending on the survey, we may have up to 10 flags programmed for various questions.  A red herring question may also be included that would also count as a flag. An example of a red herring question may be to ask about past 3-month consumption for a fictitious brand. Any respondents indicating consumption of a nonexistent brand will earn a flag.  During the soft launch phase of fielding, we can determine how many flags are occurring and looking at that distribution we can determine what level we should terminate a respondent (i.e., term a respondent if they have 3 flags, 2 flags, etc.).
  1. Going to Instant Replay—Checking Response in Verbatims Questions: We can review the open-ends to verify that the respondent is answering the survey in a thoughtful manner. Short answers or garbage responses will count as flags.

To summarize, make sure that that the data you are including in the final dataset is accurate and reflects thoughtful and engaged respondents.  Why spend all the time and money upfront and include garbage data points?  Garbage in = Garbage out.

Utilize these top tips and you’ll avoid any foul balls in your research?  Let Merrill Research help you pick up your game!

Merrill Research—Experience You Can Count On.

_________________________________________________________

[1] George Will, Men at Work: The Craft of Baseball (1990)

The Blueprints of a Proper Questionnaire

by

By David M. Schneer, Ph.D.

All too often, we hear about research efforts—DIY or otherwise—that did not yield helpful results. When digging into those efforts to identify what went wrong, we find a combination of contributing factors—perhaps the worst of which is poor questionnaire design. Nothing foreshadows the early death of a research project better than a crummy questionnaire.

And while some presume the questionnaire is the first phase of a project, nothing could be further from the truth.  Every marketing research project starts with a clear definition of the marketing and research objectives.

For example, the marketing objective may be to sell more Chardonnay wine and the research objective may be to determine which of 3 new label designs will best accomplish that.  For a tech company, the marketing objective may be to increase profitability for the next gen mobile phone while the research objective is to determine which subset of potential new features will maximize profitability.

Often this step is ignored or assumed, and that can lead to an unfortunate disconnect between the actionability of a study and the client’s decision-making process.

Crafting the questionnaire begins only after marketing and research objectives are aligned.  You notice we chose the word “craft” instead of “develop” or “write.”  That’s because questionnaire preparation is a craft—a seamless union between art and science. It is critical to have an experienced and objective researcher craft your questionnaire. Why?

  • We’ve developed the optimum survey methodology to answer your marketing and research objectives.
  • We thoughtfully craft the questionnaire so that it provides the raw data required for specific statistical analyses needed to address those objectives.
  • We go to great lengths to ensure the questionnaire is both valid and reliable, and respectful of the respondent experience.

So what’s the deal with reliability and validity? Outside statistical research, these two terms are often used interchangeably but they actually mean very different things. Reliability refers to consistency.  In the case of a questionnaire, reliability means the degree to which measurements provide consistent outcomes.  Validity, on the other hand, represents the degree to which a question or scale measures what it is intended to measure. A good way to remember this is the example of a clock. A clock measures “true” time, and does so continuously.  If the clock were to show the wrong time, we would say it is invalid.  If it were sometimes fast and sometimes slow, we would say it is unreliable.  It is possible, however, to have a measure that is highly reliable but of poor validity (e.g., a clock that is precisely 20 minutes fast consistently).

As relates to questionnaire development, reliability is a bit easier to deal with than validity.  To maximize reliability, it’s important the questionnaire includes easy-to-understand questions, clear instructions, and unambiguous scales. As for validity, the sequencing of questions is critical.  Except for the very first question asked, all questions are potentially biased by the questions that appear earlier in the survey. For this reason, care must be taken not to “tip off the witness” by inadvertently educating the respondent or by creating awareness as a result of questions asked.  For example. We’d not want to list brands and ask the respondent to indicate which they were aware of and later ask what their favorite brand is if determining the latter was a key study objective.

To make a questionnaire valid, care must be taken to ensure that the way a question is asked maps as closely as possible to what it is intended to measure.

At its best, a questionnaire is crafted in a perfectly clear, unambiguous manner, with carefully designed questions that yield results that are usable for their intended purpose.  At it’s worst, a questionnaire ignores all these critical components, and yields invalid or unreliable results.

So, don’t try this at home, folks. We seasoned researchers live and breathe this stuff. We have the experience required to ensure survey results are valid and reliable, and provide you the sound guidance you need to drive your critical business decisions.

Merrill Research, Experience You Can Count On

When Data Goes Bad: How Sour Survey Samples Nearly Ruined the Research Industry

by

By David M. Schneer, Ph.D.

Part 1

Ask any well-trained researcher what is the number one concern with any quantitative study and they will tell you it’s about respondent quality. That is, it’s about the sample, stupid. Without good sample, data truly is “garbage in, garbage out.”

With the advent of the web as a means to gather data, Merrill Research became pioneers in the use of online data collection and gradually moved away from telephone data collection (yes, we once used phones to collect data) to the web. We did this only after having done many side-by-side comparisons to ensure that the web could provide us the same level of quality data to which we and our clients had grown accustomed. And it did.

Over the first decade of web data collection, the online sample business both soared and improved. Sample became more targeted and less expensive, and we had greater choices as new firms entered this market. Fortunately, with few exceptions, the quality of sample remained solid. The 90s were truly a boon for the industry. But that was not to last.

We began seeing problems with B2B sample when supposedly qualified respondents could not answer basic questions for those in their field. This caused us to examine data down to the respondent level and to replace about 10% of the “bad data” we were regularly receiving. Even companies we considered to offer the very highest quality sample began to show poor results! Over-sampling became part of our norm and we stepped up our efforts to identify and eliminate the bad data.

Fast forward a couple years and we began to see the same thing happen with B2C samples, but the problem was even worse. Using multiple web sample partners for one study, we were amazed to see very different results. Data were so different that we knew not all sample sources could possibly be “right”. We started to implement even more stringent data checking (e.g., eliminating speeders, straight liners, and if contradictory answers were given). This “bad data” comprised about 30% of the data collected. More cleaning and more replacement interviews followed. This took considerable professional time and was not a process that lent itself well to automation.

Eventually, we added extra pre-screening to our surveys to “catch the bad guys” – the ones who pretend to be someone they are not, and those that don’t take the survey experience seriously by either rushing through it or by giving false answers to questions. We started to compare data collected from the same sample provider for the same study and we found big differences in the data. The data collected with and without the pre-screener we crafted were so different, it became evident that pre-screening clearly mattered.

There is no question that the speed in which data can be collected today is quicker than ever. And there is no question that the cost of data collection has never been lower. However, our empirically driven conclusion is that much of the data collected via the web is over-valued (i.e., even if it was free and quick, it would not be worth using). For some studies, error caused by poor quality sample is small enough that decisions drawn from the data won’t be impacted. But from what we have seen, conclusions drawn from other studies will be increasingly wrong: garbage in, garbage out.

Merrill Research – Experience You Can Count On