Tips to maximize the effectiveness of your questionnaire.

A lot of attention is being paid to how little attention we have. (Ironic, that.) Scientists have estimated that the average time a person can focus on one thing has dropped from about 2 ½ minutes twenty years ago to about 45 seconds today. This hasn’t surprised anyone who I’ve discussed it with, given our over consumption of social media, constant technology use, and life stressors.  

This is bad news for market researchers trying to measure customer opinions. Standing between your survey and success are respondent dropout, straight-lining and speeding behaviors, and inattention errors and biases. These can be managed to an extent on the back end through diligent data cleaning, but optimizing your questionnaire upfront is the best way to maximize response quality, and has the added bonus of sample efficiency. That is, you won’t be throwing out responses that could have been used if your questionnaire had been crafted better.

Optimize for Mobile Display.

Always, always optimize for mobile display. Large, complicated choice grids and graphical stimuli loaded with small font size text just don’t display well. Questions that require continual scrolling or zooming in/out aren’t optimal either. Just because you ask respondents to scroll doesn’t mean they will. In all of these cases, if respondents aren’t viewing all the information and are just “winging” their answers, it will be very difficult to see that in the data.

Consider the trade-off of disallowing mobile. Many respondents, especially those obtained through the large opt-in research panels, tend to take surveys on their device, so disallowing mobile can greatly impact response rates, requiring longer field time and potentially even decreasing quota feasibility.

Keep it Manageable.

Longer surveys correlate with higher instances of respondent dropout. There is a direct relationship that’s been proven time and again. Shoot for 10 to 12 minutes, but factor in who your respondents are, how engaging your subject is, and how easy or difficult the questions are to answer.

Trim redundancies. If you are struggling to trim a lengthy questionnaire, look for redundancies. I’ve noticed that inexperienced writers often ask what amounts to the same question stated in multiple ways, just to be “doubly sure.” For example, “What would make your shopping experience here better?” and “What improvements would you like to see in the shopping experience?” are wording spins. The same goal applies to features batteries. Rating an online retailer on “convenience” and “ease of use” will result in the same data.

Redundant questions are confusing and frustrating to survey takers, and frustrating your respondents increases dropout. Your data will be more reliable and insights more clear by eliminating redundancies. An experienced survey developer (like Level 7, hint, hint) can be a great resource here, but you could also ask a colleague to take a read from the respondent lens.

Simple is Best.

Avoid complicated, wordy language. This isn’t the place to show off how smart you are. A consumer questionnaire shouldn’t read like a high school exam. In consumer research, complicated wordy language is off-putting at best, and causes understanding errors and interpretive biases at worst. And remember, whether we researchers like it or not, it’s common for respondents to move through questionnaires quickly.

Write questions and instructions to a tenth-grade reading level. Start simple, and go more complex only when you must. Avoid marketing and technical jargon. Go with a conversational style using the language that your customers use. If you don’t know, some qualitative research can easily provide the lexicon.

Consider this verbose question: “Please rate on a scale of 1-10, where 10 is strongly pleased and 1 is strongly displeased, how likely you are to refer people in your social environments to our company based on your experiences specifically with our shipping methods and returns policy.”

Instead, you could simply ask, “How would you rate our shipping and returns? 5=excellent, 1=poor”

The first version works hard to throw in every detail we want the respondent to consider. But the response quality will be better for the pared down version.

Remember the Human Element.

Questions concerning health (e.g., “Do you have any history of mental illness in your family?”) and habits (“How often do you feed your child fast food?’) can feel interrogative and overly personal. Other questions have a socially desirable answer (“How often do you volunteer to help others in your community?”). Respondents are liable to bend the truth, which leads to poor data quality.

Adopt an empathetic approach written from the respondent perspective. This increases their comfort in telling the truth. Empathetic design may involve humor. For example, Kantar found that 27% percent of people who were shown a meme in the question “do you recycle?” admitted to never recycling, compared with 1% not shown the meme. Incorporate humor carefully and in moderation to keep the tone appropriate.

Position sensitive questions near the end of the questionnaire. If these are the final questions, tell the respondents that. By this point, respondents will feel invested and are less likely to drop out.

Q/C Along the Way.

Quality traps are a best practice. All consumer questionnaires should include appropriate traps. Here are a few basic examples.

A red herring trap captures inattentive respondents. A red herring is an invalid option in a choice list of valid ones. As an example, “Which of the following are Olympic sports? Gymnastics, Swimming, Ice Skating, Quidditch”. Obviously, Quidditch is not played in the Olympics (it’s a fictional sport from Harry Potter) but rather a trick for catching respondents who aren’t paying attention.

A straightliner trap captures respondents who are moving through the survey quickly by not reading the choices. This behavior is commonly associated with selecting the same option (e.g., choice #1) down an entire grid or question series.

A gamer trap captures respondents trying to game the system by selecting every choice in an effort to qualify for the survey and collect the incentive. This trap is trickier to write, but critically important in low incidence surveys that can be greatly damaged by this type of fraudster. One example of a gamer trap is a fictitious brand in the screener question: “Which of the following cereal brands have you eaten in the past month?: Cheerios, Frosted Flakes, Lucky Charms, Loopholes”. Loopholes is not a real cereal brand and should never be selected.

On the surface, crafting an optimally performing questionnaire may seem simple, but there is an undeniable science behind effective and efficient surveys. If you need help, the experts here at Level 7 are just a call away.