I am one of 5 market research bloggers who are writing on a common topic.  You’ll also be hearing from Annie Pettit (organizer), Josh Mendelsohn, Bernie Malinoff (Element 54) and Brandon Bertelsen.

This month, we are responding to Bernie’ Malinoff’s research on research that shows that the online research interface per se can have a significant impact on the answers.  For example, the use of sliders, drag and drop, etc. that are enabled by flash, for example, vs. text-based questions will produce different results for the same information and therefore might be a mixed blessing (better respondent experience but less consistent data). I believe that going from text-based to visual interface can influence results, although I imagine that this is less of a factor for constant sum or choice questions.

The larger issue is online research data quality…accuracy and consistency.  In particular, consistency is important as most brand research metrics from survey research are based on attitudinal measures that are intended to be compared to a normative database or trended over time.  Some have latched onto representativeness as the main lynchpin of data quality but the following graphic shows that there many equally important influencers that need to be managed.

If any of these factors are not matched from one wave of research to another, there is a risk that data comparability will be lost. A reason for this is that people are not effortlessly accessing memory and feelings.  They are reconstructing them for the purpose of answering a survey.  As they go through cognitive processes, things happen. For example, we know that longevity and survey conditioning matters.  In other words, if you send someone the same type of survey over and over again, you are likely to get conditioning effects that produce different answers. Note that this graphic has “Survey instrument” as a variable, which includes question order, length, scales, and must include, as Bernie points out, the graphic interface of the question.

Internet research has some huge advantages.  It is not only faster and less expensive; it offers an environment that is more native to our digital, interconnected world.  We must not shy away from finding the best way of harnessing the more realistic environment that internet research can offer.

Here are a few advantages that the internet offers for research:

• Interconnectedness.  Life has become an open-book exam where people can connect with anyone they want or search for information to research a potential purchase.  This aspect of real life consumerism can be mimicked in internet research, especially communities or prediction market approaches.
• Immersive environments. The behavioral economist Prof. George Loewenstein from Carnegie –Mellon University cautions when people are in “cold states” they can’t predict what choices they will make in “hot states”.  The internet offers the ability to create immersive and virtual shopping environments that will do a better job of getting a respondent into the right mindset.  Marketing science Prof. Glen Urban created the “information accelerator” which is used extensively for automotive research.
• Sight, sound, and motion.  For example, I remember when we used to use telephone research for ad tracking; we would ask a respondent in words if they remember seeing a commercial that showed XYZ.  Now we can debrand a video and stream it online before asking if they remember seeing the commercial.  Much better.
• Turnkey collection.  For example, some copy testing firms are automatically testing each and every TV commercial in a category using digital technology.

We need to be both cognizant of the effect of survey interface and progressive about testing the immersive and visual capabilities of internet research.  I’ll advise The ARF online research quality council to add survey interface elements to the ARF Quality Enhancement Process; it should be part of the structured conversation between buyer and seller.