Online research panels proven to produce different results

The ARF foundations of quality research compared results from the exact same questionnaire across 17 online research panels (including all of the big ones) fielded at two different points in time (2 weeks apart).  We found that the test retest reliability of each panel was high but that results differed across panels by more than you would think based on sample sizes (n=2,000 per panel per wave).  This insight led to the ARF Quality Enhancement Process, a series of metrics, planning, and reporting templates intended to control for this effect.

Cell phones are primary for close to 40% of US households

The most recent CDC NHIS survey found that 23% of all US households are cell phone only (46% of those aged 25-29) and another 15% have landlines but are cell phone primary.  We are changing the way we connect.  Landlines have become less important than cell phones and for many, talk is becoming a less important method of communication than text and social media updates.

The Media Ratings council has said that media research must have a solution for this, implying that landline-only research can no longer be equated with probability sampling.  Nielsen, Arbitron, and Knowledge Networks have all switched to addressed-based sampling methods to restore probability sampling properties.

Listening becomes a source of insights and marketing intelligence that anyone can access

Listening is a way of hearing in real time what people WANT to talk about, rather than what marketing wants to talk about.  People express themselves in their own words rather than the interviewer’s vocabulary.  Google’s team of economists proved that what people are searching for predicts many things from the geographic spread of the flu to auto sales right down to the brand.

Marketing research is no longer a gatekeeper to rich consumer insights as marketing, customer care, corporate communications, agency of record planners can now can tap into Twitter, forums, etc. directly. Only by listening would J&J have known they needed to pull the Motrin campaign.  One of the Ogilvy Award winners, the NBA, needed listening to find the way fans’ express and share their passion.  The research team must embrace listening as well as asking (i.e. surveys) to remain relevant and get to the front-end of marketing innovation.

Marketing research still struggling to be recognized as having significant impact

The ARF research transformation initiative has brought many leaders together and conducted executive interviewing in 2009 among 20 research leaders.  The consensus is that the research team is often brought in too late in the process, viewed by many below the C-suite as an expense rather than an investment, and as an impediment rather than an enabler.  We must prove that research creates an indispensible runway between the consumer and the boardroom that leads to making the right calls on big, future-focused issues that result in business growth.

Media companies and advertisers form CIMM

The leading media companies and advertisers came together to create the Coalition for Innovative Media Measurement, making a clear statement to the industry that they intend to turbo-charge innovation in media measurement.  Why?  They believe existing media metrics are not keeping up with the fast-paced evolution towards the multi-tasking, multi-platform, long-tail way that people consume media.

Shopper research takes center stage at understanding the effects of the recession

Numerous studies about the effect of the recession focused on changes in shopping patterns and increases in buying store brands.  In other words, shopper research became as important as consumer research this year, especially on the big issue that was keeping marketers up at night.  Marc Pritchard (leading marketer at P&G) has been emphasizing “store back” marketing.  The ARF formed a shopper insights council to inform media planning and the new era of winning at retail.  We foresee a powerful convergence of mobile and shopper marketing.

Marketers have always been more focused on brand-building than what happens at retail.  Marketing research has always been more comfortable with consumer research than shopper insights.  This must change.

Six big wakeup calls in 2009 are doing our profession a favor; refocusing us on what it will take to conduct trustworthy research, find unexpected feedback, provide anticipatory insights, measure media in a way that people now choose to experience it, and properly rebalance our understanding of how people choose brands by placing more emphasis on understanding the shopper.

This month, we are responding to Bernie’ Malinoff’s research on research that shows that the online research interface per se can have a significant impact on the answers.  For example, the use of sliders, drag and drop, etc. that are enabled by flash, for example, vs. text-based questions will produce different results for the same information and therefore might be a mixed blessing (better respondent experience but less consistent data). I believe that going from text-based to visual interface can influence results, although I imagine that this is less of a factor for constant sum or choice questions.

The larger issue is online research data quality…accuracy and consistency.  In particular, consistency is important as most brand research metrics from survey research are based on attitudinal measures that are intended to be compared to a normative database or trended over time.  Some have latched onto representativeness as the main lynchpin of data quality but the following graphic shows that there many equally important influencers that need to be managed.

If any of these factors are not matched from one wave of research to another, there is a risk that data comparability will be lost. A reason for this is that people are not effortlessly accessing memory and feelings.  They are reconstructing them for the purpose of answering a survey.  As they go through cognitive processes, things happen. For example, we know that longevity and survey conditioning matters.  In other words, if you send someone the same type of survey over and over again, you are likely to get conditioning effects that produce different answers. Note that this graphic has “Survey instrument” as a variable, which includes question order, length, scales, and must include, as Bernie points out, the graphic interface of the question.

Internet research has some huge advantages.  It is not only faster and less expensive; it offers an environment that is more native to our digital, interconnected world.  We must not shy away from finding the best way of harnessing the more realistic environment that internet research can offer.

Here are a few advantages that the internet offers for research:

• Interconnectedness.  Life has become an open-book exam where people can connect with anyone they want or search for information to research a potential purchase.  This aspect of real life consumerism can be mimicked in internet research, especially communities or prediction market approaches.
• Immersive environments. The behavioral economist Prof. George Loewenstein from Carnegie –Mellon University cautions when people are in “cold states” they can’t predict what choices they will make in “hot states”.  The internet offers the ability to create immersive and virtual shopping environments that will do a better job of getting a respondent into the right mindset.  Marketing science Prof. Glen Urban created the “information accelerator” which is used extensively for automotive research.
• Sight, sound, and motion.  For example, I remember when we used to use telephone research for ad tracking; we would ask a respondent in words if they remember seeing a commercial that showed XYZ.  Now we can debrand a video and stream it online before asking if they remember seeing the commercial.  Much better.
• Turnkey collection.  For example, some copy testing firms are automatically testing each and every TV commercial in a category using digital technology.

We need to be both cognizant of the effect of survey interface and progressive about testing the immersive and visual capabilities of internet research.  I’ll advise The ARF online research quality council to add survey interface elements to the ARF Quality Enhancement Process; it should be part of the structured conversation between buyer and seller.

Kim Dedeker who once publicly characterized our industry as, “we will be on life support by 2012” last Wednesday at ESOMAR in Chicago said, “Now we can ask, ‘how high is up’?”

What changed?  The industry stepped up to the plate and showed the fruits of effective, forceful leadership.  I presented the ARF’s progress, where industry leaders created a Quality Enhancement Process (QeP) that is modeled after the collaboration and transparency of “Category Management”, and is rooted in fact-based insights from our Foundations of Quality research.  Eight leading buyers (Bayer, Capital One, Coke, General Mills, General Motors, Kraft, Microsoft, and Unilever) are beginning their pilot tests of QeP, some of which will test how the solutions can be used in combination with each other.  For example, one pilot test plans to integrate QeP, MKTG, Inc.’s “grand mean” approach, and TruSample 3.0. Updates on ISO and ESOMAR efforts were also presented as part of the same panel.

About 100 people physically or virtually attended the ARF ORQC meeting the next day in Chicago, where we gave a progress report on setting up the pilot tests and a more detailed view into the templates. QeP is starting to show up in RFPs; while that is premature until the pilot testing is completed (January, 2010), it is a marker for the traction that QeP is getting.  Steve Coffey, NPD Chief Research Officer and co-chair of the ORQC presented a strawman for a new potential working committee structure that will better serve the next stage in the industry progression:

• QeP program committee; synthesize learnings from the pilot testing and continue to evolve the QeP templates.
• Research on Research committee; continue to mine the enormous amount of information inside the FoQ and issue updated analysis
• Methodology advisory board would be formed to ensure there is objective, inclusive scientific thinking and orchestrate a dialogue. 
  • Additionally, the research on research committee and methodology advisory board will work together to frame another round of research on research around remaining, critical knowledge gaps.

To address the issue of whether there is a science to online panels, the latest ARF ORQC council included a panel of four leading research scientists that was moderated by Bob Lederer.  Panelists were:

• George Terhanian – President of Global Solutions, Harris Interactive, Inc.
• Charles DiSogra – Chief Statistician, Knowledge Networks
• Steve Gittelman – President, Mktg, Inc.
• Doug Rivers, CEO, Yougov America

Some key points:

• Scientists and buyers unanimously agreed that, with proper procedures, quota samples from double opt-in online research panels can produce reliable and consistent (MKTG, Inc.’s main focus) data, which make them a valid choice for tracking research and concept testing.  (Defining such ‘proper procedures’ is what QeP is all about).
  • This is important because there is no other mode that offers the scalability the industry requires.
  • One leading buyer said a debate about the legitimacy of online research and quota samples was “revolting” and we need to move on.  He pointed out that quota sampling has a long history of being fit for purpose in marketing research in the US and in many other parts of the world.
• There was less agreement on accuracy.  DiSogra claimed that online access panels that aren’t randomly recruited exhibit unknown biases so, while they can be fine for tracking research, their accuracy cannot be trusted.  Doug Rivers (an architect of the 2004 Stanford study)   and George Terhanian each showed evidence from the 2004 Stanford research that the difference in accuracy between RDD and online access panels was only a few percentage points apart. Doug also pointed out that mean squared error is a function of sample variance as well as bias so in practice, given the extremely high relative cost per interview of RDD-based interviews, there might be less error with properly executed, access panel research where bigger sample sizes become affordable. 
  • I think one of George Terhanian’s slides summed this up nicely:

• A key to making population wide inferences through survey research lies in reducing or eliminating any bias between the sample, no matter how it is generated, and the target population.
• For companies that depend on online panels built by means other than probability sampling this can be hard work.
• For companies that depend on probability sampling, this can be hard work as well, as the achieved sampled may bear only a faint resemblance with the target population due to coverage error, non response, and the various other factors that have contributed to the decline of telephone and F2F research and the ascent of online research.

The ARF continues to be a place where all viewpoints and learning can converge.  Regarding Terhanian’s second point about this being hard work …hopefully, the QeP can do a lot of the heavy lifting and we should all know for sure by January 2010.

The following is provided as background information for your comments. You might skip parts of this if you were in the meeting yesterday.

Fifty industry leaders, buyers and sellers, colleagues and competitors have worked hard for two years via the ARF to develop answers to online research quality concerns. We were gratified that 200+ researchers came to learn about the ARF action plan. The chosen approach is to create a “Quality Enhancement Process” (QeP) that is intended to have buyers and sellers use three sets of templates in order to work collaboratively and transparently towards a common goal—to once again be able to take data quality for granted so we can focus energies on key marketing issues.

About the word “process”

The ARF is not offering a solution service…we are not becoming a supplier or auditor but we have provided a version 1.0 structure to the conversation that buyers and sellers need to have. A process approach has worked before; starting in the early 90s and continuing today, “Category Management” established collaboration and transparency between manufacturer (seller) and merchant (buyer) towards a shared goal (increase category sales at that retailer). Trade press tell a clear picture of the success of this process and we are borrowing that type of solution for an analogous situation.

About the word “enhancement”

Some will augment the ARF QeP templates with additional considerations. Some will want to work with services that offer addition elements to a solution. That is fine with us. One solution provider has already endorsed the ARF QeP via a press release, and we hope others follow, knowing that we will not endorse a particular provider. QeP is intended to be a minimum…a foundation. Industry leadership via the ARF is saying what needs to be done, not who to work with. As long as suppliers and other solution providers meet the needs of the QeP we welcome their efforts to layer in addition elements of a solution on top of the QeP.

Road tests

Eight buyers (Bayer, Capital One, Coke, General Mills, General Motors, Kraft, Microsoft, Unilever) intend to pilot test the QeP with their chosen suppliers in a way that is tightly coordinated by the ARF to maximize learning. It is a form of road test to make sure that the templates are consistently understood by different organizations, that it is not organizationally unwieldy, and that the process manages the root causes of data concerns that were uncovered by the $1MM Foundations of Quality research-on-research project.

Why are leaders participating in QeP pilot test?

John Willard from Bayer provided testimony to immediate payoff; he said that this structure will shape his discussions in the next few days about how he transitions a major research program from a traditional offline mode to online.

Tomas Emmers from Unilever said that when marketing teams don’t get the results they want they sometimes question the research; the QeP will take that challenge off the table.

Stan Stanunathan from Coke said data quality by using this process will become table stakes for suppliers.

All buyers on yesterday’s panel agreed that it was important to help shape the future, and be part of things as they move along instead of just seeing what others did.

Suppliers see benefit not only in serving their clients but also in aligning their internal organization (e.g. account and operations teams) around what matters to produce high quality, comparable data…something they care about just as much as the buyers.

As Category Management evolved, so will the ARF Quality Enhancement Process. In fact, we have declared a 30 day vetting process to incorporate what we learn from eight pilot test kickoffs, and from hearing back from the industry (and your comments here are part of that), we fully expecting a “version 1.1” of the templates.

Please share your thoughts!

We’re starting to see the light at the end of the online data quality tunnel.

On June 9^th , the ARF Online Research Quality Council (ORQC) presented detailed findings from an unprecedented US R&D project regarding online data quality, called “Foundations of Quality” (FoQ).  Now that we’re into the 90 day action plan stage, I wanted to provide an update via my blog.

Industry leaders on the ORQC felt that the issues we uncovered fall into three broad buckets (illustrated with some of the high-leverage issues):

• Panel management
  • Managing the sample source consistently since different panels produce different results
  • Managing panelist longevity very closely since newer recruits give more favorable purchase intent
  • Over time, migrating away from promising respondents and prospective panelists cash gifts for ad hoc surveys.  (Those who take surveys because they want to share their opinions are proven to be better respondents from FoQ.)
• Sample management
  • Setting guidelines for the minimum and maximum number of survey invitations in a month (3-10 completed interviews per month was associated with the highest quality data).
  • Specifying what type of wording in the invitation is biasing and needs to be avoided (e.g. avoid, “we are looking for people who…)
  • Pre and post-stratifying the data using longevity characteristics as well as demographics.
• Response quality
  • Survey length restrictions that all will be asked to abide by (the number one source of inattentive respondent behavior in filling out the survey was length)

Members of the “industry solutions” and metrics committees have been divided up and assigned to one of these three buckets to tackle the issues shown plus others in that bucket.  Each sub-committee has supplier and buyer representation.  Each team is tasked with creating recommended industry practices, metrics, common vocabulary and other management tools by mid-late September so buyers and suppliers can partner to collect trustworthy online research results.

Equally critical to developing the right set of recommendations is consensus-building.  If leading buyers and sellers of research do not agree on the rules of the road, if everyone tries to find their own solution, then any one player can poison the well for everyone else.

Permit me a personal story about that.   A few weeks ago, I received an invitation from a reputable survey firm to participate in a 60 minute survey in exchange for a $10 cash gift (already two “no-no-s”!).  After 25 minutes, I flunked a qualifying question, was terminated and received NO gift!  Imagine how bad survey-company practice coupled with unreasonable client demand would turn off respondents forever to ANYONE’S survey invitations.

However, I AM optimistic that the ARF will achieve industry consensus.  Many of the leading buyers (including Bayer, Capital One, Coke, ESPN, General Motors, Microsoft, Procter, Unilever), 17 leading panel companies and other research organizations all participated in the FoQ research and via the ARF continue to work collaboratively to craft a solution.  The ARF is also working closely with all other industry associations via ACE (Association Collaborative Effort).  It really is a remarkable industry collaboration.

If we stay on our timeline to provide a solution by September, and leading players all realize this is our shared future that we’re talking about, yes, we’re seeing the light at the end of the tunnel.