As agreed to, I have created a DISCUSSION blog so please share your comments; the ARF needs to hear all ideas and reactions regarding online data quality and the Quality Enhancement Process we unveiled yesterday.
The following is provided as background information for your comments. You might skip parts of this if you were in the meeting yesterday.
Fifty industry leaders, buyers and sellers, colleagues and competitors have worked hard for two years via the ARF to develop answers to online research quality concerns. We were gratified that 200+ researchers came to learn about the ARF action plan. The chosen approach is to create a “Quality Enhancement Process” (QeP) that is intended to have buyers and sellers use three sets of templates in order to work collaboratively and transparently towards a common goal—to once again be able to take data quality for granted so we can focus energies on key marketing issues.
About the word “process”
The ARF is not offering a solution service…we are not becoming a supplier or auditor but we have provided a version 1.0 structure to the conversation that buyers and sellers need to have. A process approach has worked before; starting in the early 90s and continuing today, “Category Management” established collaboration and transparency between manufacturer (seller) and merchant (buyer) towards a shared goal (increase category sales at that retailer). Trade press tell a clear picture of the success of this process and we are borrowing that type of solution for an analogous situation.
About the word “enhancement”
Some will augment the ARF QeP templates with additional considerations. Some will want to work with services that offer addition elements to a solution. That is fine with us. One solution provider has already endorsed the ARF QeP via a press release, and we hope others follow, knowing that we will not endorse a particular provider. QeP is intended to be a minimum…a foundation. Industry leadership via the ARF is saying what needs to be done, not who to work with. As long as suppliers and other solution providers meet the needs of the QeP we welcome their efforts to layer in addition elements of a solution on top of the QeP.
Road tests
Eight buyers (Bayer, Capital One, Coke, General Mills, General Motors, Kraft, Microsoft, Unilever) intend to pilot test the QeP with their chosen suppliers in a way that is tightly coordinated by the ARF to maximize learning. It is a form of road test to make sure that the templates are consistently understood by different organizations, that it is not organizationally unwieldy, and that the process manages the root causes of data concerns that were uncovered by the $1MM Foundations of Quality research-on-research project.
Why are leaders participating in QeP pilot test?
John Willard from Bayer provided testimony to immediate payoff; he said that this structure will shape his discussions in the next few days about how he transitions a major research program from a traditional offline mode to online.
Tomas Emmers from Unilever said that when marketing teams don’t get the results they want they sometimes question the research; the QeP will take that challenge off the table.
Stan Stanunathan from Coke said data quality by using this process will become table stakes for suppliers.
All buyers on yesterday’s panel agreed that it was important to help shape the future, and be part of things as they move along instead of just seeing what others did.
Suppliers see benefit not only in serving their clients but also in aligning their internal organization (e.g. account and operations teams) around what matters to produce high quality, comparable data…something they care about just as much as the buyers.
As Category Management evolved, so will the ARF Quality Enhancement Process. In fact, we have declared a 30 day vetting process to incorporate what we learn from eight pilot test kickoffs, and from hearing back from the industry (and your comments here are part of that), we fully expecting a “version 1.1” of the templates.
Please share your thoughts!
The outcome of these efforts should aid the understanding of what clients need to consider when purchasing sample. The templates that the ARF has developed, and will continue to hone, will aid in providing insight to research buyers about the performance of their surveys. My first question from attending the presentation relates to the relations with the clients and vendors. It appears these tools are designed for vetting already existing relations?
I’d also like to take this opportunity to propose a link between MRA’s Review program and the ARF ORQC efforts. It seems that if the ARF tools are aimed at enhancing existing relationships and MRA’s program is aimed at transparency prior to engagement, that they are a good fit.
To provide background, the MRA Review Program includes a web site where all panel providers and full-service owners of panels can fill out a self-report about their panel. They input information about all of the factors that are thought to affect data quality (panel tenure, attrition, sourcing details, incentives, validation offerings, etc.), and the information is then collapsed into a standardized report that visitors to the website can review. All of this is free for both panel providers/owners as well as visitors. This provides the front end transparency required during the vendor evaluation process.
The ARF ORQC then allows the clients to validate satisfaction once engaged and based on projects versus overall scope.
If further validation of the abilities and performance of a panel provider is needed MRA also offers a third-party independent review of the panel reports in order to confirm that the information is accurate.
Since both programs are designed to be “living” projects, standards of transparency change over time. Many of the transparency recommendations from the ARF project are already in place and will continue be reflected. However, additional items may be added in order to bring the two programs in line with one another.
Ultimately, with slight changes, buyers of research would be able to visit the MRA Review Program in order to see all of the panel level details about their vendors that the ARF recommends. Moreover, they would also see and have the ability to select vendors based upon whether they offer the study-level technical report recommended by the ARF.
Has any research on research been done on online river samples (e.g. Opinion Place River Sample)?