Over that past 5 years I have run projects that collected more than 5 million customer satisfaction surveys using web and IVR (phone) tools. We always found that even when we controlled for differences in the demographic characteristic between the two populations we always found large and significant differences in the attitudinal questions.

For example, the percent of people who were very satisfied could be 5 to 15 points higher for respondents us used IVR compared to web. This finding persisted across industry and time. There is just something different in the experience of taking a survey on line vs. with an IVR system. With attitudinal data it is hard to say which is more valid both seemed to be equally reliable.

Dave

cell – 847-903-5744

So you know, we are not a supplier, we are an industry association that collaboratively managed a research project that had 20 or so organizations involved in co-managing. We do not “favor” one method over the other. We factually inform the industry and let the chips fall where they may. we are on the side of truth. the analysis was conducted by an independent analyst. RDD is fine and appropriate in certain cases. Our project shows, BTW, that somewhere beyond 5.5 million people are actively engaged in online research panels, with less than 20% belonging to multiple panels, so we are happy to dispel mythology that it is a small number who take many surveys. On the other hand, it is questionable as to how many are reachable by phone. A 1999 JMR paper estimated that 3-4% account for 50% of phone surveys.

To Lance’s comment, where my biggest concern lies is when a non-profit industry organization takes a stance supporting one form of interviewing over another. As a group supported by advertisers and other such member support, I’m not sure it’s the right direction. I supported the quality initiatives, which are designed to improve and protect the online research business by setting standards everyone should be following. I can’t, however, support policing or providing information supporting one direction over another. Keep in mind, there are a ton of “Research on Research” studies that have shown very different results than yours and some that have shown similar.

I guess I have to say I’m disappointed to see that ARF is dabbling in this type of research as a means of promotion. I wouldn’t be surprised to see some advertising support impacted. No matter what type of research a firm does, I’d expect they might have concern over how the organizations they support might be impacting their business (positive or negative, it should cause concern).

It would be like surveying everyone who lived in Rhode Island and comparing it to 100 surveys from Texas – not exactly apples-to-apples.

Also, I didn’t see any mention of what kind of birthday-rule or other randomized respondent selection tool within the households was used, which of course would normally counter the huge discrepancy of age and gender that appeared to exist in the phone sample that was collected (although the lack of younger age ranges could partly be attributed to the lack of a cell-phone sample, as a much greater percentage of 18-39 year olds are cell-phone only). Was one used at all?

However, more than any of this, I am extremely concerned that a non-profit industry organization is taking on the role of researcher, and seemingly endorsing one method of data collection over another as being more valid, and what I find even more descerning is that you appear to be claiming validity for a mode that while useful, is more heavily criticized for claiming to be something it isn’t by the entire rest of the research industry. It’s taken a while for even the people who sell the online panels to finally admit more of what the panel should and shouldn’t be used for, although many of the buyers of research have already figured this out, simply by being burned numerous times (fool me once, shame on you – fool me twice . . .).

Please understand that we as a company offer online and telephone interviewing to our clients, and believe each has it’s place – it’s up to the researchers and the data collectors to help educate the rest of the community as to what works best to satisfy the needs of the specific research question at hand. So when trying to help educate as well as battle in a price war that simply can’t be won, I personally find it very upsetting to have the ARF endorse such a study with such skewed sample sizes across methodologies claiming they can report anything that even remotely reflects a “finding.”