Comments on: Achieving Consistency in Research Results http://blog.joelrubinson.net/2009/08/achieving-consistency-in-research-results/ Marketing and Research Consulting for a Brave New World Sun, 29 Jan 2012 18:51:37 -0500 http://wordpress.org/?v=abc hourly 1 By: Bob Harlow http://blog.joelrubinson.net/2009/08/achieving-consistency-in-research-results/comment-page-1/#comment-201 Bob Harlow Mon, 10 Aug 2009 16:47:39 +0000 http://blog.joelrubinson.net/2009/08/achieving-consistency-in-research-results/#comment-201 We cannot hope to attain consistency (or what us psychometricians call "reliability") unless and until we know that (1) respondents are recruited in a similar way across panels, giving us a stable sample composition and (2) respondents interpret the questions in a similar way. This second condition requires, of course, that respondents give survey questions their full attention and thus makes online research quality extremely difficult to attain. In most cases, respondents have no reason to carefully read or pay attention to questions in online surveys – unlike phone surveys where an actual person could hear one's responses and (potentially), the respondent feels some pressure to be attentive and engaged. Longer online surveys, of course, exacerbate the issue. Unless and until we tackle the issue of respondent attentiveness, consistency will remain elusive. There is no reliable statistical "fix", because you are just playing with random noise, not responses with any underlying meaning. Weighting is one red herring among many. We need a new model of respondent engagement. We cannot hope to attain consistency (or what us psychometricians call “reliability”) unless and until we know that (1) respondents are recruited in a similar way across panels, giving us a stable sample composition and (2) respondents interpret the questions in a similar way. This second condition requires, of course, that respondents give survey questions their full attention and thus makes online research quality extremely difficult to attain. In most cases, respondents have no reason to carefully read or pay attention to questions in online surveys – unlike phone surveys where an actual person could hear one’s responses and (potentially), the respondent feels some pressure to be attentive and engaged. Longer online surveys, of course, exacerbate the issue.

Unless and until we tackle the issue of respondent attentiveness, consistency will remain elusive. There is no reliable statistical “fix”, because you are just playing with random noise, not responses with any underlying meaning. Weighting is one red herring among many. We need a new model of respondent engagement.

]]> By: Annie Pettit http://blog.joelrubinson.net/2009/08/achieving-consistency-in-research-results/comment-page-1/#comment-194 Annie Pettit Fri, 07 Aug 2009 01:21:51 +0000 http://blog.joelrubinson.net/2009/08/achieving-consistency-in-research-results/#comment-194 i find the result about weighting to be so interesting. so many people assume weighting solves everything. there are just so many variables that simply can't be accounted for that weighting can never possibly hope to overcome them. i hope people take finding to heart. the final answer seems to be 'know thy panel.' i find the result about weighting to be so interesting. so many people assume weighting solves everything. there are just so many variables that simply can’t be accounted for that weighting can never possibly hope to overcome them. i hope people take finding to heart. the final answer seems to be ‘know thy panel.’

]]>