Alex Frakt wrote:peter71 wrote:Well, that's a pretty small sample of bad polls
One thing that can happen to an internet poll is deliberate respondent manipulation (e.g., a well-meaning site tries to do a legit poll but an interest group "crashes" the site), but I don't think that would happen with BH . . . then there's of course the question of the slant of the views of the people who frequent a particular site but I assume that's what the proposal wants to measure (I don't support the proposal btw, I'm just aimlessly debating survey research methodology). And then there's all the other stuff that can go wrong with any poll, scientific or no . . . in the case of voting on "best book ever" though, I wonder whether the fact of the matter is that popular opinion and elite opinion just really diverge ...
The primary issue is the self-selection process. The respondents to low response rate internet polls have a different psychological makeup than the non-respondents. In my experience, they fall into two groups. One group just likes the sound of their own voice and will respond to anything, the other is made up of those with extreme views on the subject. The former group probably is reflective of the larger population (unless the question is about the desirability of adding more user feedback options). The second is definitely not and can skew the results in unexpected and unhealthy ways.
I'll give you two more general examples of this. The first one is the original orange box M* forum ratings system. If you think about it, this was a voluntary internet poll on every post. I think we are all familiar with how small groups hijacked that system and how generally their responses did not reflect the feelings of the readership as a whole. Another example would be Minnesota's system of selecting the political parties' candidates through a caucus system. Minnesotans are generally speaking among the most moderate people in the land, yet the low turnout voluntary caucus system tends to be dominated by party extremists. The result is that party nominees usually do not reflect the real feelings of the average citizen. The disenfranchisement of the moderate middle of MN citizens from their own parties is how you get results like independent candidate Jesse Ventura winning the governorship.
BTW, the "Greatest Novel" poll is also an example of this. The question isn't whether these really are the greatest novels, the question is whether these responses actually mirror the British people's collective feeling about what are the greatest novels. I highly doubt that the list from a carefully done poll would look anything like this one.
Hmm, as a former Minnesota ultra-partisan I fear we'd have to violate the original tenets of the politics ban to get into that one, but I think the broader point is that all elections (outside Australia and a couple of other places where they fine you for not voting) are in a sense more similar to "self-selecting" polls than more scientifically rigorous ones (hence the near universality of adjusting the scientific polls via a plethora of competing likely voter models). So in a sense it all depends on what it is we're trying to measure, and if we're ok with only measuring the preferences of "motivated" or "politicized" Bogleheads then the online poll might be ok . . . we might even do kind of the opposite of a likely voter model and weight the results of the (presumptive minority) of users voting in the poll against the (presumptive vast majority) of users who'd read the thread and/or the thread index but not voted, and no doubt the proper conclusion about Bogleheads as a whole would be that most aren't all that, that political . . . but again, I'm not saying internet and/or voluntary polls don't have special problems, especially when people can see how the results are running before they vote . . . I'm just saying that, while the M* ratings sample a microscopic subsample of motivated raters, our average poll on here probably succeeds in conveying /some/ information about what "Bogleheads who care about the issue" are like.
Finally, on the novels poll, it occurs to me that potentially the biggest problem with it is one that would plague scientific and unscientific polls alike: namely, with "open ended" response options and millions of possible novels to choose from, the dispersion of responses is VASTLY wider than any credible poll I can think of . . . i.e., who knows, maybe TKAM won with 22 votes, whereas thirty other books had between 10 and 20 . . . with political polls about "what's the most important problem facing the country today" they go back and "code" your answer into 5-20 categories and an "other" category," but obviously in the case of the novels poll they couldn't really do that . . .
Anyway, it's a fun discussion!
All best,
Pete