"Online personality quizzes ... command little if any credibility," says the official report.
After months of consultations and millions of dollars down the drain, the Liberal government’s widely-ridiculed “MyDemocracy” electoral reform quiz claims Canadians are more or less “satisfied with the way democracy works in Canada.”
Does that mean Trudeau’s government is planning to stick with the status quo?
Who knows? But he’ll have a heck of a time wiggling out of a crystal clear campaign promise to “make sure the 2015 election will be the last under first-past-the-post”:
— PressProgress (@pressprogress) December 2, 2016
And although Liberal cabinet ministers have repeatedly stressed the quiz is a “scientific study done by professional scientists” that will help influence their “final outcome,” the government’s own report concedes the quiz has some “credibility” problems.
The report, written by Vox Pop Labs CEO Clifton van der Lidnen (whose company built the survey), explains that the tool being used to collect “scientific” data to influence major changes to the electoral system intentionally “emulates” the “online personality quizzes” that are so popular with teenagers on the Internet yet “command little if any credibility”:
“Associating users with an archetype emulates the viral model of online personality quizzes in that it offers the potential for self-expression in the form of shareable content designed for mass diffusion via social networks … Despite their popularity, online personality quizzes offered by Internet media companies command little if any credibility. In fact, their lack of credibility has become a defining feature of such quizzes in popular culture and yet they continue to surpass most other forms of content in terms of online audience reach.”
Yet he defends this approach, explaining that MyDemocracy’s “online personality quiz” actually “overcomes the lack of credibility common to such content by presenting users with valid inferences using a robust methodology.”
Strange they would choose to model their quiz on something that has zero credibility in popular culture considering few people are putting much stock in its “robust methodology”:
Every question in that manipulative survey is either beside the point, or misrepresents the issue, or is framed to elicit a specific answer.
— Andrew Coyne (@acoyne) December 5, 2016
Speaking of “robust methodology,” the report doesn’t even say how many times Canadians filled out the survey.
Last month, Trudeau’s government mailed 13 million postcards to every household in Canada urging citizens to fill-out the online quiz.
The report states that “383,074 unique users completed the survey.”
Except the key word there is “unique” – buried elsewhere, the report admits “MyDemocracy.ca could be used multiple times by the same user” since anyone can do the quiz “anonymously.”
The report says it used IP addresses and cookies to “remove multiple entries by the same user,” though Vox Pop Labs concedes this technique isn’t always able to tell the difference.
That’s a potential problem since MyDemocracy instructed rural and low-income users without internet access to go use a computer at their friends house, MP’s office or local library.
How many multiple responses were weeded out? The report doesn’t say.
We do know that sending 13 million postcards led to only 383,074 unique responses, although they weeded out 15,411 responses from foreign IP addresses who may or may not have been Canadian expats who live and work in other countries but are still guaranteed full voting rights.
Of the 367,633 unique Canadian responses, one-third (124,606) of all responses were removed because they didn’t provide the government with their personal details – even though the quiz explicitly tells users that providing personal details is “completely optional”:
In the end, 13 million postcards to 35 million Canadians resulted in only 243,057 valid responses, with at least 140,117 unique responses disqualified in addition to an unknown number that were never included in the final report.
Although two-thirds of speakers at town halls across Canada – including Conservative ridings and Trudeau’s own home riding – told the government they wanted proportional representation, the report says the Buzzfeed-style quiz was necessary because Canadians who showed up to town halls and wrote to their MPs “reflect a rather specific subset of the population” representing an “imbalance of interest in electoral reform.”