Saturday, December 03, 2011

Survey Madness: Gries Gun at the DPP, Taiwanese won't fight

The Taipei Times ran a story on Peter Gries' survey, whose press conference on Friday for his "non-partisan" survey I had put in my list of events....
In the survey conducted by Peter Gries, director of the University of Oklahoma’s Institute for US-China Issues, Ma’s support rate was 34 percent, while Tsai was on 26 percent.

Support for People First Party Chairman James Soong (宋楚瑜) was at 10 percent, while undecided voters accounted for 30 percent, the survey showed.

.....

The survey, conducted between Nov. 17 and Monday with a sample size of 500 people and a 6.5 percent margin of error, used an Internet survey methodology to avoid biases associated with telephone and face-to-face polls, he said.
I think, if you've been paying attention to the zillion election surveys I've posted so far, you can see that this one is an outlier. Looks like a typical case of academic not knowing the lay of the land, parachuting in, and reporting to us poor benighted people on the ground here that he has found the Holy Grail [CORRECTION: Gries says he is a visiting scholar. See his comments below]. The 8 point spread in this survey exceeds almost any of the recent surveys of even the most shamelessly pro-KMT papers (when was the last time you saw a local survey of this election that generated a 30% "don't know"?).

Tip: Gries advertised his survey as "non-partisan." When something Taiwanish advertises itself as neutral or non-partisan, it is almost always going to wind up pro-KMT.  When I saw the term "non-partisan" I just started laughing, knowing what the poll would say.

The major problem here was that the Taipei Times should have ignored this or trashed it. It was wrong to give this any credence whatsoever.

Above are two slides from Friday's presentation. Gries used an internet survey methodology whose innate biases he attempted to correct using a method called sample matching. The purpose of sample matching is to overcome the fact that different modes of survey (phone, face-to-face, internet) are answered differently by different people. Scholars who use such methods argue that sample matching enables them to overcome the self-selection bias -- the people who take internet polls are different from people who do not, and they choose to take the poll, they are not cold called as in a phone survey. For a look at this kind of methodology in political surveys, see this 2007 paper. Fundamentally, it is quite true that an internet survey avoids the biases associated with phone surveys. It does this by introducing different biases.....

Because the number of respondents with education was low, Gries had to give them greater weight in the poll. Based on what?

Note further that YouGov is a private firm and its claims about its methodology should be taken with the skepticism appropriate to any corporate advertisement.

Of course, Gries does not know WHO took the internet survey in Taiwan. So this is a survey of ghosts.

Finally, Peter Enav of AP, at Friday's presentation, wisely asked Gries if this was a survey of likely voters. Gries responded that the number of people who responded as likely voters is so low that he didn't dare say it was a survey of likely voters.

So, basically, Gries flew into Taipei and put up this "non-partisan" survey of ghosts which he knew perfectly well was an unrepresentative agglomeration of numbers and which he also could not say was a survey of likely voters -- or anything else worth knowing.

So what was the purpose of a "non-partisan" survey that shows Ma up by 8 using a for-profit platform? I think the question answers itself.

The other fascinating survey that came out this week was even more bogus. This was the widely circulated AP story on a poll which claimed that Taiwanese youth were losing their appetite to fight China. Zounds!
A survey published this week by Taiwan's Commonwealth Magazine appears to confirm that Taiwan's process of demilitarization is rapidly gaining steam. Based on a sample of students aged 12 to 17, it found only 38.7 percent would be ready to see either themselves or a family member fight if a new war broke out, while 44.3 percent would not. The remainder had no opinion.

........

The Defense Ministry declined to comment on the survey, saying it had no information on the way it was conducted. Commonwealth said it was carried out by mail between Oct. 17 and Nov. 4 and that the 3,715 responses represented a 74 percent return on the 5,054 questionnaires it sent.
This one is also a stinker. The numbers are absurd. First, it is a survey of 12 to 17 yr olds. A 12 yr old isn't capable of forming a meaningful opinion on a complex moral question of this nature. But second, the numbers are ridiculous. 5054 surveys mailed out and 74% mailed back within two weeks? For crying out loud! I want these guys' addresses for my own surveys! Seriously, in a typical mail survey months are necessary, not weeks and the researcher usually sends out follow up surveys or requests because the initial response rate is in the teens or twenties (for example). Perhaps there is some key bit of missing/misunderstood information not in the piece (they were handing out a free Porsche with each returned survey, it wasn't a mail survey, etc).

Commonwealth hosts an English version of the story here and the whole thing appears to be typical media bullshit slanted hyping. The item appears in a survey of teens. Here's the actual quote:
"Would you be willing to see yourself or family members head to battle if the country went to war with another country?" 39 percent answered they were "willing" or "very willing," but even more (44 percent) said they were unwilling. (See Tables 1-4)
Follow the link to the Table if you like. The question did not ask the students if THEY THEMSELVES would be willing to fight but included "family members." Raise your hand if you think a question that asks about Dear Old Dad going off to fight the PLA is the same in the students' minds as themselves going off to fight to protect Mom, Dad, and Sis. Nor did it ask anything specific -- "If China attacked Taiwan...." "If China attacked Taiwan and the US and Japan backed Taiwan...." "If the Philippines and Taiwan clashed over the Spratlys...."

The AP article says the students were losing their appetite for fighting China. The question does not even ask about China!!!!!

The sad part is, that this is an important issue and should be explored, especially with the growing China threat. But this isn't the right peg to hang this story on.

Of course, it is all meaningless anyway. People say one thing when there is no threat, and quite another when it materializes. As a wise observer pointed out to me, in 1933 the Oxford Union held a famous debate in which it resolved that "this house will not fight for King and country." There was no shortage of recruits from that house when WWII in Europe began six years later.....
_______________________
Don't miss the comments below! And check out my blog and its sidebars for events, links to previous posts and picture posts, and scores of links to other Taiwan blogs and forums! Delenda est, baby.

29 comments:

Marc said...

74% in two weeks? It's possible.

I think the survey was probably sent to students with a chance to win a free Ipad.

Anonymous said...

I've received phone calls for a survey, when they asked whether I voted in Taichung or not, I said no, and that ended very quickly.

George

Rust said...

Speaking of polls,did you watch the debate? TVBS & Chine Times have two ridicules polls which say Ma is the best debater! The TVBS one is amusing, since Ma is the best debater by a 30% to Tsai's 27% to Song's 20%, yet both Ma & Tsai's support drop by 2%, but the suppose-to-be worst debater's support increase by 5%! Ironic, not?

Personally, I think Tsai did best, but the same can be argued for Song. Ma did worst should be a Taiwan Consensus.

Peter Gries said...

Hi Michael, I'm sorry you are unhappy with the results of the election questions in my national Taiwan survey. I myself did not know what the results would be until the day before the press conference, when I received the data from YouGov in California.

You might consider getting some basic facts strait before spreading untruths. I am not "parachuting in, and reporting to us poor benighted people on the ground here that he has found the Holy Grail." I currently live in Taipei as a visiting scholar at NTU's Psychology Department, where I just gave a talk to their indigenous psychology group, hosted by Hwang Kwang-Kuo. Together with KK, I am part of a movement that strongly OPPOSES the tendency of Western social science to be blindly foisted upon Taiwan and Chinese realities. I do NOT claim to have the "Holy Grail," and was very frank duirng the rpess conference about the limitations of my survey. Indeed, it even clearly says in the slide that you reproduced here that "the lowest educated had to be heavily weighted." Other Taiwan surveys suffer the same problem, but rarely publicly acknowledge this limitation.

On methods, you have confused "sample matching" with "weighting." See "SAMPLING FOR WEB SURVEYS," by DOUGLAS RIVERS for more info, which I can e-mail you or any of your readers. I also encourage you to download my actual data from http://www.ou.edu/uschina/ to check the representativeness of the sample and replicate my analysis. If I have made any mistakes, please let me know.

As to why the Taipei Times covered it, perhaps its because they sent a reporter to attend the event who spoke with me and sought to report the facts. I emphasized "non-partisan" in the presentation because I was advised to highlight whatever was most "newsworthy" about the survey, and I figured that with the end of Global Views political polling, the lack of neutral surveys was a value added of my survey.

As for my own political position, I like the way Nathan Batto puts it in his "Frozen Garlic Manifesto": "I will not try to convince you that one side is “right” and the other is “wrong” or that you should vote for candidate X. The one exception to this is my normative bias in favor of democracy."

cheers, Pete Gries 葛小偽

Anonymous said...

Generally, if a survey is sent via snail mail, no response in 2 weeks means 90% chance that you won't get a response. This kind of mail generally takes only on day each way to travel in Taiwan. Unlike the US which might take a week to travel each way.

For surveys sent via email, generally expect response in a week or it just gets buried in other mail and forgotten.

George

Anonymous said...

I am quite curious what questions they asked when they conducted the survey. Questions like "would you rather be controlled by China or would you rather fight to for the current independence that we enjoy"

Traditionally, throughout history, very few records of the people initiating resistance, rather it's normally the ones that have military power initiating the resistance unless revolutionary spirit has grown among the people.

George

philippe mckay said...

@ Gries and Micheal,
what is about "likely voters"? Was this a survey that polled unlikely voters? i find that aspect interesting.

Ultimately for me, surveys are acts of persuasion where actors try to present "realities" that they want others to accept...

I'm very happy to see a response from Gries. hope this can be expanded :-)

Michael Turton said...

Personally, I think Tsai did best, but the same can be argued for Song. Ma did worst should be a Taiwan Consensus.

That does seem to be the consensus, luckily for the good guys.

Michael

Anonymous said...

I question Gries' motives for doing such a survey in the first place. Link to his page and it's all about China, not Taiwan. OU also has a Confucius Institute, too, and we all know about the conditions of having that! There's a lot of $ involved between China and OU.

Just sayin'

Anonymous said...

Michael, you're just embarassing these flawed researchers and their flawed logic!

frozengarlic said...

You guys just don't understand the American academic world very well. I have only met Peter once, so I can't speak to his motivations and prejudices, but I can tell you that creating lousy data for political purposes is a good way to kill your academic career. We academics have one standard, publications. Your career and salary depend almost entirely on your publication record. Good journals won't publish bad work. Good journals don't care about whether Oklahoma has a Confucius Institute. And most academics I know would be insulted by the idea that they would write something to shill for a donor to their university. In short, deliberately producing and publicizing bad data for political purposes is an effective way to ruin your academic reputation and academic career. Very, very few scholars are willing to take that risk.

Some of you might want to think twice about reflexively assuming that there are political motivations behind everything. Sometimes the news simply is not good for your side.

Michael Turton said...

Some of you might want to think twice about reflexively assuming that there are political motivations behind everything. Sometimes the news simply is not good for your side.

Nathan, I know you want to defend your friend Peter. But real academics don't announce their findings at pressers, they announce them to their fellow academics first AND then hold pressers. Much later, when the results have held up under criticism. Holding a presser with a dataset and results you downloaded the night before? You're either incompetent or you want to influence the elections or you're pimping some product and Taiwan is just fodder. No other option is available.

Gries has been attempting to put this together for weeks, contacting reporters. Why? No academic behaves that way -- that's what a propagandist does.

Just focus on the numbers -- it is not a survey of likely voters, had to be heavily and arbitrarily weighted, uses a controversial methodology, and has Ma up by 8 with 30% undecided. Poll numbers like that haven't come out in months. Doesn't that ring any bells for you? That very day the prediction market had Tsai up by eight points over Ma (too high, I think), and a couple days before the China Times had Ma up by ~6. Even the pro-KMT pubs don't have Ma up by eight. There's something deeply wrong with the numbers.

Perhaps you should think hard about Gries' behavior here, Nathan, so you can understand what you're defending. As you said, you don't know Gries very well.

Michael

Anonymous said...

Gries is here as a Taiwan Fellow on behalf of the MOFA. Shall we presume therefore that MOFA funded this survey? If not, who is behind it?

Gries claim that he "lives in Taiwan" is disingenuous. His CV states he is here for four months - his fellowship ends this month. Mr. Gries, do you plan to live here longer?

He has, as far as I can tell, no academic publication record on Taiwan issues. His publications on China, however, are lengthy.

Frozengarlic claims that there are no political motivations behind Gries work, and I find that terribly naive. Gries is a political scientist for chrissakes who directs an organization concerned with political relationship between the US and China.

Anonymous said...

Some of you might want to think twice about reflexively assuming that there are political motivations behind everything. Sometimes the news simply is not good for your side.

That's a cheap shot.

I'd be concerned why an outsider is making political surveys regardless of the outcomes. Especially if the outsider is a meddlesome American or Chinese

Undecided said...

30% undecided makes me think it's actually fairly accurate. Why? Simply because very few people are interested in the elections this time round, as compared to the past couple. In my work I speak with a wide range of people from many different backgrounds and covering the full spectrum of ages, and very, very few of them hold opinions strongly one way or the other about the 2012 election. The low number of undecideds in other polls makes me suspicious and I assume that a lot of people who've answered them have just said what the interviewer wants to hear. In this case, because the poll is not from Taiwanese media or parties, I expectt that people don't feel that same need and hence answer more honestly.

Ma is seen seen as a failure by almost everybody across the political but when Tsai is only polling about the same (in other polls) suggests that they see her as a failure also. 30% undecided - yep, I think that reflects reality much better than a lot of other polls.

D said...

"they announce them to their fellow academics first AND then hold pressers. "

I don't see anything inherently unethical about this. You're less sure of your results this way, but you may want to risk it given that the election is only a month away and your polling data will basically be meaningless after it's over. Academics are not immune to the quest for relevance.

Thanks for exploding the "Taiwan youth won't fight against China" poll though. I wondered what was up with that and now I see. I did like the grumpy-grandpa military reaction: "we're surprised even that many young'un's are willing to fight!"

Michael Turton said...

I don't see anything inherently unethical about this.

It's not unethical to want to have a press conference to announce results or to want to influence elections in the direction you want them to go.

Michael

Michael Turton said...

. The low number of undecideds in other polls makes me suspicious and I assume that a lot of people who've answered them have just said what the interviewer wants to hear.

That's because you are comparing apples and oranges. Gries' poll is a poll of random entities on the internet. It is completely unknown who voted/spammed in it. For all we know it was software bots.

The polls you are comparing it to are in-person or phone polls with numbers given of LIKELY VOTERS by recognized media organs, for the most part. Ideally these are conducted with proper sampling (heh). Naturally likely voters are more likely to have made up their minds -- that's why they are going to vote, and why their numbers are so different from Gries'. Gries' numbers are a completely different population which he himself avers, is not a population of likely voters.

In other words, Gries' called a press conference to say absolutely nothing about the election.

Michael

D said...

"or to want to influence elections in the direction you want them to go.“

He at least (here in the comments) claims that's not what he's attempting to do and it's hard for me to see why he would, or why he would even think his little poll could have an influence on the election itself. If the accusations were going the other way I'm certain you'd be demanding more convincing evidence than questionable methodology.

I know zero about polling techniques but here's a "likely voter" anecdote. This weekend I witnessed a foreigner asking two Taipei locals in their 20s how they would vote in the election. Both squirmed and then said "I don't think I'll vote" and "I don't know who to vote for". So these are presumably "unlikely voters". But maybe they'll end up voting and maybe polls should include them.

Also, their "I don't know" really sounded like "I don't like Ma or the KMT very much but I'm worried the DPP will fuck things up." My sense is that the DPP needs to put more weight into ameliorating this perception, even if you think it is a misperception. No?

Michael Turton said...

My sense is that the DPP needs to put more weight into ameliorating this perception, even if you think it is a misperception. No?

I agree, this is a problem that the DPP faces. Tsai I think radiates a kind of calmness and braininess that has gone a long way to address this issue, all by itself. Plus the contempt for the DPP also has a strong social class component -- one reason Chen was hated so much was because he was a hick from down-home Tainan, rather than one of the wealthy or political elites with a foreign education like Ma or Soong or to a certain extent, Tsai herself.

I've found over the years that people need know a bit about who they are speaking to before they begin revealing their political selves, which is why I don't put much stock in anecdotes about what people say when they talk to foreigners. I prefer to watch their behavior. When you ask around you find that people say they don't know who to vote for, or they are not going to vote.... and then turnout is 80%. So I take all such squirming with a grain of salt.

Michael

D said...

"and then turnout is 80%"

All the more reason to discount any distinction between "likely voters" and non- in a poll?

Michael Turton said...

Can't though. If it were the same poll with the same sample, you could perhaps generalize, but two different samples, one identifiable, one not, conducted in two different survey modes, one by phone, the other by internet.....

frozengarlic said...

I'm really not all that interested in specifically defending Peter, who I barely know. Besides, I'm sure he can take care of himself.

I'm much more interested in attacking the assumption that political motives underlie everything. To me, the most revealing statement in your post is, "So what was the purpose of a "non-partisan" survey that shows Ma up by 8 using a for-profit platform? I think the question answers itself." When I was in graduate school, one of my professors would roast me any time I used the words "obvious" (as in "It is obvious that Nathan believes Peter is innocent.") These kinds of phrases usually indicate intellectual laziness, a willingness to accept the convenient causal relationship rather than to think things through and question whether there are other reasonable possibilities. In the case of survey results, the assumption often seems to be that any result that shows a KMT candidate doing well must have been deliberately shaped to produce that result.

(By the way, it isn't entirely clear to me why polls showing Ma doing well are always interpreted as advantageous to Ma's re-election effort. Huge survey advantages didn't help Wu Den-yi in 1998.)

To me, it doesn't seem that strange that someone using a brand-new methodology produced a result that is out of line with those produced by more established methods. (As far as I can tell, he is using a very sophisticated quota sample. Most pollsters are skeptical toward quota samples.)

For all I know, Peter might be a China shill. But I don't have anywhere near enough evidence to draw that conclusion, much less openly accuse him of it.

Michael Turton said...

I'm much more interested in attacking the assumption that political motives underlie everything.

I dunno why, since neither my commenters nor I hold that opinion.

accept the convenient causal relationship rather than to think things through and question whether there are other reasonable possibilities.

You mean, exactly what you are doing in your comment here. In fact there is another reason I phrased my comment the way I did, and if you come down off your superiority complex for a moment, it may occur to you.

For all I know, Peter might be a China shill. But I don't have anywhere near enough evidence to draw that conclusion, much less openly accuse him of it.

Thanks, Nathan. As you said, you don't have anywhere near enough evidence to accuse anyone of anything.

Not that it stopped you, of course.

Michael

frozengarlic said...

Well, this is degenerating into a rotten-tomato throwing contest. Looking back at my own comments, the tone is far more aggressive and adversarial than I intended. Perhaps this type of conversation simply isn't possible (for me) in this forum since the written word is so much more impersonal than face to face conversation. I'm going to apologize for any offence and leave this topic to another place and time.

Todd Sandel said...

I'm a professor at the University of Oklahoma and a friend of Peter Gries. I'm also a "Taiwan" scholar, having spent 10 years living and working in Taiwan on a range of projects. (Check out my work: http://faculty-staff.ou.edu/S/Todd.L.Sandel-1/vitae/publications.html)

It's wrong to say that Gries and OU are beholden to China. While we do benefit from a Confucius Institute and ties with China, we are an equal opportunity institution. Just this past week we approved a proposal to establish a "Taiwan Academy," run through Taiwan's Houston office.

Peter Gries is doing good work in Taiwan. This is another way of doing polling. The validity will become apparent on election day. My take is kudos to Taiwan for having an active medium for expressing political points of view. Wish we could say the same about China.

Todd

Michael Turton said...

Thanks, Nathan.

Thanks, Todd.

Michael Turton said...

For those interested in Confucius Institutes, I have a post here.

Michael

Michael Turton said...

Postscript:

The validity will become apparent on election day.

haha. Ma won by six percent, not eight, and Soong got a tiny percentage of the vote, not ten. This poll was, as I said, utter nonsense.

Michael