Wednesday, December 08, 2010

The 2004 Presidential Election TVBS Exit Poll

I was arguing about the recent assassination attempt just prior to the municipal elections this year on Forumosa.com, when I stumbled across this statistical paper on the exit poll conducted by TVBS after the 2004 election in which Chen Shui-bian was shot by a disgruntled KMT supporter the day before the election. This exit poll was the first ever for an election in Taiwan. ADDED: One of the authors is a TVBS poll center employee.

According to this paper, the poll was conducted from 8:00 am to 3:00 pm on the day of the election. More than 13,000 individuals were interviewed by trained poll-takers using a paper and pencil questionnaire. The paper describes:
The poll center of TVBS collected the exit poll data and sent them to Mitofsky International by Internet at 3:38. Mitofsky International processed the data analysis and sent the result back to TVBS at 3:55, which predicted the support rates of President Chen Shui-bian (Democratic Progressive Party, DPP) and his opposition Kuomintang (KMT) challenger Lien Chan were 47% vs.53%. Since precincts in some regions, such as Taizhong, Kaohsiung, were not closed until 5:00pm, TVBS decided to delay the projection. Until 5:12pm, they projected that the supporting rates of two candidates were 47% v.s.53%. However, since the margin of difference was less than 8%, according to the principle of the projection of exit polls, they did not declare who was the winner.

Then TVBS sent the quick counts (real voting results) of those precincts which were sampled in the exit polls to Mitofsky International, until 5:20pm the real results of 80 precincts had been sent to U.S. At 5:36pm, based on these 80 precincts’ real counts, Mitofsky International revised the prediction as 50% vs. 50%. At 6:01 pm, they received the data from real counts of all 150 precincts sampled and confirmed the revised support rates (50% vs. 50%).
Note that a couple of hours before the poll counting had finished, based on returns from 80 and then 150 precincts, Mitofsky International already called the election at 50%-50% though the TVBS poll had Lien-Soong ahead 53-47. The paper notes that revisions to exit polls are common for large geographical areas. In sum, what this paper shows is that contrary to some of the propaganda floating around, the TVBS exit poll, once revised with a few early returns, showed the KMT and the DPP in a statistical dead heat.

What were the problems with this poll? They are a legion. The sampling error was rather larger than in American surveys. Non-sampling errors are more important; they show a statistically detectable bias in the poll against the DPP, which accounts for its skew toward the KMT. This is probably not the result of TVBS' innate bias against the DPP but is the result of several issues outlined in the paper. For example, there is the problem of incentives for participation in the poll. Read closely:
In 2004 Taiwan presidential election exit poll, they boldly employed incentives with intent to increase the response rate. They used magnets with logo of TVBS, the sponsor of the survey, as incentives (worth about $3 each). From the aspect of advertisement of the company, it was a good way to enhance the company’s image. TVBS has a relatively neutral image among media in Taiwan, which helps it get trust from the voters.
Not only were they handing out TVBS paraphenalia, but the poll-takers at the individual precincts were all wearing caps with "TVBS" emblazoned on them. Imagine how a DPP voter might react to being accosted by representatives of a rabidly pro-KMT TV news station for polling information. (ADDED: TVBS did not become openly pro-KMT until after this time). Further, the Chen Administration had sent down word that its voters were not to participate in the poll.

In addition to these obvious dampeners to participation by DPP voters, the exit pollsters were forced to stand 30 meters from the polling places. It is known that in exit polls error increases significantly when distance exceeds 25 feet/8 meters. The paper does not define "polling place" but given that polls are conducted in schools and community centers where the voting booths are deep inside the building, it is likely that in practice distances were even greater than the 30 meters suggested here. For example, I stopped by the polling station in my neighborhood during the election, and non-voters were being kept outside the school, roughly 50 meters from where voting was taking place.

That last issue may explain why there are no exit polls for Taiwan elections, because under the current rules that govern access to polling stations on election days, pollsters cannot get close enough to make them reliable.

UPDATE: Mitofsky himself noted in a piece on exit polls that the student poll-takers were not well-trained:
Regrettably, bad sampling and no estimation are the rule and not the exception in emerging nations. It seems to be common practice to tabulate the results in SPSS with little or no weighting. There are other types of missteps, too. Interviewer selection and training in recent exit polls in Venezuela and Taiwan led to biased results. Recent partisan exit polls in Azerbaijan and Mexico led to deliberately distorted results.
_________________
Daily Links:
_______________________
Don't miss the comments below! And check out my blog and its sidebars for events, links to previous posts and picture posts, and scores of links to other Taiwan blogs and forums! Delenda est, baby.

15 comments:

Ben Goren said...

TVBS has a relatively neutral image among media in Taiwan, which helps it get trust from the voters.

... rofl ... yeah and the Pope is an atheist.

Taiwan Echo said...

From wiki (http://zh.wikipedia.org/zh/2004%E5%B9%B4%E4%B8%AD%E8%8F%AF%E6%B0%91%E5%9C%8B%E7%B8%BD%E7%B5%B1%E9%81%B8%E8%88%89):

"在選前兩個禮拜,根據各個不同媒體、智庫、和政治團體所發布的民調,雙方幾乎是不分上下。台灣智庫所公佈的數字是陳呂40.4%、連宋39.5%(3月8日);民進黨的數據是陳呂37.6%、連宋36.1%(3月7日);國民黨的數據是陳呂29.8%、連宋34.8%(3月5日);世新大學的數據是陳呂27.3%、連宋27.1%(3月2日)"

Polls two weeks before the 2004 election (Chen-lu vs Lien-song):

Taiwan Thinktank: 40.4 : 39.5
世新大學: 27:3: 27:1
DPP: 37.6 : 36.1
KMT: 29.8 : 34.8

Those polls (except that of KMT) showed that the 2004 shooting delivered no significant effect.

Many people denied most poll results but that of KMT's. If they had taken all results into considerations, they wouldn't be able to accuse the shooting.

Michael Turton said...

Yeah, the poor KMT polling in '04 was one reason they went to more scientific systems in the most recent election.

Thanks for the additional polls!

Anonymous said...

Those two kids you were debating with on Forumosa (Betelnut and Paogao) are idiotic morons.

Okami said...

Any links to the Nantou wildlife refuge like maps or English language press?

Anonymous said...

It's not correct that most polls showed the race as tied before the shooting. Most polls showed Lien-Soong with a 8-10% lead until two days before the election. TVBS has a nice time series in this report. Scroll down to look at the graph.
http://www.tvbs.com.tw/FILE_DB/files/osaka/200404/osaka-20040427193847.pdf

As I remember, ERA conducted surveys throughout the blackout period and showed roughly the same trend. Unfortunately, their website is not responding right now, so I can't cite those numbers.

Also, you have to remember that TVBS in 2004 was not the TVBS of today. Then, it did have a much more neutral image. It may surprise many of you to know that in the mid- and late-1990s, TVBS was considered the pro-DPP station. It started out as a very professional news organization, and it was the first channel to provide news without a pro-KMT bias. In later years, TVBS reoriented itself as a pro-KMT channel, but in 2004 survey respondents would not have necessarily had that knee-jerk reaction.

Anonymous said...

Another thing. The 2004 exit poll was the only presidential exit poll, but it was not the only exit poll done in Taiwan. There was an earlier exit poll done for the 1998 Taipei City Mayoral race. If you recall, that was the heavyweight battle between two future presidents.

I found the dataset for the 1998 exit poll on my hard drive. Unweighted, it shows Ma winning 50.4 to 44.9, which is almost exactly the final result. (I think the sampling method was designed so that the data didn't need to be weighted.) I don't remember which network was involved with this exit poll. I think it might have been TVBS.

There was also a proto-exit poll that was done in conjunction with ERA. I can't remember exactly, but it must have been the 2000 presidential election. The idea was to get the election result an hour or two before the votes were finalized. We selected about 200 precincts from around Taiwan, and ERA sent a reporter to each one to watch the vote counting. As soon as they had a count, they called in the results (on new-fangled cell phones!), we weighted the data, and produced a number. Unfortunately, it didn't work very well. ERA tried to save money by having one reporter for every two precincts. Since the precincts weren't near each other, that meant a lot of time was eaten up in travel time. Also, the counting went a lot faster than we expected. By the time we had enough data to produce an estimate, we could announce exactly what every could already see in the actual tally. But as I recall, our estimate was pretty good. The only bright side is that I resisted all pressure to go on TV and announce the results. By that time, I had finally learned to keep my face off the tube.

Anonymous said...

Keep in mind that one of Mitofsky's most important goals is to cover his ass by blaming locals for everything that he can. It's fascinating that (1) they blame the locals for lots of methodological failings and (2) they claim that their final estimate (50-50) was exactly right. I suspect there was some fudging involved to protect the reputation of the firm.

Unknown said...

Re: "Having children may be beneficial to one's mental health. Clearly these researchers are childless."

And clearly the researches are not women who have children, either! I mean, jeesh!

Michael Turton said...

Keep in mind that one of Mitofsky's most important goals is to cover his ass by blaming locals for everything that he can. It's fascinating that (1) they blame the locals for lots of methodological failings and (2) they claim that their final estimate (50-50) was exactly right. I suspect there was some fudging involved to protect the reputation of the firm.

Haha. I wondered. But it is hard to imagine how they fudged the results at 5:36, unless the paper is lying about the timeline.

Most polls showed Lien-Soong with a 8-10% lead until two days before the election. TVBS has a nice time series in this report. Scroll down to look at the graph.
http://www.tvbs.com.tw/FILE_DB/files/osaka/200404/osaka-20040427193847.pdf


FG, that time-series is only for the TVBS poll, which has never been a reliable poll. I can't recall even a single credible poll that had Lien-Soong leading by such large numbers. Do you know of any others? In any case polls almost invariably undercount DPP voters, especially TVBS polls, so an 8% gap in the TVBS signals that the results are going to be close in real life. TVBS is quite predictable that way.

As for TVBS, I didn't pay much attention to them until 2005, after they had already become ardently pro-KMT. I seem to have some vague memory of them being more neutral in the late 1990s. One problem is that we don't and never have owned a TV, so I remain out of that cultural loop.

Michael

Michael

Michael Turton said...

Also, the Mitofsky claim that untrained interviewers were responsible for the poll biases is indeed self-serving, but so is the paper's claim that the interviewers were well trained. Because I know how half-assed student work is here, I tend to side with Mitofsky on this one. :)

Michael Turton said...

Looked it up. The great shift for TVBS must have occurred in 2005 when it was purchased by Hong Kong Chinese owners.

Michael

Michael Turton said...

The oldest poll on the ERA site is from last year

http://survey.eracom.com.tw/item/p.jsp?id=24&pageno=24

Anonymous said...

Unfortunately, I didn't collect poll results from the 2004 presidential election because I didn't plan to write a paper about it. I have lots of polls on the 2004 legislative election though :) Actually, this is one of the main reasons that I am trying to collect polls on my blog -- I find that our memories shift over time and I want to have some solid historical data.

As I remember it, ERA was doing polls throughout the blackout period and I saw that data after the election. They showed a similar gap to the TVBS data (as did UDN) until the day before the election. That last day showed the race had become a dead heat, just as the TVBS data does.

Why was 3/19 so traumatizing to blue voters? One reason is that they were not psychologically prepared to lose. Every poll for months had shown Lien with a comfortable lead. Suddenly there was a shooting and they lost. There were a lot of people in absolute shock because the election had seemed in the bag to them just 30 hours ago. It was a shock to the system.

At least that's how I remember it.

Anonymous said...

Most polling organizations give interviewers a couple of hours of training. The academic surveys (such as Taiwan Election and Democratization Survey) usually do a full day of training. Of course, that is no guarantee of quality. In international perspective, this is about average (I think). The American National Election Studies Survey uses professional (full-time) interviewers, and in comparison to them, the Taiwanese interviewers were certainly not well-trained. However, I doubt many surveys Mitofsky associates himself with around the world are up to that standard.

Do you think Mitofsky worried about these sorts of issues before there was a controversy? He could have demanded to know exactly how the data would be collected, and if he was not satisfied with the answer, he could have refused to attach his name to the project. That did not happen.