Trent Buskirk looks the part of a data wonk's data wonk. He stepped to the microphone and wanted to lighten the mood.
So he told a joke.
Have you heard the one about the three people from a company who went out to lunch? One is the marketing director, one the head of operations and the third, the survey researcher.
They decide to take a car. The marketing director has his foot on the gas, the ops guy has his foot on the brake, and the survey researcher is looking out the back window telling them where to go.
That drew awkward laughter. After all, the message was being delivered from a survey researcher to a room of about 1,000 of America's top pollsters and social science researchers at their biggest conference of the year.
The future of polling — and how to look out the front window instead of the back — was a major theme at the American Association of Public Opinion Research conference this past week in Denver. And one particular presentation stood out for its attempt to break the mold and threatened to rock a quarter century of how exit polls are conducted.
Exit polls are surveys conducted outside a large sample of polling places across the country, asking voters whom they voted for, their demographic information and many other questions about the issues that motivated their vote. They have been used by the AP and many broadcast networks for decades to calculate projections as results start coming in on election night.
Fox News and the Associated Press announced they are pulling out of the media consortium that has funded exit polls, as we know them, since 1992. But they officially announced, at this well-attended Saturday morning session, that they will be conducting their own polls in all 47 states that have governors and Senate races in this year's midterm elections.
The problem
Their reason for doing so? They contend, and many others agree, that there are growing challenges facing traditional exit polling, with increases in early voting, rising costs and concerns about people not talking to pollsters, therefore skewing the results.
Fox News executives decided after the 2016 election, in which early-wave exit polling on Election Day led many to believe incorrectly that Hillary Clinton was going to win, they would end their involvement with the traditional exit polls.
"We thought it was necessary to try something new," Arnon Mishkin, head of Fox News' Decision Desk, told the room of data scientists.
He pointed out that when Fox executives said they wanted to no longer participate in the consortium, he went through three stages of grief: (1) "OMG, what are we going to do?" (2) "Can't we fight this?" and, finally, (3) "Can we come up with something better?"
"There's a real problem," Mishkin added. "It's not the methodology; it's the respondents."
That's a nice way of saying the exit poll isn't polling all the right people.
Murray Edelman, a consultant to CBS News who helped develop the traditional exit polls and worked on iterations of them for 50 years, acknowledged there is a problem with Democratic bias in the exit polls. It's not intentional, and Fox doesn't accuse them of that. It stems from the difficulty of getting older voters, who lean overwhelmingly conservative, to fill out exit poll forms.
"There's an issue with overstating Democrats," Edelman said. In recent cycles, the exits have consistently shown, in early waves, a Democratic lean. That has shaped the narratives at many networks, which are on the air for hours before polls close in most states or before sizable chunks of the vote are in from most places.
It's not the only problem the exits have faced, though. During the 2016 election, for example, they underestimated the number of whites without a college degree, a key part of Trump's base.
So, endeavoring to try something new, Fox and AP quietly tested a new approach in three high-profile elections in 2017 — the governors' races in New Jersey and Virginia, and the Alabama Senate special election.
They predicted the winners correctly in all three contests.
So what are they doing exactly?
First, to call this approach an exit poll would be incorrect, as pollsters at the conference pointed out.
That's because it's really a massive pre-election poll of likely voters, conducted a few days before an election and up until Election Day. They interviewed some 5,000 registered voters in each state contest by telephone and online. (To give context, a decent statewide poll interviews about 500 people or so. This is 10 times that sample size.)
The pollsters, led by a team of respected researchers at NORC at the University of Chicago, also weight to some 20 small regions within each state. (The traditional exits only do that in five, Fox and AP pointed out in their presentation.)
After each election, they — like traditional exit polls — try to adjust to the election results to give a demographic cross section of who voted and how.
But they're not speaking to known voters, people coming out of polling places. So to call it an exit poll is not correct, and it means there is a different kind of statistical risk than the current exit polls have because they'll have to estimate who will actually show up to vote.
The AP and Fox have been pitching the idea to other media outlets, hoping to get them on board and break away from the National Election Pool consortium.
Many of the national broadcast outlets are under contract with the consortium through the 2020 elections. After that, it could lead to a historic crackup, especially if the new effort gets about the same results — for less money.
But those are a lot of ifs.
Getting it right
The Fox-AP polls have had some problems, too, in their test runs. They didn't exactly get right how some of the demographic groups voted, especially when it came to education and race, a familiar problem for pollsters.
In Alabama, for example, they got the ultimate result right, but underestimated black turnout.
The Fox-AP-NORC researchers readily acknowledge that there are tweaks they need to make — and that they're figuring it out as they go along.
"With this approach, we're starting from scratch," David Scott, a deputy managing editor at AP, told NPR. "We don't have a long history of a record. There's going to be a lot of learning as we go."
While Fox-AP-NORC got the outcomes right in New Jersey, Virginia and Alabama, some at the conference scoffed at the idea that that means these polls should definitely replace the exits.
Here's why. Take a look at the polls' pre-election estimates versus the actual results:
Here's where other pre-polls stood, according to the RealClearPolitics average:
So the Fox-AP-NORC poll did not fare much better, if at all, in these races than traditional pre-election polls. All of those were largely well within the margins of error.
The problems with exit polls are magnified, given that they help shape the narratives coming out of elections and, therefore, often the policy prescriptions afterward as political strategists and policymakers look to the exit polls to see how voters want officials to address issues.
Yet despite some high-profile misses, the traditional exit polls do get it right most of the time. For every Florida in 2000, there are hundreds of calls that network decision desks get right, based, in part, on those exit data — and often with little percentage of the vote in.
But Edelman said that those who run the traditional exit polls are already working on a fix for the problems of undersampling lower-educated voters within the poll.
Many believe that's a good thing, complaining that they have for a decade been ringing alarm bells that often fell on deaf ears. And that may be the biggest result from the competition — lighting a match under the traditional exit pollsters to make changes.
The Fox-AP-NORC experiment, many believe, is worth it — and even Edelman applauded the effort, joking with Mishkin, who was standing nearby, "I told him good things," he said, tugging his rainbow suspenders and pointing to this reporter.
Indeed he did.
"I think it's good," Edelman said. "It's a healthy thing. What they're addressing is real."
After the 2016 election, Americans were left confused and distrustful of polls. The opening joke, delivered by the conference's committee chairman, rang all too true with many practitioners there.
In an era of Big Data, bots and a political landscape with President Trump in the White House fueling calls of fake news and polls, there's a challenge for pollsters to stay relevant, to win back Americans' trust. In this new era, looking at the foggy road ahead, many at the conference said the industry needs to look at new approaches and welcome Fox and AP's effort — even if imperfect.
"We're in a moment of flux as an industry," said Chris Jackson, vice president at Ipsos Public Affairs, which conducts some of NPR's polling. "Anyone who tells you they have the single answer probably doesn't understand the scope of the problem."
Copyright 2023 NPR. To see more, visit https://www.npr.org.
