Here at 538, we think a big part of our job during election season is to explore and explain how much trust you should put in everyone who tells you who’s going to win. More than anyone, given the amount of data they produce and the voracious appetite of the press and public, this includes pollsters. That’s why we do things like publish assessments of pollster accuracy and transparency And create complex election forecasting models to explore what would happen if polls were as wrong as they have been historically.

Polls are also important to the reporting we do here at 538, which is rooted in empiricism and data. Furthermore, the quality of the public opinion data we obtain is important not only for predicting election results and doing political journalism, but also for many other aspects of our democratic process.

Suffice it to say, if polls become more or less accurate, the public should know. And now that the 2024 election is in the rearview mirror, we can get our first look at how accurate the polls are.

Just a note on scope before we begin: In this article, I’ll take a broad look at the poll results in states where the results are final or near final. This means that we will not yet assess the accuracy of national polls, given the number of votes still to be counted in California and other slow-counting states, and we will not assess the accuracy of individual pollsters, which we will do when we update our pollster ratings next spring.

Polls in 2024: low error, medium bias

Despite the first story swirling in the media, 2024 has been a pretty good year to be a pollster. According to analysis of 538 polls conducted in competitive states* in which more than 95 percent of the expected votes were counted as of 6 a.m. Eastern Time on November 8, the average poll taken over the three final weeks of the campaign missed the margin of the election. by only 2.94 percentage points. In the seven major swing states (except Arizona, which has not yet reached a 95% reporting rate), pollsters did even better: they missed the margin by just 2.2 points.

This metric, which we call “statistical error,” measures how far behind the polls were in each state, regardless of whether they systematically overestimated support for a candidate. And by that measure, the state-level polling error in 2024 is actually the lowest it has been in at least 25 years. For comparison, state-level polls from 2016 and 2020 had an average error close to 4.7 percentage points. Even in 2012, which turned out to be a good year both in terms of polling and election predictions, the poll results missed the election results by 3.2 percentage points.

At this early stage, we can only speculate as to why the error was so small this year. One reason could be that pollsters have mostly abandoned polling using random digit dialing – a type of polling that has grown in popularity. recently serviced to generate results that swing more wildly from one survey to the next than other methods. Selzer is a notable pollster who still uses RDD. & Co., which saw Vice President Kamala Harris lead President-elect Donald Trump by 3 points in his final Iowa poll This year. Trump ended up winning the state by about 13 points, making it a 16 point error. It seems possible that Selzer’s poll there were too many Democrats and college-educated votersfactors the company generally does not attempt to correct due to Selzer’s philosophy of “keeping dirty hands out of the data” (to be fair, this approach had worked perfectly until this year; Selzer is one of the highest-rated pollsters in 538 pollster ratings ).

Quinnipiac University, which also uses RDD, also generated polls that didn’t seem consistent across states, although they ended up being closer to the result than Selzer. Meanwhile, other leading pollsters that previously used RDD have now I stopped using the method. This includes ABC News, which after releasing an RDD poll that found current President Joe Biden in front of Trump by 17 points in Wisconsin in 2020 (something the pollsters behind the survey rightly identified as an outlier when it was released), now sources its supplies from Ipsoswhich conducts online surveys of respondents who are recruited randomly by email and telephone.

Another factor is that pollsters are increasingly balancing their samples on both demographic and political variableslike the revoked vote of individuals in the last election. Although this may cause strange resultshe generally stabilizes polls And produces fewer outliers than would be expected by chance alone.

According to our preliminary results, pollsters who used this aggressive modeling approach had fewer errors than others. Although only a loose approximation for further analysis, we found that pollsters who conducted their surveys with online probability panels surveyed people using automated calls or included text messages or phone calls as part of a larger mixed mode. the sampling design tended to use more complex weighting schemes (and relied particularly on recalled voting) – and also had lower errors than pollsters using more hands-off modes:

But the news is not all good. Even though the polls had a historically good year in terms of errorthey had average to poor in terms of statistical biaswhich measures whether polls miss the result in the same direction. By our calculations, state polls overestimated support for Harris by 2.7 points on average, at the margins of competitive states.

This is less than the statistical bias of the 2016 and 2020 polls, which underestimated Trump by 3.2 and 4.1 points, respectively. But this figure is higher than that of the elections of 2000, 2004, 2008 and 2012.

This is not good news for pollsters. That means they haven’t entirely resolved their 2016 and 2020 problems of getting enough Trump supporters to turn out to vote. While these issues may have been mitigated by pollsters weighting their data more aggressively or improving their sampling plans, they are still clearly present. You can really see this if you look at the poll bias trend in competitive states from 2016 to 2024:

Even though pollsters managed to reduce their bias in some states, notably Wisconsin, between 2020 and 2024, the trend in the industry is still the same: Pollsters are having trouble reaching the types of people who support Trump.

You should expect errors in the survey

While bias in the polling industry is troubling, it is not necessarily unexpected, especially after the last election. And it bears repeating that a 3 point error on the margin is indeed very small historically. Political pollsters have designed a tool that, on average, can measure the public opinion of hundreds of millions of people to within 1.5% of its “true” value (by converting vote margin to vote share). When you think about it that way, it’s actually remarkable that the polls are as accurate as they are.

After the 2020 election, a year in which U.S. pollsters had their worst performance since 1980, the American Association of Public Opinion Research (the professional society of pollsters and survey researchers) issued a warning to people trying to predict election results in 2022 and 2024. “Polls are often misinterpreted as accurate predictions,” he says. “It is important in pre-election polls to highlight uncertainty by contextualizing poll results in relation to their accuracy… Most pre-election polls lack the precision needed to predict the outcome of semi-close elections. “

In other words, polls simply aren’t up to the task of decisively determining the outcome of a close race before it happens. In such a case, the margin between candidates would be too small for observers to reliably conclude that a candidate was ahead, given the uncertainty inherent in polls.

Let’s put this warning in the context of the 2024 elections. At first glance, this may seem as if the polls had a bad year because they predicted a close election and Trump seems to be heading towards a 312-226 victory in the Electoral College. But as I wrote last weekbecause he led in the Sun Belt swing states and was tied in Pennsylvania, polls didn’t even need to underestimate Trump for him to win the election. And, I warned, if they underestimated him by 2 points – which would be small compared to other historical misses – he could sweep all seven swing states.

Well, it looks like that’s exactly what happened. In fact, it was the modal result in our final forecast. AAPOR’s warning is more relevant today than ever.