 Sometimes outliers are right. Yes, outliers are less likely to be accurate. Yes, they might simply be off by chance, but they might also reflect a new shift in the race that other pollsters haven't reported yet. They might reflect something novel and correct about a survey's methodology, and the average of other polls might not be as trustworthy as it looks. What if other pollsters found something surprising as well, but then made tweaks to bring the results into line with the consensus? That's called herding.