what's wrong with this survey?

A national survey released this week named my home town of Miami as where the rudest drivers in the U.S. live. There's only one problem: their survey logic doesn't work.

2500 regular drivers in 25 metropolitan areas rated fellow drivers on tailgating, anger, and talking on the phone. So why doesn't it add up?

The survey compares apples to oranges. It's not clear if drivers in Miami are actually worse than they are in LA or if people surveyed are just less tolerant of bad behavior. None of the people in the survey drove in multiple cities because they don't live in multiple cities.

If they were comparing airports, it would be a different story since the same respondent might plausibly hit three or four airports in addition to her home. But they're not. They're comparing frustration with driver behavior.

Now I'm not a market research expert, although I've frequently hired researchers and made use of survey data in product requirements. Most of the time, if numbers don't make sense, there's an obvious reason, like the survey is flawed.


Despite this, inaccurate statistics are used in trend stories all the time.

For example, when Time magazine announces that 97% of Americans won't vote for a Mormon or an atheist for president, what this generally means is they asked 900 "random" people and only 25 said yes. Usually they don't show you the question itself, which often has bias in its wording. Sometimes they ask the same questions year after year, as with the driver survey. But if they don't ask the same people or a closely matching demographic, it's not fair to claim that Minneapolis is worse this year than last. Especially when only 100 drivers—a bare minimum for statistical significance—were surveyed in each location.

With emerging trends where there's no baseline, how do you ensure that your sampling is representative? There are 300 million people in the U.S. Which 100 you choose makes a huge difference in the results, regardless of whether the polling company has any agenda or not.

A little healthy skepticism is in order. (Just ask anyone who's ever talked to a startup about how many active users they have.) And all because I find it hard to believe that drivers are ruder in Miami than in San Francisco.

Despite conventional wisdom, numbers lie all the time.





*