How Can a Small Survey Describe a Whole Country When a Huge Poll Got It Wrong?

1. A Magazine Collected 2.4 Million Answers in 1936 and Still Named the Wrong Winner

In 1936 a magazine called the Literary Digest mailed out ten million mock ballots. Almost 2.4 million came back[2]. The ballots asked whom the person planned to vote for in the presidential election between the sitting president, Franklin D. Roosevelt, and the Republican challenger, Alfred Landon. The magazine had run such "straw polls" for years, and this one was enormous.

Its final forecast was that Landon would win 54 percent of the popular vote and 370 electoral votes. In the real election, Roosevelt won more than 60 percent of the popular vote and 523 electoral votes, carrying every state except Maine and Vermont[2]. One researcher describes the forecast error as nearly 20 points[1].

That is the puzzle for the rest of this note. Millions of answers should feel like strong evidence. Why did they point the wrong way? And why is a poll of a thousand people today treated as useful at all?

2. The Poll Went Wrong Because of Who Was Asked and Who Answered, Not Because of How Many

For decades, textbooks told one story. The magazine took its list from telephone books and car registrations, and in the middle of the Great Depression, people with phones and cars were mostly better off and more likely to vote Republican. So the sample leaned toward Landon[1]. (The mailing list also included club rosters, city directories, and lists of registered voters[2].)

More recent researchers say that is at most half the story. A survey Gallup ran in 1937 asked people whether they had taken part in the Digest poll. Studies using it found that voters with telephones and cars also backed Roosevelt, and that the people who did not return a ballot were overwhelmingly Roosevelt supporters[1]. Only 24 percent of the ballots came back[2]. One study concludes that the main culprit was non-response, meaning the people who chose not to answer[1]. Others say the low response rate and the lopsided list combined to produce the error[2]. Researchers still weigh the two causes differently, but they agree that the problem was not a lack of people.

A simple way to picture it is a class vote about a field trip. If you ask "Who wants to go?" and count only the hands that go up, more of the keen students end up in the count than in the whole class, and the answer looks like "almost everyone." Counting more hands from the same keen group does not fix that. The tilt stays, and the bigger the count, the more confident the wrong answer looks.

3. Choosing People by Chance Lets Surveyors Know How Likely Each Person Was to Be Asked

So how do surveys avoid the tilt? They do not let people choose themselves, and they do not rely on whoever is easiest to reach. They pick people by chance, using a method set up in advance, so that the chance of each person being picked is known. This is called a probability sample, or a sample chosen by chance.

The U.S. Current Population Survey is a good example. It is run by the Census Bureau and is the main source of the monthly unemployment figures. It uses a probability-selected sample of about 60,000 occupied households[3]. Census Bureau interviewers contact those households, mostly in person or by phone, and one adult usually answers for everyone in the home who is 15 or older[3]. Each household stays in the survey for four months, leaves for eight, and then returns for four more, so that results month to month can be compared while the sample is gradually refreshed[3].

The key idea is a fair lottery. Think of a classroom that wants to know the favorite recreation game of the whole school. Asking only one class might give a funny result, because one class can be unusual. Drawing a few names from every class gives a better miniature of the school. A real survey design is more careful than a simple draw, but the goal is the same: the sample should look like a small copy of the whole group.

4. Chance Still Causes a Small Wobble, and Quadrupling the Sample Only Halves It

Even a perfectly fair lottery will not match the whole country exactly. Each draw wobbles a little. This is called sampling error, the wobble caused by chance, and the good news is that statisticians can calculate how large it is likely to be.

Here is a standard rule of thumb for a yes-or-no question, assuming a simple random sample and an answer near 50 percent. With 1,000 people, the range is roughly plus or minus 3 percentage points. If you took a fresh sample many times and built such a range each time, about 95 percent of those ranges would contain the true value. With 4,000 people, it is about plus or minus 1.5 points. That is our own arithmetic from the usual formula, not a figure from the sources below, and real surveys have more complicated designs.

Notice what happened: the sample became four times bigger, but the wobble shrank only by half. The error falls with the square root of the number of people, so every extra bit of accuracy costs more than the last. A sample of 2.4 million, by this same formula, would have a random wobble of well under one point. The Digest poll was off by nearly twenty. This is why the wobble is the small part of the story. The big problem was the tilt, and the formula does not measure tilt at all.

5. No Amount of Extra Data Fixes a Biased Sample, and Even Good Fixes Have Limits

Researchers have asked whether the 1936 poll could have been rescued after the fact. In 2017, Sharon Lohr and J. Michael Brick showed that the ballots asked how people voted in 1932. If the pollsters had used that to adjust their results, a technique called weighting (giving some answers more weight than others) that was already known in the 1930s, the poll would have projected Roosevelt as the winner. They also stress that even after the adjustment, the bias in the estimates was still very large[2].

That is a useful lesson for today. Non-response is still a worry for modern polls, and pollsters often weight their data to try to make up for the people who do not respond. Weighting can help, but it depends on assumptions about the people who stayed silent, and nobody can check those assumptions by asking the people who did answer.

The result is two kinds of error. Chance wobble is predictable, and it shrinks as the sample grows. Tilt from who was left out is hard to measure and does not shrink with size. A survey earns trust when it keeps the wobble small and works hard to keep the tilt small. That is why careful surveys spend so much effort on how they choose households, how they ask questions, and how they follow up with people who did not answer at first.

6. Try a Small Fair Draw in Your Own Class or Family

With an adult, pick something to measure in a group of about 30 people, such as how many walk, cycle, or take a bus to school, or which of several snacks they like best. Keep answers private and compare only the totals. First ask for volunteers and note the answer they give. Then write every name on a slip of paper, mix the slips, draw 8 without looking, and ask only those people. Compare both results with the answer from asking everyone. You will not get the same outcome every time, so repeat the draw a few times and see how much it wobbles.

Notice which method tends to drift in one direction. The sources below are the best next step, especially the Lusinchi and Lohr papers on the 1936 poll. A good question to explore next: how do pollsters today guess what the silent people think? We did not settle that here.

Sources

Revision History