Sampling Tricks

The first thing to do to conduct a statistics research is to take samples since for most of the time it is impossible to survey each individual. Well, this very first step can be incredibly tricky.

Assume I surveyed 2 graphic designers and one of them is red hair. 50% of graphic designers have red hair! You really should consider dying your hair if you want to be a graphic designer. Does it prove something meaningful? If you surveyed 5, 10, 100 or more graphic designers does it show the same percentage? Well, small sample size does not give you meaningful conclusions. Because when the sample size is too small, it is very easy to arrive in such absurd conclusion. Similar example is if you toss a fair coin 3 times and get heads up all three, will you assume that the chance of getting heads up in a coin toss is 100%? Trends and rules are hidden in big numbers, so if you want a “general” conclusion you need to have a big sample. (Source: http://www.verticalmeasures.com/content-marketing-2/misleading-statistics-how-visual-data-can-go-bad-graphics/)

 

Big sampling can still go wrong if you surveyed the wrong population. True story: During Spanish-American War the death rate in Navy was nine per thousand whille civilizans of New York City was sixthteen per thousand. These figures are drawn from a fairly big sample. So Navy recruiters use this figure to show that the Navy member live longer than others and are safer. Does it mean Navy is a safer place? Do people actually live longer if they choose to join Navy?

But it is important to point out that Navy members are generally healthy young men while outside of Navy is a larger pool of people. There are sick or unhealthy people who might not have enough access to proper medication. This explains why people outside of Navy, especially in big city like New York, have higher death rate compared to Navy members. This conclusion was obtained by comparing two different groups of people and result is nonsense. (example from How to Lie with Statistics)

 

Another example. In 2000, a poll result showed that 9% of people agreed that Ralph Nader was the only candidate worth voting for, so he was expected to have around 9% votes. But it turned out that he got roughly 3%. The reason is that the samples drawn for the poll was not randomly drawn from the population. A lot of Nader supporters participated in this poll to make Nader more viable. So number of his supporters in this sample is not in proportion to the population, thus leaves an inaccurate results.

(Source: http://www.econoclass.com/misleadingstats.html)

When you see a statistical result, especially percentiles, always check:

1. Is sample size big enough?
2. Are samples taken from identical groups or representative groups? Are groups comparable?
3. Is the sample drawn properly from the population? Does the sample represent the entire population?

Want to read more about sampling tricks? Click here