Surveys are great tools to find out information about some population that can help guide your business decisions. But bad surveys generate bad data and result in bad decisions. Let’s continue with our look into the first step of any survey: picking out your targets.
In the previous post we looked at the total population we are interested in, the sampling frame that represents the list of the entire population, and the sample that we will actually use for the survey. We rarely try to survey an entire population (saturation sampling) unless it’s a small one, and will usually use a small subset, or sample. Selecting the sample from the full sample frame can be done by selecting members randomly (random sampling), or by picking every Nth member (systematic sampling). We looked at the danger of ending up with a sample that does not accurately represent the make-up of the total population and investigated the use of stratified sampling as a first step to allow us to end up with the right % makeup for key parameters like gender or age or income.
Moving on from there, we should also consider that there might be parameters that make someone either a good selection or a bad selection. These will make up a set of eligibility criteria. Rules that must be met for an individual to be included are called inclusion criteria. An example of this might be that someone has to be alive (at least, in most states) to be considered an active voter. Rules that must not be met to be included are known as exclusion criteria. An example of this might be that unregistered voters are excluded from the list of active voters. And yes, whether something is an exclusion or inclusion criteria depends largely on how you word it.
So far we have been looking at closed populations – ones for which we can obtain a complete sampling frame. There are also many instances of open populations – ones where we do not have a complete sampling frame available to us. For example, we know with certainty every member of a school’s student population. We have no idea who the members are of the set of homeowners with blue carpets in New York. We can certainly create a list of some of them using a variety of sources, but it will not be an exhaustive list. Selecting a meaningful and representative sample from this group is challenging because we don’t know the characteristics of the whole population. We have no idea how many men vs. women own blue carpet in total, just in our limited sample. So to create a useful sample that represents the general gender makeup of the total population is impossible. Nevertheless, we can still learn interesting and useful things from these samples.
There are two basic ways to deal with open populations. The first is to recruit members to participate on a panel of respondents, and the second is to restrict the population to a specific sub-set, like visitors to a certain web site, for example.
Pre-recruited panels involve work to create and then maintenance to hold on to. If you are large enough and have enough resources you might be able to grow such a community. Or you might engage a company that does maintain a stable of panelists and use them. If you want to recruit a blind panel that doesn’t know who you are, as opposed to a branded one that does know who you are, you almost have to go with an outside supplier. You do need to be careful that the answers you get from a panel are “real world” answers and are not skewed by their familiarity with surveys.
It should be noted that participating on a branded panel also has the benefit of helping to improve brand loyalty by making the panelists feel like they have a connection with the brand.
Gathering your survey respondents from a specific site (online or physical) is known as “intercept sampling” and can be as simple as popping up a window to all your website visitors (or every Nth visitor) and asking them to participate in a short survey. Here, too, you need to be careful about using results to make assumptions about the general population. People that visit your web site already are a specific type of person (by displaying interest in your products or company) and therefor not a completely random sampling.
In fact, it is important to always keep in mind that there will always be conditions and filters placed on your sample, no matter what you do. If you are surveying employees, you might be able to force them all to answer the survey. But in most other cases, you cannot force people to even see your survey, let alone answer it. When you send out a survey, some people will never even get it whether that’s because their email filters catch it as spam, or they simply toss an unrecognized envelope in the trash (physical or electronic). So right there you have further limited your sample to “people that do not have strong spam filters”. There will be many people that simply don’t like to answer surveys, so of the people that open your email, you are selecting “people that are willing to answer surveys”. Even if they are willing, they might not have the time to answer a survey and so you end up with “people that are not too busy to answer a survey”. Finally, you probably offered an incentive and that will appeal to some people more than others. If your incentive is a drawing to win a vacation to Alaska you have now also included people with a propensity to gamble (they feel positive about a random drawing) and people that like Alaska.
Do we care about all of these implicit selection filters? Some, probably not. But if we are conducting a survey by email about overworked employees, keep in mind that they very target we are seeking is probably too busy to answer the survey. If we are looking into attitudes about beach goers to types of sun block then offering a chance to win a trip to Alaska is probably going to appeal more to the wrong crowd. In these two examples we won’t eliminate the target we are going for but we will reduce it. Offering a membership in the bacon-of-the-month club (yes, that’s a real thing) as a prize to members of the Jewish community is an extreme example, but the world is full of people who have done sillier things. Just ask yourself at each stage, “How will this impact who responds to my survey?” Some answer will matter, some won’t, but it’s important to know.
That is a quick overview of some of the issues with sampling and sampling methods. There is much more to say about that, and many other survey topics to investigate that can help you to make sure your end results are both accurate and meaningful. Otherwise, garbage in… garbage out.