Analytics without intelligence: Simpson’s paradox

You should never use analytics without intelligence for so many reasons. One of them involves Simpsons Paradox. This, quite simply, is the odd effect that combining data sets in just the right way can actually reverse the individual results. That means that you can often decide whether you want to prove or disprove something simply by deciding how to analyze your data.

The paradox is named after Edward Simpson who first described the effect in a paper in 1951, although the effect had been noted and mentioned by earlier mathematicians. Quite simply, it says that if you have two possible outcomes, A and B, and you run two tests to see which is more likely. In the first and second test, A can come out the clear winner. However, when you combine the data from the two tests together, then B can be the winner. How can this be true?

Let’s take a very simple case. I want to find out who is the better marathon runner: Bob or Sue. I tell each of them to run 5 races over the next two weeks. The first week, Sue is feeling a little run down so  she only runs 1 race and doesn’t win. Bob runs four races and wins one of them. Clearly, for week one Bob is better than Sue. Sue wins 0% of her races, and Bob wins 25% of his races. In week two, Sue runs 4 races and wins three of them. Wahoo! Sue won 75%. Bob runs his remaining race, which he also wins. Also wahoo! Bob won 100% of his races. In both weeks, Bob had a higher winning percentage than Sue. Bob can honestly say that each week, he wins a higher % of races run than Sue. Poor Sue.

But wait… what happens if we look at the total over the whole trial period, and not just by each week? Sue won nothing the first week and 3 the second week for a total winning of 3 out of 5, or 60%. Bob won one the first week and one the second week for a total of 2 out of 5, or 40%. At 60% success, Sue is clearly the better runner of the two and can claim that in the trial, she had a higher percentage of wining than Bob. Poor Bob.

We can claim victory for either Sue or Bob, using the same test results, depending on whether we want to focus on total results or who won each week. If we have all the data to examine, we see what’s going on fairly easily. But if we are simply reading a promotional claim and all we know is that either Bob wins each week or that Sue won the test… we have no way of knowing that something is fishy.

What happened? The effect seen here is caused by a “lurking variable” also called a “confounding variable”. In this case, the confounding variable is the number of races run each week. They are not the same, and it’s not a fair comparison. We should be looking at total races run, not who had the higher rate of success each week. And yet, asking who had the higher rate of success each week is a perfectly reasonable thing to ask, even if in this case it’s the wrong thing to ask.

When you are analyzing data, it is so important to understand what your data represents, and what you are trying to decide. The good news for Bob and Sue’s promoters is that both can lay claim to success in this trial, depending on how you analyze the results. However, if you are trying to decide who to send to represent you in the Olympics, you need to understand the data and what the results mean. If you ask the wrong question (“who had the highest winning percentage each week”) you will get an answer that, while technically correct, is misleading because you were not aware of the confounding variable (races run per week for each runner). Analytics cannot be done in a vacuum – you need to understand the data and understand what you are trying to prove. Only then will you ask the right questions, and only then will you correctly interpret the results as they apply to your specific business challenge.

Posted in Data and Analytics | Comments Off on Analytics without intelligence: Simpson’s paradox

Privacy? What privacy?

Google sees all. This is o.k. because it uses this information to better serve ads that match your interest. It is not, however, a perfect system.

One of the problems with tracking search and other behavior to serve ads is that you will often need to look up information that actually has nothing to do with your purchasing intentions. You might be researching a topic for work, or looking at a location for a school project. That doesn’t mean you want to keep seeing ads for them since you are not going to buy the product or travel to the location. On top of that, after you do buy something Google rarely know this and so you continue to get bombarded with ads that are no longer useful.

Still, you’ve accepted that Google is tracking your every move, but that’s o.k. because you trust Google (mostly) and it serves a purpose most of the time – getting ads that you actually care about.  And fortunately, you do have some control over the topics Google thinks you are interested. Go to “My Account”, and under “Personal info & Privacy” click on “Ads Settings”. Here you have some control over topics that you have been tagged with, and can even opt out of the whole program altogether. The down side is that you then go back to getting random ads that are not useful to you. One nice feature is that if you opt out, your whole topic history is deleted. If you then turn it back on again, you start with a clean slate.

But what about all the other ad serving companies out there? Google is not the only game in town, and unless you look closely at each page’s source code (or use tools to control ads), you don’t really know who is tracking you. For that you should check out the Digital Advertising Alliance Consumer Choice Page. There are 127 different companies participating in this organization and you will see which ones are currently tracking you. This includes Adobe, Google, Facebook, Amazon, and a raft of smaller ad companies you’ve never heard of. But they’ve heard of you! At least, they know your browser. The url to go there directly is http://www.aboutads.info/choices/  and you can specify which companies, if any, you want to opt out from. Note that these are only the companies that have voluntarily participated in this, but all the big ones are in there.

Great! Or so it seems, but one big caveat to remember is that this does not actually mean they will stop storing data on you (although for Google it does), it just means that they won’t serve ads based on that information. If you want privacy, get a blocker like Ghostery at https://www.ghostery.com/ .

Unfortunately cookies are not the only way to track you. There are other ways to fingerprint your computer besides cookies.  Your browser, your system environment, even your graphics card behavior can be used to create fairly accurate fingerprints for each unique system. Not as good as a cookie (and really, what is, except maybe for cake?) but still pretty close. Using a system like Tor (https://www.torproject.org/ ) helps by blocking a lot of the information needed to accurately fingerprint you, but nothing is perfect.

Going beyond just browsing, there are lots of companies tracking what you do on their site. These are companies that offer a free service but make you log in to access it (Facebook, for starters). As the old saying goes, if you are not paying for something then you’re not the user, you’re the product. A site that offers you a free service or free information about much of anything – recipes, company data, whatever, is also tracking what you are looking at. If that company makes you log in to access the information, even if it is free, they know exactly who you are and what you are looking at. That might also be harmless but you do need to keep that in mind.

One more recent thing to watch out for are browser extensions., You can get browser extensions that provide extra information about the content of any web page. There are two things to consider here. First of all, most of these only offer a minimal amount of information and are really just ads for their full info. They will often give out a little actual information, tell you that they have more information, and give you a way to buy the extra data. That might be useful, but just keep in mind that it’s not really free information, it’s an ad.

What is far more disturbing is that many of these extensions give you extra information about all the web sites you visit. That means that they are tracking *EVERY* web site you visit. Every site. And most of them require you to log in to a free account to use the service. So they know who you are, and they know *EVERY* web site you visit, all day long. Let that sink in. That is a huge amount of private information that you are giving away. Make sure that the extension works on-demand only (so you have to tell it go fetch the info instead of it automatically presenting it) and if you are not sure, you should disable it when you are not using it. Better yet, just don’t use it.

So in the end, you are going to be tracked by a lot of different companies who are gathering a lot of information about you. In some cases, you have no choice if you want to use what the web has to offer. Google gives you search information, and they track your searches. Facebook lets you share information with friends, but they track you and make you log in. That’s the trade-off you have to make. Just make sure you make it knowingly and are aware of the data you are sharing with these different companies.

 

Posted in Business, Life, Marketing | Comments Off on Privacy? What privacy?

4 Easy Steps To Being A Video Star

Presenting to a video camera is similar in some ways to giving a live presentation but in other key ways it is very different. Here are a few tips to guaranteeing that you look like a rock star when it’s your turn.

I really shouldn’t even need to say this but obviously video is an important part of the marketing world today. Think you don’t need to ever go in front of a camera yourself? Think again. Explainer videos, recorded product demonstrations, and even narrated presentations give a wide reach with an on-demand medium. But watching several minutes of slide after slide, even when narrated well, gets boring. Adding a personal touch by alternating between slides and shots of a live person talking make the end result far more engaging. Assuming of course, that you don’t look like a carboard cutout. Here are a few tips to avoid that.

  1. Know What To Say

You don’t want to appear unprofessional, so unless you are actually a stand-up comic in real life, you are going to want to have a script to work with. Nothing looks worse in a recorded video than “umm”ing, repeating yourself, forgetting key points, and appearing disjointed. Should you read the script? No. Unless you are a TV personality and used to reading from a teleprompter, you will come off wooden at best and it will usually be obvious that you are reading from a script. Write out the script, yes, read through it a few times. Then make up a single page of notes that you can read from (more on that later). Bullets only – just speaking prompts so you don’t forget the flow or any key points. Use that as an aid, but not something to read.

  1. Practice, Practice, Practice

Don’t assume that because you wrote the script and read through it a few times that you will sail through the recording. Try reading off the notes a few times. Most importantly, record yourself during one of your practice runs, and then watch it. Have someone else watch it too – someone who is not afraid to give you critical feedback. “Wow, that looks great” is not helpful feedback. Find someone who will pick it apart. Do you have any unconscious habits? Do you fidget? Do you say “umm” between each sentence? Many people do these things without even being aware of it. If you don’t have access to the actual environment that you will be recording in that’s o.k., even doing a few dry runs at home in front of your cellphone will let you know what you look like and sound like. A few practice sessions is good, but some people will even do 20 or 30 dry runs. You can only get better with each practice.

  1. Setup For Success

Make sure that you get some practice time at least once in the actual environment that you will be recording in. Whether it is a studio or a conference room or just your office. Make up a large poster board with your key bullets on it and attach it just above or below the camera to look at. Make sure you can see it easily, but try not to stare at it. Crisp and concise bullet points will help here – large and simple so that you can see them out of the corner of your eye. If you absolutely have to look at the poster, at least keep it as close to the camera as possible. If you actually have access to a teleprompter this is even better because then you never have to take your eyes off the camera. But again, unless you have experience reading from a script directly, only put your bullet points on the teleprompter.

  1. Be Alive

Keeping up a good level of energy can be difficult when you are presenting to a camera. It’s not live, it has no reactions or emotions, it gives you no feedback. Do not present to a live audience. Yes, that helps make you less dry, but it will be obvious that you are looking at something or someone off camera.

Pretending that you are presenting to the camera person can help, but only if they are directly behind the camera. If they are off to the side you won’t be able to help looking at them and away from the camera, and it will show on the video. Without anyone to present to, you are just going to have to mentally think of the camera as a person so that when you talk to it you feel like you are talking to someone. This is where practice will be particularly important.

The obvious exception to the “no audience’ rule is when you are, in fact, presenting to an audience – one that we can see and/or hear. Then it’s o.k. to look at the audience and not always at the camera because then we are recording you speaking to the audience, not to the camera. But in that case it is important to see the audience or else it will just look like you are speaking to empty space off-camera.

These are a few key things to remember when you have to go in front of a camera. You may not ever be great, or perhaps you might, but with a little practice and a few key tricks you can certainly get better. There’s a great line in the movie “All That Jazz” where the director tells a struggling dancer, “I can’t make you a great dancer. I don’t even know if I can make you a good dancer. But, if you keep trying and don’t quit, I know I can make you a better dancer.”  And better is a good start.

Posted in Life, Marketing | Comments Off on 4 Easy Steps To Being A Video Star

Reviews are awesome – or are they?

Reviews and endorsements are great tools to help you make informed purchases, except when they’re not. Here are a few tips to keep in mind that will help you separate real reviews from fake news.

Reviews are everywhere – Amazon product reviews, Yelp restaurant reviews, Glassdoor company reviews. At their best, they provide valuable insight and information that can help you make an informed decision. At their worst, they can be misleading tools that suck you into a morass of bad choices. The problem stems largely from the open and anonymous nature of a review. When you are allowed to use a pseudonym, or even no name at all, we know nothing about the validity of the review. Even if you use a real name and are a real person, we know nothing about your motivation. And that means we don’t know whether to pay attention to your review or not. Unfortunately, there is no magic answer that fits all cases, but there are a few things to watch out for. Here are four things to consider when reading any review.

1: Strip off the outliers. First of all, does it seem reasonable? Looking at a company review that simply gushes on and on about all the pros of the company, gives it a 5, and lists “none” for cons. Or the more engaging (and my favorite) response for cons “I honestly can’t think of any”. Why does someone take the time to go on a site like Glassdoor and write a review that makes it seem like the company is heaven on earth? Sure, people will vent if they are upset, but sadly, few people wake up in the morning and think, “Ohmigosh, I’m so happy with my job, I think I’ll go write a fabulous review of it”. That’s a big red flag right there. Is it a real review, or did someone in HR write that?

Conversely, a review of 1 with nothing nice to say and paragraphs about what a horrible place/product can be either a competitor (who will refuse to say anything nice) or a disgruntled employee. Disgruntled employees, especially if there are a lot of them, are worth noting but one or two people in any crowd are always going to be ticked off about something. As with super-good reviews, take super-bad ones with a grain of salt. Someone who has only good things to day or only bad things to say looks suspicious. Nothing is all good or all bad (except me, of course – I’m fabulous).

I generally drop the top 10% and bottom 10% of reviews just to remove outliers in either direction. And while we are talking about numbers, the spread is the most important thing. Assuming, of course, that you have enough reviews to make up a meaningful sample. A product/service/company with only a few reviews is too easy to manipulate and you should use any of them warily. If there are enough, then look at the distribution. Are most of the reviews 4s or 5s? That’s good. 1s and 2s? Stay away. Even if 20% of the reviews are 4 or 5, if the other 80% are 1 and 2 that can indicate very real problems.

2: How does it read? The next flag is to look at the language used. This applies to products, companies, and pretty much everything. Does the review talk about a “superior feature set”, or “solving their customer’s mission-critical enterprise challenges”? This is also either coming from HR or Marketing, or else someone who seriously drank the Kool-Aid. If the words sound like they came right out of a company or product brochure, they probably did.

3: Is it relevant? Another type of review to toss is the confused reviewer. “The product worked great but UPS driver left it by the garage instead of the front door – one star”. Or the always popular “The iPhone 5 case didn’t fit my iPhone 7 – one star”. Don’t laugh – I’ve seen sillier ones. If the substance of the comment is about something that is actually unrelated to the proper use of the product/service.

4: Are the dates suspicious? One last thing to look for is the date that the reviews were submitted. If there is a cluster of similar reviews around one week or month, it could just be a push from the company to get their users to write reviews. Or it could be a push from the HR/Marketing department to submit a series of fake reviews. If there is a cluster of negative reviews it could be a drive from a competitor to downgrade the reviews, or it could be the result of a big layoff or other negative company event. But again, clusters of reviews are worth looking at in more detail. Normal reviews come in a steady stream, not all at once.

 

 

 

Posted in Business, Data and Analytics, Life, Marketing | Comments Off on Reviews are awesome – or are they?

Bombarded by advertisements

Are we really exposed to 5,000 advertisements a day? And does it even matter?

We are bombarded by advertising each day. This is obviously true, and it is not even a new problem. As far back as 1759, Samuel Johnson said, “Advertisements are now so numerous that they are very negligently perused, and it is therefore become necessary to gain attention by magnificence of promises, and by eloquence sometimes sublime and sometimes pathetick.” And that was before Facebook!

If you try to look up data on exactly how many advertisements people receive each day, you will find wildly different statistics. I’ve seen numbers that range from 500 to 10,000, with 5,000 being the most often quoted. While this is great for sensationalist stories, is it accurate?

How many advertisements do we see each day? The first premise is that we are looking at city dwellers, who spend a normal amount of time out and about each day. Where do the ads come from?   Billboards, obviously, either on the train and bus or by the road side on your way to work. Ads on TV – watching the news as you get up, or as you relax at the end of the day. Online ads as you check Facebook in the morning and then as you Google things during the day. Radio ads, perhaps, if you listen to the radio during work or on your commute. And, of course, email. Physical mail? Probably some both at home and at work.

Does our definition of “advertisements” include any brand exposure? Store names as you walk to a restaurant during lunch? Product names in the store front windows? How about the logo on the clothing of the people you pass? The logo on the mug of your co-worker? On your laptop? Logo exposure ups the game quite a bit.

But 5,000? That still seems like a lot. Let’s look at it another way. Assuming 8 hours of sleep in a day, that leaves 16 hours of waking time. That comes to around 5 ads a minute. Obviously the only way you can hit that number is to get large blasts of advertising during the day, and that means huge logo exposure as you walk down the street. To be honest, we should separate brand exposure resulting from a glimpse of a logo, from actual advertising that includes messaging and possibly some CTA. And just based on numbers, that has to take us well below 5,000. In fact, just intuitively 500 now seems a bit high when we start to think of it.

And also beside the point.

The real point is not whether we are flooded with advertising (we are), or how big that number might be, the real point is what our tolerance is for an invasion of our personal space. And that is quite low. Witness how we readily spend money to avoid them. I “Tivo” through advertisements on TV and while the ability to manage and time-shift my TV viewing is the primary function of my Tivo, I would cheerfully pay the same price just to avoid ads. I throw away, unopened, any physical mail that smells of an advertisement (which is how I almost threw away my passport recently because it came in an unmarked envelope). Network providers, companies, and individuals all spend money on spam filters. People install ad blockers for their browsing.

These ad-avoiding tactics all reflect a basic desire to be left alone and not bothered by advertising, and one that we are willing to spend money on. If you were really subjected to 5,000 ads a day you’d probably start walking around with a bag over your head. As it is, we spend good money to block a mere handful of intrusions. So in the end, while 5,000 or even 500 sounds like a scary large number, it really isn’t accurate. But even though the real number is much lower, what counts is how much the real number is above our level of tolerance, And that level of tolerance is quite low. So perhaps you only receive 100 ads a day, or 50, or even 10. That may be enough to send you over the edge and start implementing ways to block those ads. At that point, advertising itself begins to fail as a tactic. Oops.

Posted in Business, Data and Analytics, Marketing | Comments Off on Bombarded by advertisements

Apophenia vs. Analytics

795e3777-6f1c-43b8-b2d4-d6fd425cb839

Data is important. It forms the basis for all of our decisions. When we use it correctly we make good decisions. When we use it incorrectly we make bad decisions. And why do we use it incorrectly? Because of a mixture of bad assumptions, partial data, and apophenia.

Apophenia, a term coined by the German neurologist and psychiatrist Klaus Conrad, is the perception of meaningfulness in unrelated phenomena. We find this in many places, including the perception of patterns in data. Not real patterns, of course, but ones that you think you see. This is easier to do than you might think. Especially when you start off looking to prove a pre-conceived conclusion. Then you can pick and choose the data that matches your pre-held conclusions. This is fun when you look at a cloud and see the shape of a camel. No so much when you look at your business intelligence and see market trends that aren’t really there.

Seeing patterns in your data that don’t really exist can result in false conclusions about correlations. This is known as a “spurious relationship” where two or more variables are not actually causally related to each other, despite having the appearance of being related. In the statistics world, this usually results in a Type 1 error, or false-positive result.

Take a look at http://www.tylervigen.com/ for a good laugh. The site has a number of graphs showing variables that appear to be related based on historical data, but obviously could not possibly be related. My favorite is the clear correlation between the between the number of people who drowned in pools with the number of films Nicholas Cage appeared in.

Should we then ignore correlations? Definitely not. Finding correlations is a very important step in highlighting relationships, but it is only the first step and should be used with caution. A correlation is merely a spotlight on something that deserves attention. It is a valuable pointer to where we should look. Often, we will be able to find an actual cause and effect relationship where no one had thought to look before. But until such a relationship is proven, correlation is merely a guide – an indicator of where to look – and not a hard and fast predictor of future events. Which means that, at least for now, it is probably safe to let Nicholas Cage make more movies.  At least for people who are near swimming pools.

Posted in Business, Data and Analytics, Marketing | Comments Off on Apophenia vs. Analytics

Brand equity for better or worse

mynameisEveryone knows it takes time and effort to build brand equity. Or do they? I’ve watched over the years as both good and bad brand decisions were made, and usually with predictable results.

Brand equity covers a number of things. It is the value that you get from how your customers perceive your brand as identified by a name. It is different than their opinion of a specific product, although we can look at brand equity for a name that applies to a single product. We can also look at brand equity for a name that describes a product line or even a company. As a company, Apple and Google have enormous brand equity. As a product, a Mustang (the car, not the horse) and a 747 both have enormous brand equity. I’ll be there are people who know of the Mustang but don’t know it is made by Ford, and I’ll definitely bet there are people who know what a 747 is but don’t know it’s made by Boeing.

Naming a brand, whether it’s a company or a product, can be tricky. Lots of companies spend lots of money in consulting fees and agonize over “just the right name”. This certainly has an impact on the B2C world, where many purchases are based on emotional attachment, but B2B is not immune from this either. B2B buyers may not be quite as attached to a cool name, but they will certainly remember or forget a product based in part on its name, and an unfortunate name can still hurt you.

Naming is a complex subject, but here are my top 2 tips for naming. I would think they are obvious but I have seen them violated time and time again, so perhaps not.

As the saying goes, put all your wood behind one arrow-head. Pick a name and use it everywhere. I’ve seen software products that had one name that it was marketed under, and yet you accessed the service using a portal or app that had a different name. Guess what customers thought the name was? If you call your music playing service “Fred’s Music System” but the app is called “Internet Tunes” (probably a bad name in itself), people are going to think the product is “Internet Tunes” because that’s what they use, not “Fred’s Music System” which it is marketed under.

And by the way, when it comes to trademark infringement, the courts are going to look at what your product is commonly referred to as well as what you actually call it on your web site. If you call your music playing service “Fred’s Music System” with the app called “Internet Tunes” and all your customers abbreviate it to “ITunes”, you have a problem. The commonly used name is clearly a trademark violation, and it doesn’t matter whether you did that intentionally or it just happened.

Pick a name, use it everywhere. Be consistent. And be consistent not just in three dimensions, but in four as well. Everyone loves to change names. There is nothing more exciting for a fresh VP of Marketing or new CEO than rebranding the world in your image. Replacing old names from the previous regime with really cool, impactful new names as part of your house cleaning. I appreciate the desire to rename something and make it yours (and prove that you have better names than your predecessor), but this is a risky business. Sure, the existing name might not be awesome, but do people know it? It takes a long time and a lot of effort to establish brand equity. Do you *really* want to give all that up just to come up with a cooler name? Think carefully on that one.

It takes a really, really long time for people to associate a new name with your existing product. Unless you are planning on spending a boatload of money on advertising the new name, expect to simply confuse your audience with a name change. Of course, there is the opposite problem where you do want to hang on to the brand equity in a name but you have changed the underlying product or service. Sure, people will know the name, and it might well get them to answer your call or open your email. But be prepared for a lot of confused conversations after that when they find out what you really do today. Getting those doors opened might well be worth it, but again you should think carefully about that one and make sure you (and your sales team) is prepared to open all of your customer conversations with “we are not that company anymore”.  Just as people will continue to use an old product name for a long time, so too will they continue to associate an old name with an old product or service.

Pick a name, any name, and don’t stress about it. It really doesn’t have to be a great name, just try to avoid using a bad name. And then use it. Everywhere. And forever. And slowly, but surely, you will build up that brand equity that makes your life so much easier. And sure, break the rules when it makes sense. But make sure you are doing it for the right reasons, and not just because someone came up with a really cool new name that is just too hip not to use. Save that for your next new product launch, or even your next startup.

Posted in Business, Marketing | Comments Off on Brand equity for better or worse

Conducting surveys: part 2

data entrySurveys are great tools to find out information about some population that can help guide your business decisions. But bad surveys generate bad data and result in bad decisions. Let’s continue with our look into the first step of any survey: picking out your targets.

In the previous post we looked at the total population we are interested in, the sampling frame that represents the list of the entire population, and the sample that we will actually use for the survey. We rarely try to survey an entire population (saturation sampling) unless it’s a small one, and will usually use a small subset, or sample. Selecting the sample from the full sample frame can be done by selecting members randomly (random sampling), or by picking every Nth member (systematic sampling). We looked at the danger of ending up with a sample that does not accurately represent the make-up of the total population and investigated the use of stratified sampling as a first step to allow us to end up with the right % makeup for key parameters like gender or age or income.

Moving on from there, we should also consider that there might be parameters that make someone either a good selection or a bad selection. These will make up a set of eligibility criteria. Rules that must be met for an individual to be included are called inclusion criteria. An example of this might be that someone has to be alive (at least, in most states) to be considered an active voter. Rules that must not be met to be included are known as exclusion criteria. An example of this might be that unregistered voters are excluded from the list of active voters. And yes, whether something is an exclusion or inclusion criteria depends largely on how you word it.

So far we have been looking at closed populations – ones for which we can obtain a complete sampling frame. There are also many instances of open populations – ones where we do not have a complete sampling frame available to us. For example, we know with certainty every member of a school’s student population. We have no idea who the members are of the set of homeowners with blue carpets in New York. We can certainly create a list of some of them using a variety of sources, but it will not be an exhaustive list. Selecting a meaningful and representative sample from this group is challenging because we don’t know the characteristics of the whole population. We have no idea how many men vs. women own blue carpet in total, just in our limited sample. So to create a useful sample that represents the general gender makeup of the total population is impossible. Nevertheless, we can still learn interesting and useful things from these samples.

There are two basic ways to deal with open populations. The first is to recruit members to participate on a panel of respondents, and the second is to restrict the population to a specific sub-set, like visitors to a certain web site, for example.

Pre-recruited panels involve work to create and then maintenance to hold on to. If you are large enough and have enough resources you might be able to grow such a community. Or you might engage a company that does maintain a stable of panelists and use them. If you want to recruit a blind panel that doesn’t know who you are, as opposed to a branded one that does know who you are, you almost have to go with an outside supplier. You do need to be careful that the answers you get from a panel are “real world” answers and are not skewed by their familiarity with surveys.

It should be noted that participating on a branded panel also has the benefit of helping to improve brand loyalty by making the panelists feel like they have a connection with the brand.

Gathering your survey respondents from a specific site (online or physical) is known as “intercept sampling” and can be as simple as popping up a window to all your website visitors (or every Nth visitor) and asking them to participate in a short survey. Here, too, you need to be careful about using results to make assumptions about the general population. People that visit your web site already are a specific type of person (by displaying interest in your products or company) and therefor not a completely random sampling.

In fact, it is important to always keep in mind that there will always be conditions and filters placed on your sample, no matter what you do. If you are surveying employees, you might be able to force them all to answer the survey. But in most other cases, you cannot force people to even see your survey, let alone answer it. When you send out a survey, some people will never even get it whether that’s because their email filters catch it as spam, or they simply toss an unrecognized envelope in the trash (physical or electronic). So right there you have further limited your sample to “people that do not have strong spam filters”. There will be many people that simply don’t like to answer surveys, so of the people that open your email, you are selecting “people that are willing to answer surveys”. Even if they are willing, they might not have the time to answer a survey and so you end up with “people that are not too busy to answer a survey”. Finally, you probably offered an incentive and that will appeal to some people more than others. If your incentive is a drawing to win a vacation to Alaska you have now also included people with a propensity to gamble (they feel positive about a random drawing) and people that like Alaska.

Do we care about all of these implicit selection filters? Some, probably not. But if we are conducting a survey by email about overworked employees, keep in mind that they very target we are seeking is probably too busy to answer the survey. If we are looking into attitudes about beach goers to types of sun block then offering a chance to win a trip to Alaska is probably going to appeal more to the wrong crowd. In these two examples we won’t eliminate the target we are going for but we will reduce it. Offering a membership in the bacon-of-the-month club (yes, that’s a real thing) as a prize to members of the Jewish community is an extreme example, but the world is full of people who have done sillier things. Just ask yourself at each stage, “How will this impact who responds to my survey?” Some answer will matter, some won’t, but it’s important to know.

That is a quick overview of some of the issues with sampling and sampling methods. There is much more to say about that, and many other survey topics to investigate that can help you to make sure your end results are both accurate and meaningful. Otherwise, garbage in… garbage out.

Posted in Business, Data and Analytics, Marketing | Comments Off on Conducting surveys: part 2

Conducting surveys: part 1

Group Of Happy Doctors4 out of 5 doctors recommend…what? How do we know what 4 out of 5 doctor’s recommend? Probably because someone conducted a survey. Sounds simple, doesn’t it? But obtaining accurate results can actually be difficult.

Conducting surveys can provide valuable information to help you guide your business. Whether you are performing a brand survey, or a best-in-class research project, finding out what people think and what they do is an important activity. Sometimes you find out new information that you didn’t know, which is obviously valuable, but validating information that you already believed or suspected is just as important.

Where to start? The first thing to consider is who to survey. This may sound simple: you have a mailing list and you can simply send out your survey to your existing mailing list. While simple and convenient, this presents a number of problems as well.

To start with, we need to look at what our population consists of. The population is the entire community that we are interested in. This might be all consumers if you sell iPhones, or all B2B marketers if you sell business marketing solutions. For this population you need to come up with a sampling frame which includes all of the members of this population. Some sampling frames are easy to come by – a list of all students at a particular school, a list of all members of an organization. Some are more difficult to come by like coming up with a list of all business buyers in a whole country.

Whether or not you are able to obtain a list for your entire sampling frame, you often are not able to survey an entire population (saturation sampling)  simply because of the number of members. In most cases, you will select a sample from the entire sampling frame and use this smaller sample for your survey. While this makes the survey much easier to conduct, it also presents a number of challenges. Primarily, we need to know what the relationship is between the make-up of the sample to the make-up of the whole population. If 75% of the population thinks A, but in our sample we pick mostly the people that think B, our survey results will not be representative of the whole. There are a number of ways to address this.

The first way to select a subset is to simply pick members at random. This is the easiest way and can often result in a good representative sample. Another way to create a sample is to use systematic sampling and select every nth member of a list. Both of these methods, however, also have drawbacks.

If the sampling frame that we have is not actually the set of the whole population, and if it is skewed in some way, than a random or systematic sample pulled from the frame will probably be skewed as well. Of course, there is always the chance that a random selection itself can be skewed even if the sampling frame is complete. For example: if we have a population of all potential voters, we can easily use a voter registration list. But if we find out that anarchists are half as likely to register as people with all other types of political affiliation, we can be confident that the voter registration list is not an accurate representation of our entire target population. Or if we have a population that is 50% male and 50% female, but a random sample happens to include 60% female and 40% male, then our sample is not going to be representative.

One way to address this is to create a stratified sampling, often as a first step in a two pass sampling selection process. In a stratified sample, we split the whole list into groups and then proceed to apply a sampling technique like random or systematic to each group. After that, we combine the results. To look at our previous example where half of the population was male and half female, we can split the whole sampling frame into male and female groups, and then select x number of random samples from each, and combine the results. This will give us a sample that is guaranteed to have the same 50/50 ratio of male to female members as the larger sampling frame. Or in the case of voters, we can select twice as many anarchists as other political groups (assuming we do know for certain that they register exactly half as often). Stratified sampling can be used with any type of groups that are important to your research. If you are conducting a survey on favorite flavors of ice cream, it might not matter whether your ratio of male to female members is accurate but it might well matter that your age grouping is correct. If you are surveying voter affiliation with Hillary Clinton, it probably is important to get the male/female ratio correct in your sample in order to get accurate results.

To be continued in part 2…

Posted in Business, Data and Analytics, Marketing | Comments Off on Conducting surveys: part 1

Customer incentives that actually incent

enter to win! keyboard key fingerIf you are going to give stuff to prospects or customers, at least make sure they are actually useful.

Companies spend lots of money on prizes, incentives, giveaways and other “stuff”. But in far too many cases, the “stuff” is generally thought of as “junk” and the prizes and incentives fall flat. Why is this?

First of all, every customer base is different. Urban 20-somethings are going to have vastly different interests than suburban 40-somethings. You need to have some basic understanding of who your target audience is and what they are interested in. I know that sounds simple, but all too often I see companies that have failed to really think about what their audience wants.

Even if you do know who your audience is, many times companies fall back on traditional give-aways or get seduced by bright shiny (cheap) objects and ignore their target audience’s interests. Stress balls can be cute, especially in some fun form like a brain. If they are novel enough they might wind up on a shelf but how many people actually use them? How about the ever popular logo pens – do they make it past the end of the day? Does anyone even write anymore? Coffee cups are useful, but heavy to travel with and how many people need a new coffee mug? Of course, if you can make any of these interesting or novel in design then yes, they can still be relevant. But be careful about things that might make it home but will go straight to the kids to play with. They are probably not going to be customers for a few more years.

How about something new? Maybe a desk clock – doesn’t have to be too expensive? Or a clock with temperature and humidity so people can see if the office really is as hot as it seems? This might get used and is perhaps something not everyone has. USB sticks can be used by everyone but mostly they get thrown in a drawer and never seen again (next to the pen if it was lucky enough to actually make it home).

The other thing to consider that is often a big fail is who benefits. If you are giving out prizes and the winner gets a discount on your product, or a free trial, or a free sample… does your target really care? If they own the company then yes, they care. If they have stock in the company then they might care. But if they are just another salary worker, then it’s the rare employee that gets all excited about saving the shareholders or owners a few dollars.

I was on a business trip recently and stayed in a big brand hotel that was being renovated. I mention this because I assume that was the reason I found a 2 inch cockroach skittering across the floor the next morning. I immediately grabbed my bags (fortunately I hadn’t had a chance to unpack) and asked to be moved to a room on the far other side of the hotel. When I checked out they took one night’s charge off my bill which was nice of them but, as I pointed out, I was the one who had to deal with the giant cockroach, not the shareholders of the company (of which I am not one). To their credit, they then added on some free points to my account which was a good catch but they could have asked me in the first place what would make me happy and make up for my personal unpleasant experience.

So before you offer a grand prize discount, or order another 2,000 pens, think long and hard about who your audience is and what they actually want.

 

Posted in Marketing | Comments Off on Customer incentives that actually incent