As a direct marketer, have you ever asked any of these questions:

How big should my cell sizes be to test the effectiveness of these new communications pieces?
We already did our test…what conclusions can we reach with the response results?
I want to hold out a control cell and compare by stimulated group to these controls afterwards…how big should my control group be for meaningful results?

Let’s take a quick, business-as-usual, example and see if this sounds familiar:

Results come in from a direct mail campaign in which two creative versions were tested by evenly dividing a lead pool of 250K leads into two groups:

Creative Version A:  2.10%

Creative Version B:  2.20%

In this test, Version A is the incumbent ‘hero’ package, and Version B is a new contender.  The marketer does a statistical test of significance and gets a p-value of 0.04 (remember your business stats class here…).  Great!  Version B generates a higher response rate, so that becomes the new ‘hero’ piece!

What’s wrong here?  A couple things:

1)      Just looking at the response rates, if you saw 2.10% vs. 2.20%, would you consider those substantially, or meaningfully, different enough to change your hero piece?  Perhaps, but probably not.

2)      If this test were based on evenly dividing only 200K leads instead of the 250K, the statistical test would have led the marketer to determine that these two response rates are “not significantly different.”  The larger the sample size of the groups, the more likely it is that the test for differences will be significant. 

In the above example, the test is significant, but we cannot say with a lot of confidence that Version B will consistently generate a response rate that is .10 percentage points higher than Version A.  The reason is that there is another side of the statistical comparison that most people are not aware of – statistical power.  In simple terms, it is the probability that your conclusion is correct.  In this case, although the p-value is small enough to say the two response rates are different, we would only have statistical power of about 53% in saying that “Version B’s response rate is .10 percentage points higher than that of Version A.”

In short, determining optimal sample sizes depends on four things:

1)      Expected response rates

2)      Difference in response rates that would be meaningful, or expected

3)      Desired statistical power in detecting the meaningful difference in (2) above

4)      Significance level of the comparison (commonly referred to as ‘confidence level’ by marketers, as in a “95% confidence…” but this is different from statistical power

The good news is that items (3) and (4) are usually ALWAYS the same:

3)      80% – statistical power

4)      .05 – statistical significance level of the comparison

So as a marketer you need only be concerned with having a fair idea of the expected response rates, and what kind of difference you are seeking to identify with confidence.  A good example of identifying a ‘meaningful difference’ is the case where Version B costs more than Version A, and for it to be beneficial to use Version B the response rate has to be at least X% higher to cover the increased expense.  We would then plan our test cell sizes to be sufficiently large enough to be able to detect, with confidence, the difference of X%. 

Calculating Sample Sizes

Here’s a simple calculator you can use to determine the minimum required sample size for a test that will compare two mail cells (double click in boxes to enter values):

The larger the difference you’re trying to detect, the smaller the required sample size.

Happy testing!

Source:  Statistical Rules of Thumb by Gerald van Belle;  http://vanbelle.org/