Analytics tools are like fitness equipment. They are a means to an end. Simply implementing a tool won’t get you anywhere. That’s like buying a treadmill and never getting on it.
The purpose of the tool is to gain valuable insights to improve your online presence. I hear clients and analysts constantly talking about why they can’t justify spending money on a Digital Analytics tool. That point is almost always followed up with, “but we need to do something.” (more…)
As a direct marketer, have you ever asked any of these questions:
How big should my cell sizes be to test the effectiveness of these new communications pieces?
We already did our test…what conclusions can we reach with the response results?
I want to hold out a control cell and compare by stimulated group to these controls afterwards…how big should my control group be for meaningful results?
Let’s take a quick, business-as-usual, example and see if this sounds familiar:
Results come in from a direct mail campaign in which two creative versions were tested by evenly dividing a lead pool of 250K leads into two groups:
Creative Version A: 2.10%
Creative Version B: 2.20%
In this test, Version A is the incumbent ‘hero’ package, and Version B is a new contender. The marketer does a statistical test of significance and gets a p-value of 0.04 (remember your business stats class here…). Great! Version B generates a higher response rate, so that becomes the new ‘hero’ piece!
What’s wrong here? A couple things:
1) Just looking at the response rates, if you saw 2.10% vs. 2.20%, would you consider those substantially, or meaningfully, different enough to change your hero piece? Perhaps, but probably not.
2) If this test were based on evenly dividing only 200K leads instead of the 250K, the statistical test would have led the marketer to determine that these two response rates are “not significantly different.” The larger the sample size of the groups, the more likely it is that the test for differences will be significant.
In the above example, the test is significant, but we cannot say with a lot of confidence that Version B will consistently generate a response rate that is .10 percentage points higher than Version A. The reason is that there is another side of the statistical comparison that most people are not aware of – statistical power. In simple terms, it is the probability that your conclusion is correct. In this case, although the p-value is small enough to say the two response rates are different, we would only have statistical power of about 53% in saying that “Version B’s response rate is .10 percentage points higher than that of Version A.”
In short, determining optimal sample sizes depends on four things:
1) Expected response rates
2) Difference in response rates that would be meaningful, or expected
3) Desired statistical power in detecting the meaningful difference in (2) above
4) Significance level of the comparison (commonly referred to as ‘confidence level’ by marketers, as in a “95% confidence…” but this is different from statistical power
The good news is that items (3) and (4) are usually ALWAYS the same:
3) 80% – statistical power
4) .05 – statistical significance level of the comparison
So as a marketer you need only be concerned with having a fair idea of the expected response rates, and what kind of difference you are seeking to identify with confidence. A good example of identifying a ‘meaningful difference’ is the case where Version B costs more than Version A, and for it to be beneficial to use Version B the response rate has to be at least X% higher to cover the increased expense. We would then plan our test cell sizes to be sufficiently large enough to be able to detect, with confidence, the difference of X%.
Calculating Sample Sizes
Here’s a simple calculator you can use to determine the minimum required sample size for a test that will compare two mail cells (double click in boxes to enter values):
The larger the difference you’re trying to detect, the smaller the required sample size.
Happy testing!
Source: Statistical Rules of Thumb by Gerald van Belle; http://vanbelle.org/
“I know that Direct Mails get higher response rates than FSI’s in the newspapers but can I justify the higher costs of DM?” or “if we launch our new signature product, how do I know it will work and outperform our current product, especially at a steep 20% discount?”
The former question is one that we hear often from our clients. The latter question was probably uttered by Jeff Moody, the CEO of Subway, before he decided to launch their foot-long sandwiches at $5 to a resounding success of $3.8 billion in sales and catapulted Subway past Wendy’s and Burger King in terms of market shares. In fact, the success of the “$5 FOOTLONGS” campaign has positioned Subway to overtake the market leader, McDonald’s restaurants, in terms of the number of stores worldwide in early 2010!
Picking the next “$5 FOOTLONGS”
How does one answer the above questions, but more importantly, how can you be sure that you have found something akin to the “$5 FOOTLONGS” and able to convince your boss to launch it before your competitors get to it. The answer to this question lies in using a data and analytics driven “Test and Learn” methodology, commonly known as Design of Experiments (DOE).
In essence, DOE has been used for a long time in scientific and medical research to establish the incremental effect caused by a particular “treatment” while eliminating the effects from any other factors. So in the case of Subway, how would Jeff Moody determine the increased traffic observed by the trial store was caused solely by the $5 price and wasn’t due to any other mitigating factors? For example, local conditions specific to Miami or the way the store owner, Stuart Frankel, managed his stores and/or the promotion?
To answer this question, one should first address the question of whether the increase in traffic was caused by the promotion at all i.e., the question of “causality” and then isolate the effect of the price promotion from the other factors that may also impact Mr. Frankel’s Subway sales. (more…)
Aspen Marketing was chosen ExactTarget’s Embedded Partner of the Year in the 2009 Subscribers Rule Awards, which recognize the industry’s top email marketing programs. Aspen, an ExactTarget Certified Gold Partner, was chosen from among a pool of nominations by more than 7,000 clients worldwide.
ExactTarget’s Subscribers Rule philosophy stresses the importance of respecting subscribers’ preferences in regards to marketing messages. It contains three simple beliefs:
Awards were presented during ExactTarget’s annual user conference, Connections ’09, in Indianapolis in front of more than 1,300 attendees.
Is it just me, or have we all lost our ability to clearly describe what’s happening around us? The old adage of Keep It Simple Stupid (KISS) appears to going the way of the do-do bird.
As agencies, we are struggling lately with one of our core “reasons for being” – making things easier for consumers to understand. Could it be that one of the key reasons organizations are abandoning agencies is that we have simply stopped making sense?
In a marketing world of fractured audiences, a virtual shutdown in spending and unprecedented levels of consumer input direct to brands, agencies should be the one cutting to the benefit quickly and succinctly. Have we forgotten how to avoid marketing speak in what we say and write? Case in point, isn’t it easier for consumers to know what a “sale” is versus a “savings extravaganza” or a “sales event”?
And since when was it OK for a non-personalized direct mail piece to become an “exclusive” offer, or a promotion “reserved” for the “Current Resident?” Let’s give people more credit folks… (more…)
We had an amazing turn out for my Social Media 101 seminar at last week’s show at AmericasMart Atlanta. As promised, here is the slide deck and some useful links from the presentation.
For those of you that weren’t able to attend, here are a few key points and some notes to keep in mind when deciding if social media is the right thing for your business: (more…)
West Chicago, IL, August 25, 2009 – Aspen Marketing Services, one of the nation’s largest privately-held marketing services agencies, today announced the hiring of Nathaniel Lin as President of Advanced Analytics.
In his new role, Lin will be responsible for strategy and execution of all analytics initiatives deployed to drive leading-edge CRM, direct marketing, digital and web solutions. He will also oversee the company’s Advanced Analytics team.
Mark on AirTran was a daring campaign on all fronts, even as it was complicated by Homeland Security/TSA logistics and FBI requirements. Nevertheless, the campaign redefined experiential marketing by taking full advantage of social media, events, digital and public relations, growing its reach and depth exponentially as Aspen Marketing guided communication streams to build upon each other.
For more information on the campaign, please visit: www.MarkonAirTran.com
Wednesday July 28, DesignChat will be live streaming video of the design and social media discussion with Pete Cashmore. The chat begins at 9pm EST at mashable.com/chat and will last approximately and hour. We will be talking about Mashable.com’s success and how the worlds of social media and design combine to create some of the most interesting interactions of the 21st century. The chat starts at 8pmCST and will last for roughly an hour. Follow DesignChat’s founder and Aspen Art Director, Ryan McGovern (@hupajoob) and DesignChat (@designchat) on Twitter for updates about the show.
If you miss the live broadcast, the video will be posted on www.designchat.info within a week of the discussion.
You have to love U.S. Bank’s aggressive play to retain customers in danger of trading their NWA WorldPerks Visa cards for the Delta SkyMiles AMEX. It reminds us that while credit card marketing may be down – it is certainly not out.
U.S. Bank’s “FlexPerks Travel Reward Visa” is one of the clearest indications yet that there are some things card issuers will not sit idly by and let happen, rising charge-offs and unemployment be damned. In this case, U.S. Bank is not ceding free reign to AMEX to cherry pick from what was arguably U.S. Bank’s most prized card portfolio.
With U.S. Bank’s rollout of this product, Delta (with ample help from AMEX we’re certain) has been pushing their own product heavy via email and paid search (when you search “Flexperks”, look what the top two sponsored links are) and amping up their offers a bit. With this battle for customers clearly “on”, Delta and AMEX had better prepare a rich product with a strong acquisition offer to secure the WorldPerks Visa customers they want the most. (more…)