In today’s world of decreased marketing budgets, optimal testing of Direct Mail, Email Campaigns or Web marketing is essential.  Industry in general has historically adopted a split test approach, where a list is divided into a test group and a control group. Other names for this approach are “A vs. B” or “Champion Challenger”.  Such designs are often sub-optimal, inefficient and don’t maximize the learnings that can be extracted from the test.

There’s a new game in town: Experimental design.

Experimental design originated in the early 1900’s when farmers needed to increase crop output while minimizing risk of lost crops. By simultaneously testing many factors in as few test runs as possible, farmers were able to minimize lost crops due to unsuccessful tests, and quantify the increase in crop yield specifically attributable to the factors being tested. Armed with that knowledge, they could determine if a specific increase in tested factors, or a simultaneous combination of factors had a positive ROI.

Now… let’s leave the farm and think about marketing again. The application is almost identical, but instead of fertilizer and irrigation, we’re testing copy, offer, creative, channel, images, colors and anything else you can imagine. The goal of almost any piece of direct communication from a business to its customers or prospects is relevancy: Getting the right offer to the right person at the right time.

The ultimate direct mail piece is a harmony of all the factors I just mentioned — and the harmony is different for different population segments. It is very rare that only one factor in a direct marketing contact affects response. How do you figure all of that out quickly, efficiently and cost effectively?

Let’s say you wanted to test 3 offers, 2 formats, 2 channels, 4 segments, 2 color schemes. In a split run test, that would take:

3 x 2 x 2 x 4 x 2 = 96 versions!

Utilizing Experimental Design you can do the exact same test in as few as 9 versions, all the while learning more and enjoying substantially lower risk exposure due to much smaller, as in 90% smaller, sample sizes.

There are challenges to implementing an experimental design as well. Tests are intricately constructed, and any deviation from the designed tests, either accidentally or purposefully, causes problems. At the outset of such a test, it is important that all of the necessary instruments to execute it are in place. Results can be confusing or impossible to extract if the test is not executed properly.

A great deal of the time and effort we spend in helping clients with this approach is spent assuring that we’ve done everything to design the test so that it fits into their systems, production and execution environments. However, we have found that once an experimental design is executed successfully, the results are amazing, and our clients refuse to do it any other way.

Any marketer involved in testing and optimizing communications with their customers or prospects should consider experimental design.