The basic methodology for evaluating consumer marketing event performance is so widely known that it hardly needs to be mentioned: classic experimental design –whether ‘test vs. control’ or ‘pre-test/post-test’ observation– is the universally preferred method. But often it is not practical or even possible to execute an event in the context of an experimental design: budgets can be exhausted; last-minute opportunities may present themselves for can’t-miss on-strategy events, and thereby leave no time to set up for proper measurement… Anything and everything can happen, and while the purists might cringe, triage and hard decisions are hallmarks of real world program measurement and evaluation. Control groups can fall by the wayside, but the need for rigorous measurement remains. I’ll share some of our thinking and proven methods for making the most of situations where classic experimental design for measurement of a marketing event is not an option.
Stare Decisis
Students of law are familiar with the concept of established reasoning serving as an immutable standard for the present. Stare decisis – or literally: let the decision stand- is a helpful tool in the measurement of events when identifying a control group of consumers is impractical or impossible. In law, stare decisis is evident when a court’s judgment is informed by the precedent of earlier decisions; simply, all that needs to be said in regard to a matter has already been said, and so further deliberation is not necessary. But we’re concerned with event marketing, not law, and so we borrow this concept in the figurative sense. We apply the concept by identifying and using an acceptable precedent to serve as the baseline metric against which to compare post-event observations. Bear in mind that we always strive mightily to work with a proper control group, but in cases where a control is impossible or impractical we will borrow from an established source and, in effect, declare “the ‘decision’ of earlier work will stand; the baseline metric is XYZ.” One may borrow (note: disclaimers appear below!) existing knowledge and repurpose it to approximate what we would expect to find in a formal study of a control group. Existing knowledge stands as the standard, and so the need to collect and interpret new evidence (back to the legal analogy!) for a control group is negated!
Typical sources for a borrowed baseline include brand tracking/health metrics, industry group data, syndicated tracking studies, and proprietary studies conducted by the host venue (e.g. mall, stadium, exhibition hall, etc.).
The practical application requires post-event performance data collected via standard methods (e.g. consumer surveys, sales figures, web traffic, etc.). These metrics are juxtaposed to the standing/borrowed baseline as though the substitute were a classic control. Find in the post-event period that positive mentions of your brand come in at 35% among ‘treated’ consumers? Then the national average of 20% from your quarterly tracking study indicates that a lift has been achieved. 11% of visitors at the theater performance you’ve sponsored claim that they will consider your brand when they are next shopping? Better carefully examine the size of the samples to determine if there is grounds to claim a lift.
In repurposing data to create a baseline it is important establish whether it’s comparison to treated consumers stretches the bounds of validity beyond a point of reasonableness. There are no hard and fast rules to identify when, say, industry group data or older in-house work is too far off of the mark to be useful, but there are some rules of thumb. The most important variable in our deliberation is the comparability of the consumers comprising the baseline and those who will be ‘treated’ via the marketing event. A demographic match is a good start, but can it be demonstrated that the test and ‘control’ groups are similar in their interest in your brand’s category? Identical in the horizon of time until their next purchase? An industry study of hunters is probably an acceptable fit for consumers reached via a product and information display at a hunting & fishing show. But is it reasonable (probably not) to compare the sports equipment-buying habits of these same show attendees with general population measures lifted from US Department of Labor data?
From the Horse’s Mouth
Self-reported lifts via consumer surveys can be an option, but it is one that must be approached with great care. Briefly, the method involves interviewing consumers who have experienced a particular marketing event. Typically, a consumer is asked to comment about their experience and then report whether or not the experience has impacted their attitudes toward a brand, their likelihood to shop a brand, etc. The chief dangers inherent in peering into this proverbial horse’s mouth for answers are (1) the likelihood that a consumer can accurately detect and report changes in attitudes and/or likely behaviors, and (2) inadvertently leading the consumer to appease the surveyor with an inflated account of their positive takeaways.
As to the first point, the solution is to recognize that consumers (most, anyway) are not behavioral scientists and, accordingly, do not ask them to self-diagnose as though they are. It is difficult for consumers to judge and quantify changes in attitude on finely calibrated ten-point scales, but it is fair to ask whether their attitudes have changed for the better or worse as a result of their experience. Reduce the likelihood of measurement error by simply asking if change has occurred and in which direction. Avoid scaled responses in matters of self-reported change. Make the query simple, and ask no more of your data than to indicate the up or down direction of change. Cross-event comparisons are legitimate, though, if the basis of comparison is simply the proportion of consumers indicating a certain degree of change.
Finally, psychology literature tells us that it is distressingly easy to lead an individual to become a unwitting conspirator who exaggerates an attitudinal lift out of a sense of sympathy or desire to be helpful. And while we at Aspen have not found a way to undo volumes of published articles, we have found that time and distance can ameliorate the likelihood of bias and error. Briefly, do not interview on-site, or at least not within the shadow of your brand’s activation elements, and not by an interview staff outfitted with branded apparel. The best option is an interview after the event via telephone, internet, text, or mail. The second best option is an on-site chat, but after the experience of the activation has had some time to cool down. Either interview some distance away from the activation footprint, or recruit participants at the footprint and invite them to return after some time has passed and the natural urges to be helpful have passed.