Chapter 15
Getting to the Bottom Line: Tracking and Measuring Your Campaigns
In This Chapter
Defining what counts as a response
Calculating basic cost metrics
Figuring response metrics
Assigning revenue to your campaigns
As I say repeatedly throughout this book, your ability to measure the success of your campaigns sets your database marketing efforts apart from other marketing disciplines. Other disciplines can measure costs and, to some degree, the benefits associated with their efforts. But you can use your database to give very precise financial results.
This ability arises from your use of the scientific method discussed in Chapter 14. You approach your campaigns as experiments. This means that you form a hypothesis and then proceed to test that hypothesis in a systematic way.
In this chapter, I examine some of the most common metrics that are used to evaluate database marketing campaigns. Ultimately, I discuss the assignment of financial metrics that allows you to clearly and convincingly measure your contribution to your company’s bottom line.
Defining Responses Clearly: A Couple of Things to Keep in Mind
In Chapter 14, I talk about creating a tracking mechanism when you set up your campaigns. You want to clearly define what counts as a response. And you want to be sure that you can connect responses back to the target audience for your campaign. This needs to be done before you execute your campaign.
But you need to keep in mind a couple of things after the fact. In this section, I talk about some pitfalls related to counting responses. I also explain a method for using response data to figure out how long you should wait before you stop counting customer behavior as being in response to your campaign.
Counting responses
The expression the devil is in the details is particularly true when it comes to counting responses to database marketing campaigns. If you’re not careful, it’s easy to overstate or understate the number of responses you’ve received to your campaign.
Be clear about what counts as a customer
My wife and I recently bought a new car. I’d received an offer in the mail saying that the dealership would like to buy back my car and apply the balance to a new one. Because we had just moved from the South to the North, I was actually concerned about the car my wife was driving because she didn’t have any experience driving in the snow.
So we went into the dealership and ended up buying a new car for her, and I started driving her old car. Because the new car was hers, we put it in her name. My name appeared nowhere in the transaction. Now here comes the question: Does this purchase count as a response to the direct-mail campaign that was sent to me? The answer is unequivocally yes.
Another variation on this example occurs in catalog retailing. If you send a catalog to my house, it’s perfectly plausible that both my wife and I will separately make purchases from it. She may order a new suit, and I may order a new pair of shoes a few days later. Again, both purchases can be reasonably associated with that catalog campaign.
You want to be a little bit careful in this situation, though. You may credit the revenue from both purchases to your campaign. But you probably don’t want to count these purchases as two separate responses. A response is usually defined as a household that makes at least one purchase.
Be careful with online data
The online world is more complicated in other ways as well. In the case of e-mail campaigns, responses are usually tracked back to the e-mail address. Other online marketing campaigns may be tracked back to registered users. Even anonymous users can be tracked by dropping cookies on their machines when they visit your website, as I discuss in Chapter 13.
In the case of direct mail, the usual challenge is to make sure you count everything you can as a response. You’re trying to avoid undercounting. Online, the challenge is frequently the exact opposite: There are a lot of ways to overcount if you’re not careful.
It may seem to you that a purchase is a purchase, so how could you overcount them? One answer is that you may attribute the same purchase to two different people.
Here’s an example: Suppose you send an e-mail offer to a target audience. Some of that audience is registered users of your website, and some aren’t. You decide that you’re going to track results at the e-mail level.
Now suppose further that both my wife and I receive your e-mail offer. She’s registered on your site, and I’m not. So you have no way of knowing that we’re in the same household. She ignores your e-mail, but I bite. When I go to your website, I use my wife’s login credentials. When I make the purchase, I enter my own e-mail address to track the package.
Here’s where the problem comes. You now have two different e-mail addresses associated with that purchase, mine and the one my wife used when she registered. And you mailed your offer to both of them. Which one do you associate with the purchase?
Closing the tracking window: How long do you wait for responses?
You can’t wait forever to evaluate the success of your database marketing campaigns. A limited amount of time can pass during which you can reasonably attribute purchases or other responses to a particular campaign.
If your campaign includes an offer that expires after a given period of time, then you obviously only track responses that occur during that offer window. But in cases where there’s no inherent time limit to the offer, you need to put a little more thought into how long to wait before you consider the campaign over. In Chapter 14, I hint at a way to use your control group to do this.
Your control group is a randomly selected sample of your target audience, remember. The control group is held out from your campaign — in other words, you don’t contact customers in the control group. The basic idea is that the behavior of the control group represents what the target audience would have done had you not contacted them. This allows you to measure the effectiveness of your campaigns, as I explain later in this chapter.
At first glance, you may think that eventually “responses” will simply stop. That your response rate will drop to zero. But you’re not operating in a vacuum. There are always other advertising and marketing campaigns out in the marketplace that are driving customers to purchase, register, or whatever else you might want them to do. This means that you’ll continue to see purchases among your target audience that are unrelated to your specific campaign.
Getting a Handle on Costs: Some Common Metrics
When evaluating your campaigns, you obviously need to understand their costs. In fact, you need to understand costs even before you execute your campaigns. You have a limited budget and you want to make sure you’re using it effectively.
In most cases, these cost metrics relate to costs that are specifically tied to a given campaign. You typically won’t try to factor in the cost of your salary or the department copy machine into your analysis of marketing campaigns.
The costs you’ll focus on have to do with the production and delivery of your marketing messages. In the case of direct mail, creative development, printing, and postage costs are taken into account. In the online world, development costs, production costs for videos, and e-mail delivery and tracking charges are typical examples of costs that are assigned to campaigns.
Database marketers have inherited some traditional cost metrics from the broader disciplines of marketing and advertising. These are very simple calculations that you’ll see (and make) again and again. In what follows, I explain these calculations and how they’re used.
Cost per thousand: CPM
An extremely simple but extremely common way of measuring costs is to calculate what it costs to reach a thousand customers. This is known as your cost per thousand, usually abbreviated CPM. The roman numeral M represents one thousand.
Using CPM in database marketing campaigns
CPM = 1,000 × (total campaign cost / target audience size)
Don’t get confused: A similar metric in advertising
In the case of advertising, costs are typically reported based on the size of a TV or radio audience or on the basis of the subscriber base to a newspaper or magazine. This type of reporting is also common in online advertising, when banner ads or popups are purchased, for example.
In this situation, advertisers speak about the cost per thousand impressions. An impression essentially means someone had a chance to see the ad. It amounts to the number of times that an advertisement was put in front of a customer, regardless of whether they actually saw it or not.
For a newspaper advertisement, the number of impressions would be the newspaper’s circulation. For an online popup ad, it would be the number of times the ad popped up.
In these cases, the cost per thousand impressions or CPMI are calculated by replacing the target audience size with the number of impressions, as follows:
CPMI = 1,000 × (advertising cost / number of impressions)
Using CPM to compare campaigns
The main advantage of CPM metrics is that they make it easier to compare costs among different types of marketing campaigns. Your marketing executives are constantly balancing the trade-offs between different marketing and advertising strategies. CPM metrics give them a sense of the relative costs of reaching consumers through different channels.
CPM metrics all represent costs for a standard audience size of 1,000 consumers. This means they can be compared head to head without having to take differences in audience size into account. They represent a way of comparing the efficiency of different marketing programs.
Measuring Marketing Effectiveness in the Online World
When you communicate using direct mail, you’re basically in the dark from the time the mail drops until the customer heeds your call to action. The only real feedback you get in between has to do with whether your mail piece actually got delivered. If you pay for return service, you can use the return mail to clean up your database, but that’s about it.
Your online marketing presence, on the other hand, generates one heck of a lot of data about consumer behavior. This means that there’s also one heck of a lot of ways to approach measurement online. Whether you send an e-mail or recognize a visitor on your website, you can see everything that’s going on.
In this section, I explain a few common metrics that are used to understand consumer behavior online. I point out (repeatedly) some pitfalls related to overcounting transactions online. I also discuss some of the advantages and shortcomings of various ways of looking at this data.
Getting the customer to your website: Metrics related to e-mail campaigns
When you execute an e-mail campaign, you almost always include a link to your website. You may be trying to increase website registrations. You may be trying to drive purchases. You may simply be trying to increase website traffic. In any case, you can see every interaction the customer has with you from the moment they open your e-mail.
Did the e-mail get delivered? Understanding bounce rates
When an e-mail fails to be delivered, it’s said to have bounced. Your bounce rate — the percentage of your e-mails that bounce — is a measure of the quality of the e-mail addresses in your database. And it’s something you need to pay attention to.
An e-mail may bounce for a couple different reasons. The first reason, known as a soft bounce, is when an e-mail server is busy, down for maintenance, or otherwise simply can’t process a delivery request. These aren’t particularly problematic, and many e-mail service providers will simply queue these soft bounces up and re-send them later.
The more problematic situation is when you e-mail an invalid, expired, or nonexistent e-mail address. These hard bounces need be removed from your database.
Chapter 4 talks about complying with regulations regarding e-mail spam. If you appear as though you might not be complying, your e-mail service provider will simply stop servicing you.
Did the customer opt out? Understanding retention rates
If an e-mail hard-bounces, the address is no longer of any use to you. Another way e-mail becomes useless is through opt-outs.
It’s standard practice in e-mail marketing, to include a link at the bottom of the e-mail message that allows the customer to opt out of hearing from you. If they click this Unsubscribe link, then they have effectively removed their e-mail from all future mailing lists. These are known as unsubscribed e-mail addresses.
Unsubcribes are used in a common metric that measures how much your e-mail list shrinks due to a campaign. Because you’ll invariably have bounces and unsubscribes in any campaign, any campaign has the effect of reducing the number of e-mail addresses available to you. The subscriber retention rate, which I abbreviate SRR, measures how many e-mail addresses survived your campaign. Here’s the calculation:
SRR = 100 × (target audience size – bounces – unsubscribes) / (target audience size)
Did the customer see the e-mail? Understanding open rates
Another widely reported metric related to e-mail campaigns is the open rate. At first blush, this may sound like an incredibly simple metric. It’s just the percentage of e-mails that you send that are opened, right? Wrong. It’s actually a little more involved than you might think.
First of all, you want to calculate the open rate as a percentage of delivered e-mails. You exclude bounces from your open rate calculation. Second, the definition of opened requires a little explanation.
When tracking open rates, e-mail marketers use the term open as shorthand for the technical term tracked open. To count as a tracked open, the customer has to do more than just open the e-mail. The customer needs to interact with the e-mail. One way to interact with the e-mail is to click a link in the e-mail.
But there’s another way that an e-mail is counted as a tracked open. If the customer downloads the images in the e-mail, it counts as an interaction. I usually have my e-mail account set up to block remote images. If I open an e-mail and then click the Download Images button, that e-mail counts as tracked open. If I don’t, it doesn’t.
Typically, you want to count an e-mail as opened only once. What you’re interested in is how many e-mails got seen. If someone opens an e-mail three different times, this may tell you something about their interest. But you don’t want to count it three times toward your open rate. In other words, open rate usually means unique, tracked opens.
So with all that in mind, here’s the formula for calculating the open rate for an e-mail campaign:
Open rate = 100 × (unique tracked opens) / (target audience size –bounces)
Did the customer bite? Understanding click-through rates
Your e-mail campaigns are generally designed to drive customers to your website. You usually include one (or more) website links in your e-mail campaigns. So it’s only natural for you to want to understand what percentage of your target audience is actually making it to your website. This metric is called click-through rate or CTR.
What you’re really interested in is unique click-throughs. The traditional way of calculating click-through rates is by dividing unique click-throughs by the number of e-mails that are delivered:
CTR= 100 × (unique click-throughs) / (target audience size – bounces)
There’s another, related, metric that’s sometimes used to understand how effectively an e-mail campaign is at driving web traffic. This metric, known as the click to open rate, or CTOR, compares unique click-throughs to tracked opens rather than to delivered e-mails:
CTOR = 100 × (unique click throughs) / (unique tracked opens)
Understanding browsing behavior: Some simple web metrics
As I point out in Chapter 13, click-throughs don’t tell the whole story. Once a customer lands on your website, a whole other world of behavior data becomes available to you. Understanding what customers are doing on your site once they arrive can give you clues to what is and isn’t working. A number of companies offer these services. The largest is Google, which offers many metrics free of charge. Omniture (now owned by Adobe) charges for these services but also customizes the reporting platform to integrate it with your other data.
In this section, I describe some simple metrics that measure, to some degree, how interested a customer is in your website. These metrics all represent some sort of rate or average for a given audience. When describing these calculations, I refer generically to site visits, or the number of visitors to the site.
However, a user can land on your web site in a number of different ways. They can click an e-mail link. They can click a banner ad or popup ad. They can click a link from another site. They can click a link served up by a search engine. In all these cases, you’ll have visibility to how they got there.
This gives you wide latitude to define exactly what site visits you’re interested in when you calculate browsing metrics. You can compare these metrics between and among different audiences based on how they landed on your website.
Understanding what a web session is
Most web metrics depend to some extent on a couple of simple ideas. The first is the notion of a page view. This means exactly what you think it means. An individual user clicks on a link and a particular web page appears on their screen. Tracking page views is a central part of doing analysis of users’ web-browsing behavior.
Another fundamental concept is a site visit, sometimes called a web session. Your intuitive understanding of this term is also correct. A site visit starts when the user lands on a website and ends when they leave. But it’s important for you to understand the nitpicky definition of what a site visit actually is.
One of the original metrics that was used to evaluate a user’s interest in a website involved looking at how long a web session lasted. Metrics about session duration are still used. But many marketers now prefer other methods of gauging user interest.
Because of the way a session is defined, you don’t have to worry about situations where a user leaves the browser open and goes to bed. You won’t get metrics back saying that a user browsed your website all night. But there’s still confusion about how session duration is defined.
Another common confusion about web sessions is that they don’t require a user to remain on the site the whole time. They can browse another site and come back to yours without being deemed to have ended one session and begun another.
Did they bother to stay? Bounce rates
The simplest and clearest signal that a customer isn’t bowled over by whatever page they’ve landed on is that they immediately leave. The way this is generally measured is by looking at web session data.
Each web session contains a specific number of page views. If a user lands on your page and never requests another page view, then the number of page views is 1. This single page view session is called a bounce. Your bounce rate is calculated as follows:
Bounce rate = 100 × (single page view sessions) / (site visits)
In the context of web-browsing metrics, the term abandonment rate refers specifically to shopping cart abandonment. In fact, it’s often called the cart abandonment rate. It measures the percentage of users who get so far as to place an item in their shopping cart, but fail to actually complete their purchase.
How interested were they? Understanding the depth of their visit
As I mention earlier in this chapter, one common measure of how interested a user is in your website is how long they stayed on your site. But many marketers have come to prefer page view counts over time to measure interest. Page views indicate interaction with the website which is more clearly associated with a user’s interest.
There are a couple of simple metrics associated with page view counts. One is the average number of page views per site visit. This is called average page depth.
Average page depth = (page views) / (site visits)
There are some variations on how this metric is used. Often, you’ll be interested in how long (in terms of page views) it takes a user to get around to doing something in particular, like registering or making a purchase. It’s easy enough to modify this metric to do that. For example, you could make the following calculation:
Page depth per purchase = (page views before purchase) / (purchases)
Another metric related to page views measures the percentage of your sessions that reach some threshold. You may notice that most purchases don’t happen until a certain number of pages have been viewed. For the sake of this example, let’s say that number is 7. You might then be curious how many sessions last that long. For example, you might make this calculation:
Seven-page visit rate = 100 × (seven page sessions) / (site visits)
There’s seemingly no end to the metrics that are potentially available to help you understand how users are using your site. Many of these metrics are also available from search engines to help you understand how users are managing to find your website. In the online world, change is continual and fast. So stay abreast of what new data and metrics are being developed.
How Did You Do? Assigning Value to Your Database Marketing Campaigns
Chapter 14 talks at length about setting up your campaign as an experiment. The key component of that experiment is your control group. You’ve held out a random sample of your target audience from your communication. You’ve executed the campaign and collected the response data. Now it’s time to put that control group to use.
Understanding lift: Calculating your net response rate
In controlled experiments that test medications, the placebo effect is well documented. Some patients who receive a placebo — meaning they don’t receive the actual treatment — still show signs of improvement. This placebo effect needs to be taken into account when the effectiveness of the drug is reported.
You’re in a similar situation with respect to measuring your direct-marketing campaigns. You can’t take sole credit for the responses to your campaign. Advertising campaigns and other marketing efforts have contributed to your customers’ awareness of and interest in your products. But you can take sole credit for some of it. This is what that control group is for.
Gross response rate
The first thing you need to do is calculate the overall response rate, called gross response rate, to your campaign. Suppose you sent out 100,000 communications and 6,000 consumers in your target audience actually responded. They bought something from you. Your gross response rate is the ratio of these two numbers — in this case, 6 percent. The formula is this:
Gross response rate = 100 × (total responders) / (target audience size)
Net response rate
Once you have your gross response rate in hand, you need to look in on your control group. What were these folks doing while your campaign was in market?
To continue my example, let’s suppose that your geek had recommended that your control group contain 5,000 households. Of these 5,000 households, 250 made a purchase that would have qualified as a response to your campaign. That means the control group response rate is 5 percent. Your control group response rate is calculated the same way as your gross response rate:
Control group response rate = 100 × (control group responders) / (control group size)
Now comes the key to your being able to take credit where credit is due. The difference between the gross response rate and the control group response rate is all you. This difference is known as the net response rate — or more commonly, the lift — associated with your campaign:
Lift = (gross response rate) – (control group response rate)
This lift represents the proportion of the responses that can be attributed specifically to your communication. The only difference between the target audience and the control group was whether or not they received your message.
In this example, you had a 6 percent gross response rate and a 5 percent control group response. This gives you a lift of 1 percent, or 1,000 responses that you can take sole credit for. A check with your geek will confirm that this lift is significant — well above the 95% confidence level, given the 5,000 members in your control group.
The bottom line: Net revenue and return on investment
Now that you’ve calculated your lift and hopefully verified that it’s significant, it’s time to calculate your contribution. (If you don’t have significant results, then you can’t justify claiming any contribution.) This calculation depends on two additional numbers. You need your total campaign cost and the total revenue that’s tied to your campaign.
Suppose your campaign cost was $350 per thousand, or 35 cents per piece. This is actually pretty typical of a large postcard campaign. In this case, your total campaign cost would be $35,000:
Total campaign cost = (cost per piece) × (target audience size)
Your total revenue is simply the sum of all the purchases made by your responders. Remember that it’s okay to count multiple purchases made by the same responder as long as those purchases fit your definition of a response.
Let’s say that your 6,000 responders purchased on average $100 worth of merchandise. That means your total revenue came out to $600,000. This is usually referred to as the gross revenue associated with the campaign. Your net revenue is simply gross revenue minus campaign cost.
Now you’re ready to claim credit for your share of that revenue. That share, or the incremental revenue associated with your campaign, is calculated by applying your lift percentage to net revenue:
Incremental revenue = lift × [ (gross revenue) – (campaign cost) ]
The net revenue turns out to be $565,000 ($600K minus $35K). Because your lift is 1%, you can take credit for a $56,500 contribution — a pretty good return on a $35,000 investment. And in fact, campaign results are often presented exactly that way. Your return on investment or ROI is calculated as follows:
ROI = 100 × [(incremental revenue) – (campaign cost)] / (campaign cost)
In the example, this turns out to be a little over 61 percent. This example isn’t particularly outlandish. Database marketing can be an extremely efficient way to spend marketing dollars.