On Tuesday, I talked about how Simpson’s paradox means you shouldn’t use just overall retention as a metric. Rather, you want to pair it with subgroup metrics so someone doesn’t achieve their retention goals by cutting off acquisition.
I’ll generalize from that. As Newton would have said if he were a direct marketer, each metric must have an equal and opposite metric.
This is because of that favorite focus of economists: incentives. If Peter Drucker was right and that which is measured gets improved, then I posit a corollary: what isn’t measured is sacrificed to improve that which is measured.
In all likelihood you don’t have evil people in your organization trying to game your metrics. However, a way to ensure this is to set up a system where metrics can’t be gamed. Like the framers of the Constitution, you want to set up a system that survives even if there are evil people in charge.
So what metrics do you want to pair?
Response rate vs. average gift: This one is the most obvious. If you measured only response rate, someone could reduce the ask string and lower the heck out of the amount of the ask in order to spike response rates. If you focused solely on gift amount, you could cherry pick leads and change the ask string to favor higher gifts. Put together, however, they give a good picture of the response to a communication.
Net income vs. file health: Anyone can hit their net income goals by not acquiring new donors. Suffice to say this is a bad idea –possibly one of the worst ideas. Likewise, an acquisition binge can increase the size of a donor base quickly, but spend money even more quickly.
Cost per donor acquired vs. number of new donors acquired: If you had to design a campaign to bring in one person, you could do it very inexpensively – probably at a profit. Each successive donor becomes harder to acquire, requiring more money. That’s why if only cost is analyzed, few donors will be acquired, and vice versa.
Web traffic (sessions or unique visitors) vs. bounce rate: Measuring only one could mean many very poor visitors or only a few very good visitors. Neither extreme is desirable.
Click-through rate vs. conversion rate: If only your best prospective donors click on something, most of them will convert. More click-throughs mean a lower conversion rate, but no one should be punished for effectiveness in generating interest.
List growth vs. engagement rates: Similar to website metrics, you want neither too many low-quality constituents nor too few high-quality ones. Picture what would happen if someone put 1,000, 10,000, or 100,000 fake email addresses on your email list. Your list would grow, but you would have significantly lower open rates and click-throughs. Same with mail – as your list increases, response rate will go down – you need to find if the response rate is down disproportionately.
Gross and net revenue: I Probably don’t even need to mention this one, but if you measure gross revenue only, you will definitely get it. You will not, however, like what happens to your costs.
Net revenue vs. ROI: Usually, these two move in concert. However, sometimes, additional marginal costs will decrease ROI, but increase net revenue per piece as in the example yesterday. In fact, most examples of this are more dramatic, involving high-dollar treatments where high-touch treatments increase costs significantly, but increase net revenue per piece more. A smart direct marketer will make judgment calls balancing these two metrics.
Net revenue vs. testing: This is clearly a cheat, as testing is not really a metric, but one way to safeguard your revenue is not to take risks. For example, mailing all control packages… using the same phone script you always have… or running the same matching gift campaign online that you did last year. Testing carries costs, but these are costs that must be borne to preserve innovation and prevent fatigue in the long run.
Revenue vs. donor satisfaction: Can you get more out of your donors by communicating more? Yes, sometimes. (Feel free to read any of 20 other Agitator posts on why this only happens sometimes.) But you will build up donor irritation. Monitoring your donor satisfaction is a canary in the coal mine for whether your short term imperils your long-term.
These are just a few of the metrics to look out for, but the most important point is that any single metrics can be gamed (whether intentionally or unintentionally). One of the easiest ways to avoid this is thinking in the extreme – how would you logically spike the metrics. From there, you can find the opposing metric to make sure you maintain a balanced program.
Nick
