Kitcatt Nohr Alexander Shaw

They say we live in testing times but I'm not so sure.

Do you do as much testing as you used to? Or do you see as much testing going on generally in DM? Allow me to express the question differently. If I gave you a significance table, would you know what to do with it? Or would you associate it with a Native American ritual, like the peace pipe?

I'm hopefully still too young to qualify as an old fart, but when I came into the industry I had a book of significance tables thrust into my hand as part of my induction, as well as books by American direct marketers showing the most anal tests imaginable. List tests, outer envelope tests, copy length tests, picture tests, handwriting tests, reply form tests, incentive tests, seasonality tests, day of the week tests, frequency tests... did these people not have a life?

The justification was clear. If Test A outperforms Test B by 0.5 per cent (or whatever constituted a significant difference) then that could result in 500 extra policies sold. But that kind of thinking seems to be deeply unfashionable now, apart from in a handful of companies who continue to churn out large volume mailings. Why is that?

Having pondered the question and tested it on a few colleagues, the answers seem to break down into two categories: the defensible explanation and the indefensible. The indefensible explanations include the following (with ready-made responses in parenthesis):

- Testing two things costs more than testing nothing (er, yes, but the potential upside is greater if you learn something useful).

- Testing is pointless because we know the answer anyway (really? When did we all get so smart?).

- Testing takes up too much time because you have to wait for the results (why are we all in such a hurry these days? And anyway, has no-one heard of online testing? It can be done overnight).

- Testing implies sending a stronger offer/idea/incentive to some groups and a weaker one to the other, which risks not optimising the activity (yup, but if you knew which one was weak you wouldn't be testing it, would you?).

And my favourite one of all - testing is risky, and implies the possibility of failure (uh-ha, but what's the alternative, mediocrity necessitated by habitual risk aversion?).

As you can see, I have a patience deficit when it comes to sloppy reasoning. But once you've considered and rejected the more feeble arguments against testing, there are some more robust and illuminating ones too. One is that marketing has become more sophisticated, an observation which has several different manifestations. Such as media planning - campaigns may layer brand and response objectives, and so combine brand and response media, making it difficult to isolate the differential effect of one channel over another.

And metrics have become less one-dimensional too. Single (response) metrics have given way to softer brand metrics (Version A is more in tune with the way the brand talks than Version B), or qualitative concerns (Version A may pull harder than Version B but it might also alienate more consumers).

Campaign planning has become more sophisticated too. The impact of account planning on agency processes means that consumer attitudes are often probed prior to campaigns being briefed, concepts are shown to consumers before they run live and technological advances in testing (such as online panels) mean that campaigns may well be more robust before they're ever sent to consumers.

Which takes us where exactly? The arguments against testing are all worthy of consideration, but none of them actually invalidate the principle. Namely that by testing intelligently (testing the right things, at the right time, in the right order) we can encourage innovation without throwing out what we know works. And we can perpetuate the cycle of learning which seems to have lost favour in recent years.

Moreover, the opportunities for testing have never been greater. Heard of online? Email testing - be it in subject lines, content, style or length of emails. And how about website testing - look at the average web page and tell me there aren't 20 things you couldn't test straight away - from overall site architecture to navigation to copy.

And if this is all too granular, consider this. Audience testing. Media testing. Offer testing. They're as valid now as they were 10 years ago, just more challenging to execute. But who said life was going to get easier? They say we're living in testing times. I hope so.

This column originally appeared in Marketing Direct.