Time series analysis in AdWords

15.01.2009 Posted in Search

Internet advertisers love data. This is a statement that has become more and more true as the markets and systems have matured. A form of advertising that lets the advertiser monitor every penny of their spend and know how efficiently it is being spent raises images of equations and formulae monitoring and optimising every variable of a campaign.
But just how wise is this strategy?

Data Mining

Data Mining is a derogatory term in econometrics and statistical circles. But it is loved by business. Especially internet business where data is so readily available. It refers to a practice of trying to find useful patterns in data starting with the data.

Imagine a firm wants to find out what controllable aspect of their pay per click campaign has the biggest effect on traffic. Get all the data, regress the clicks on all the independent variables, and voila! Instant figures detailing the strength of the relationships between them. Great, isn’t it?

Well, no. Statistical analysis is all about probabilities. Depending on the volume and quality of your data you can say that I am 95% sure about this result. But when you are data mining you are finding many results from one set of data. You are asking for every relationship to find the strongest. So if you have a 95% certainty that an output is correct and you have tested 100 outputs (very easy to do with enough combinations of possible variables) then 5 outputs will come out valid. The strongest of these is not necessarily the correct one.

So you see the problem? How do you trust your results?

The answer is the scientific method. Go back to basics. Hypothesis, test, analysis. Never forget the first step. Know what you want to test before you begin, and run only that test to determine its truth. Now the chance of that test being incorrect really is just 5%.

Moving Averages

So what else can we do to analyse our data, if we can’t just look for the strongest relationships? Well we need to know what we want to look for. One of the most common types of analysis (and the focus of this article) is to do time series analysis. Time series analysis involves a detailed look at the movement of factors over time, correlating movements in the factors being tracked (clicks, impressions, ad position) against movements in the factors we control (keyword lists, ad text, CPC bids).

However the problem occurs because of the number of non-controllable (and in many cases unknowable) factors we cannot see or measure directly. How do we measure the effect of an increase in our CPC bids if we don’t know if our competitors have changed theirs? How do we monitor our daily clicks versus the day before if we don’t know the effect of the day of the week on user search patterns? We can smooth white noise changes (random changes that don’t follow a moving trend) with a multiple point moving average. Three days or five days are usually adequate. This will smooth your data so that changes like one day of low search queries will not look like a sudden drop in your data that gets you hunting for campaign changes you might have made.

Filters

Short term cyclical data (ie a drop in searches every Friday) or long term cyclical data (increases in searches around Christmas) are an aspect of a time series that cannot always be obviously seen, especially if they are complex or there are several cycles of different lengths interacting. But you can apply a filter algorithm to time series data to remove cyclical components. The Hodrick-Prescott filter is one of the most common. The details of this formula are outside the scope of this article, but in summary it includes two terms, one that penalises a cyclical component and one that penalises differences in the trend. By adjusting the coefficient of the second term it is possible to give each term different weighting within the formula to filter out more or less each component when the formula is minimised.

This (and several other examples of time series filters) can be useful for large amounts of data where random changes are characterised as white noise. However the random changes in a pay-per-click campaign are difficult to characterise as white noise. They will have cycles of their own. I.e. competitors may change their CPC bids around a different set of non-random patterns to yours.

Conclusion

There are plenty of methods available for data-heavy reports for internet advertising to be analysed. It can be very tempting to use this richness of data to run your campaigns. But be wary: incorrect analysis is worse than no analysis. It is easy to begin running campaign according to complex, and completely invalid statistics. You can either make your high-quality campaigns underperform, or fail to ever realise the potential of what looks like a mediocre campaign.