Re: Ad Click EMetrics - Filtering out spiders and robots

From: Cliff Kurtzman <moderator_at_o-a.com>
Date: Wed 10 May 2006 15:08:23 -0500

Suzanne Errera wrote:

> I manage online advertising for a large corporation. We have ads on
>several sites like Yahoo, AOL, MSN, Weather, etc. I pull reports from those
>sites on our buy and evaluate how our buy and our creative are performing
> based on number of clicks vs number of impressions shown.
> My client is using webtrends to track how many people come to their
>website. The number of clicks that she shows on her webtrends report is
>significantly lower than the number of clicks I show went to her website
>from Yahoo, AOL, MSN, etc. reports I pull.
>Could it be that the bots and spiders are showing as clicks to Yahoo and the
>others but not counting as clicks/ visitors to my clients website?
>Does anyone have any other explanations?

I'd love to have folks from Google/Yahoo! chime in with more
definitive responses, but this is my sense...

Web page ads which are generated from Javascript code (like Google
AdSense and Yahoo Publisher ads) are probably not as likely to be
"clicked on" by spiders and robots. Try this test... go to
http://www.o-a.com and then look at the source code for the page in
your browser. You can see the hyperlinks for all the self-hosted ads,
but for the Yahoo Publisher and AdJungle hosted ads, you don't see
the links in the HTML, all you see is the Javascript code. That means
that a robot or spider isn't going to see those links and click on
them either, unless it is also executing the Javascript code.

Now go to Google and do a search on the phrase "online advertising"
and again look at the source code for the resulting page. Here is the
direct link to it:
http://www.google.com/search?hl=en&lr=&q=%22online+advertising%22&btnG=Search

The ads and search results on this page are not Javascript generated,
so if a bot or spider follows the above hyperlink, it might SEE the
links and potentially click on them. But also consider that this is
entirely a user generated page based on my search. Unless I place
that hyperlink onto the web somewhere, no spider will find it.

Of the two sites I mentioned in my previous post, I should have
mentioned that one had on the order of 10,000 visitors a month while
the other site had on the order of 100,000 visitors a month. My guess
would be that on sites with greater human traffic, the ratio of human
to spider generated ad clicks will improve, whereas on sites with
less human traffic, the ratio will likely be even worse.

I'm also seeing a more frequent pattern in log files of human clicks
in which there is no referral data provided when the user first links
into the web site, but referral data is provided for subsequent
clicks internal to the web site. I used to take this to indicate that
the person was entering the site by typing in the URL instead of
clicking a link. But that no longer seems to be the case in most
instances... some web browsers must have a privacy option that allows
the user to prevent referral from being provided to the host when
entering a new site. We know this for certain when the user is
linking between two sites, both of which we have log data on, so we
know that a link was clicked in the originating site, but the
referral URL doesn't show up in the destination site.

This might explain some of what you are seeing. The first
thing I'd do in your situation would be to analyze the raw logs to
understand what WebTrends is filtering out (sometimes nothing beats
using the UNIX grep command to understand what is really going on).
I'd also be making sure all your ad links go to unique URLs that
identify the source (e.g., linking to http://www.o-a.com/?adfromgoogle
instead of simply http://www.o-a.com) so that you are able to see the
traffic from each ad without relying on referral data. Then if the
discrepancy still remains, it is time for talking with the ad seller to
resolve the problem.

--Cliff

Clifford R. Kurtzman
CEO | Moderator
ADASTRO Incorporated | The Online Advertising Discussion List
http://www.adastro.com | http://www.o-a.com
(281) 480-6300








Received on Wed May 10 2006 - 15:08:23 CDT