Re: Tracking and Looksmart

From: Alexis D. Gutzman <alexisg_at_marketingsherpa.com>
Date: Thu 25 Jul 2002 11:59:20 -0500

I had said:

> > If you need to be absolutely sure that only spiders make it
> > to the protected pages without getting the form, you can use server-side
> > detection to identify the footprint of the spider (there aren't that
> > many that really matter) and let the server show these agents the
> > content behind the barrier without seeing the form.

Steve Werby replied:
> I think your reply and my earlier reply to Jill crossed paths, but I want
> to reiterate that this isn't foolproof. If you know what user agents most
> search engines regularly use and you know how to set your user agent, then
> you can trick sites that don't know better into thinking you are a speciif
> search engine. In fact, that's a common technique used to view content,
> including meta tags, which competitors publish for specific search
> engines.Plus, many search engines have caught onto the specialized
> content/tag game
> and are now indexing, at least in part, with different user agents and
> sometimes different IP ranges to determine if sites are using this
> technique. If you combine user agent and IPs it'll be a better solution,
> though coming by the info. required to do that will probably be difficult.

You are, of course, correct that combining the two is better, and there may
be ways that SEs try to confirm results by sending different agents out from
different IP addresses. And yes, of course, agent footprints can be spoofed.
However, a competitor (in this example) could just fill out the form as
Elmer Fudd and see the content that's currently protected.

What marketers need to understand is that *very few search engines matter*.
Recent traffic data suggests that (at least for US traffic) only ~4 spiders
matter. As of spring, SE's ranked by reach, were: Yahoo, MSN, Google, AOL,
AskJeeves (with just under 10%). Every other "search engine" had
considerably less. Yahoo gets its spidered results from Google. MSN and AOL
get their spidered results from Inktomi. AskJeeves gets theirs from Teoma
(themselves). This means that three spiders matter. I would count Alexa in
because they're part of Amazon and you can't ever count Amazon out. That
makes four spiders. I wouldn't lose any sleep over whether DogPile or
Mama.com could find my protected content. The footprints of these spiders
are well known.

Still, the site map option is the cleanest, because no one will call you a
spammer for using it. This other stuff is very questionable, and Google has
zero tolerance for questionable tactics.

Regards,
Alexis
---
Alexis D. Gutzman, Managing Editor
MarketingSherpa's Knowledge Store
http://sherpastore.com <- SEO Guide now on sale!





Received on Thu Jul 25 2002 - 11:59:20 CDT