Re: Tracking and Looksmart
I had said:
> > If you need to be absolutely sure that only spiders make it
> > to the protected pages without getting the form, you can use server-side
> > detection to identify the footprint of the spider (there aren't that
> > many that really matter) and let the server show these agents the
> > content behind the barrier without seeing the form.
Steve Werby replied:
> I think your reply and my earlier reply to Jill crossed paths, but I want
> to reiterate that this isn't foolproof. If you know what user agents most
> search engines regularly use and you know how to set your user agent, then
> you can trick sites that don't know better into thinking you are a speciif
> search engine. In fact, that's a common technique used to view content,
> including meta tags, which competitors publish for specific search
> engines.Plus, many search engines have caught onto the specialized
> content/tag game
> and are now indexing, at least in part, with different user agents and
> sometimes different IP ranges to determine if sites are using this
> technique. If you combine user agent and IPs it'll be a better solution,
> though coming by the info. required to do that will probably be difficult.
You are, of course, correct that combining the two is better, and there may
be ways that SEs try to confirm results by sending different agents out from
different IP addresses. And yes, of course, agent footprints can be spoofed.
However, a competitor (in this example) could just fill out the form as
Elmer Fudd and see the content that's currently protected.
What marketers need to understand is that *very few search engines matter*.
Recent traffic data suggests that (at least for US traffic) only ~4 spiders
matter. As of spring, SE's ranked by reach, were: Yahoo, MSN, Google, AOL,
AskJeeves (with just under 10%). Every other "search engine" had
considerably less. Yahoo gets its spidered results from Google. MSN and AOL
get their spidered results from Inktomi. AskJeeves gets theirs from Teoma
(themselves). This means that three spiders matter. I would count Alexa in
because they're part of Amazon and you can't ever count Amazon out. That
makes four spiders. I wouldn't lose any sleep over whether DogPile or
Mama.com could find my protected content. The footprints of these spiders
are well known.
Still, the site map option is the cleanest, because no one will call you a
spammer for using it. This other stuff is very questionable, and Google has
zero tolerance for questionable tactics.
Alexis D. Gutzman, Managing Editor
MarketingSherpa's Knowledge Store
http://sherpastore.com <- SEO Guide now on sale!
Received on Thu Jul 25 2002 - 11:59:20 CDT
HOW TO JOIN THE ONLINE ADVERTISING DISCUSSION LIST
With an archive of more than 14,000 postings, since 1996 the
Online Advertising Discussion List has been the Internet's leading forum focused on professional discussion
of online advertising and online media buying and selling strategies, results, studies, tools, and media
coverage. If you wish to join the discussion list, please use this link to sign up on the home page of the Online Advertising Discussion List.|
Online Advertising Industry Leaders:
Local SEO with Video
Austin Web Design
Add your company...