Re: Tracking and Looksmart

From: Mark Bishop <markb_at_aala.com>
Date: Tue 23 Jul 2002 20:43:13 -0500

Crawlers are smart enough not to jump into a /cgi-bin directory because of
the possibility of running into some sort of linking database-a mind field
to a search engine. In addition, some crawlers have difficulty in reading
specific symbols like: ?, &, %, + and $.

When you use dynamic content that's created on the fly, there's really not
much an engine can index. However, there are methods that can help you to
get listed and perhaps obtain a deeper crawl.

If your only problem is getting past the form page, definitely include as
many links as possible. Avoid frames, Flash, Javascript, or anything else
that might distract the crawler/spider on this important main page.

Create a separate "crawler" HTML page that lists your important links on
it. I often will achieve this by creating by adding a single clear pixel
with a header tag pointing to my crawler web page. Here's an example:

<h5><a href="crawler_mywebsite.html"><img src="clearpixel1.gif" height="1"
width="1" border="0" alt="Description Here"></h5></a>

In addition, you can also get your page indexed into the PPC (Pay Per
Click) engines as they will index just about anything. Google will index
dynamic content too.

Here's a very good article in getting dynamic web content indexed.

http://spider-food.net/dynamic-page-optimization.html

I hope this helps.

Sincerely,


Mark Bishop



Received on Tue Jul 23 2002 - 20:43:13 CDT