Re: Tracking and Looksmart

From: Mark Bishop <>
Date: Tue 23 Jul 2002 20:43:13 -0500

Crawlers are smart enough not to jump into a /cgi-bin directory because of
the possibility of running into some sort of linking database-a mind field
to a search engine. In addition, some crawlers have difficulty in reading
specific symbols like: ?, &, %, + and $.

When you use dynamic content that's created on the fly, there's really not
much an engine can index. However, there are methods that can help you to
get listed and perhaps obtain a deeper crawl.

If your only problem is getting past the form page, definitely include as
many links as possible. Avoid frames, Flash, Javascript, or anything else
that might distract the crawler/spider on this important main page.

Create a separate "crawler" HTML page that lists your important links on
it. I often will achieve this by creating by adding a single clear pixel
with a header tag pointing to my crawler web page. Here's an example:

<h5><a href="crawler_mywebsite.html"><img src="clearpixel1.gif" height="1"
width="1" border="0" alt="Description Here"></h5></a>

In addition, you can also get your page indexed into the PPC (Pay Per
Click) engines as they will index just about anything. Google will index
dynamic content too.

Here's a very good article in getting dynamic web content indexed.

I hope this helps.


Mark Bishop

Received on Tue Jul 23 2002 - 20:43:13 CDT


With an archive of more than 14,000 postings, since 1996 the Online Advertising Discussion List has been the Internet's leading forum focused on professional discussion of online advertising and online media buying and selling strategies, results, studies, tools, and media coverage. If you wish to join the discussion list, please use this link to sign up on the home page of the Online Advertising Discussion List.


Online Advertising Industry Leaders:

Local SEO with Video
Houston SEO
Austin Web Design

Add your company...

Local SEO with Video


Online Advertising Discussion List Archives: 2003 - Present
Online Advertising Discussion List Archives: 2001 - 2002
Online Advertising Discussion List Archives: 1999 - 2000
Online Advertising Discussion List Archives: 1996 - 1998

Online Advertising Home | Guidelines | Conferences | Testimonials | Contact Us | Sponsorship | Resources
Site Access and Use Policy | Privacy Policy

2323 Clear Lake City Blvd., Suite 180-139, Houston, TX 77062-8120
Phone: 281-480-6300
Copyright 1996-2007 The Online Advertising Discussion List, a division of ADASTRO Incorporated.
All Rights Reserved.