Re: Tracking and Looksmart
"Jill Paskoff" <jillp_at_specsource.net> wrote:
> Our site is database driven and I believe spiders don't crawl it because
> some selection is required by the user to continue (a form action). Once
> the initial selection is made, the links are infinite. Is there a way to
> get the spiders past the initial selection process so that they can crawl
> the remaining pages?
Jill, create a page that generates a list of hyperlinks to all of the
database-driven pages. It should be trivial to build the page dynamically.
Then add a link to that page from a page which is indexed by search engines,
such as your main page. If you don't want humans to be able to see the link
or visit the linked page, you'll have to control access based on the user
agent field or IP addresses through your webserver (easy to do in Apache) or
code within your linking page and hyperlink list page (trivial). Since user
agents can be spoofed by humans and more and more search engines are
spoofing user agents or making theirs harder to predict, this isn't
foolproof so if it's important to you that search engines see the page and
humans don't you'll want to research this further.
--
Steve Werby
President, Befriend Internet Services LLC
http://www.befriend.com/
Received on Tue Jul 23 2002 - 20:43:56 CDT
