ontrary to popular belief, However this is not always fool
the search engine spiders proof. Search engine spiders have
sent out by the major a habit of going away from a page
search engines do not have to and then coming back and looking
search everything on a site. You at the page a second time later.
can actually technically keep a As a search engine crawler may
search engine spider away from a keep a cached copy of this file,
page by instructed it through a it may on occasion crawl pages a
certain robots meta tag or a file webmaster does not wished
not to come near the page. crawled.
Webmasters can instruct spiders Pages that most webmasters prefer
not to crawl certain files or not be crawled include login
directories through the standard specific pages such as shopping
robots.txt file in the root carts and user-specific content
directory of the domain. such as search results from
Additionally, a page can be internal searches. Other pages
explicitly excluded from a search that you might not want crawled,
engine's database by using a depending on the content might be
robots meta tag. If for some a guest book that you expect to
reason you do not want a search be filled with spam or a feedback
engine spider to crawl a page you system that is not very
do have the means to do so. flattering to you. It is also a
good idea to instruct the spiders
When a search engine visits a not to crawl a page with a lot of
site, the robots.txt located in animation or flash on it as this
the root folder is the first file can be mistakenly read by a
crawled. The robots.txt file is spider as a malfunctioning site.
then parsed, and only pages not
disallowed will be crawled.
About the Author:
Chris Angus is a SEO and website promoter, he can be contacted at sales(at)brilliantseo.com Car Insurance
Mobility Scooters
Asbestos
Read more articles by:
Chris Angus
Article Source: www.iSnare.com 
