In order to avoid undesirable content within the lookup indexes, webmasters can instruct spiders to not crawl specified files or directories in the typical robots.txt file in the root directory in the area. Furthermore, a web site could be explicitly excluded from a internet search engine's database by making use of a meta tag distinct to robots (g