# robots.txt files are highly configurable. For more information try a # Google search. One starting point is the following article at the Wikipedia: # http://en.wikipedia.org/wiki/Robots.txt # # The settings listed below keep spiders from indexing pages pointlessly. Disallow: /css Disallow: /img Disallow: /js Disallow: /designer Disallow: /include Disallow: /test # Feel free to add any other pages on your site that you don't want to be indexed by # the search engines. # PLEASE NOTE: Any pages that you list here should be secured by other means if you # don't want people to be able to view them, as some malicious users will look at a # robots.txt file to try to find "hidden" or "secret" areas of web sites to find # confidential information. (Glep automatically protects access to database data # no matter how robots.txt is configured.) # PLEASE NOTE: THE Google Image Bot can consume a lot of bandwidth searching for images # so it is blocked by default below. User-agent: Googlebot-Image Disallow: /