# /robots.txt file for http://webcrawler.com/ # mail webmaster@webcrawler.com for constructive criticism User-agent: * Disallow: /admin/ Disallow: /RegDB/ Disallow: /beginningPhrases/ Disallow: /DONT_UPLOAD/ Disallow: /partner/ Disallow: /remove-admin/ Disallow: /remove-DesignPackages/ Disallow: /testbrand/ Disallow: /picture_library/ Disallow: /plesk-stat/ Disallow: /Templates/ Disallow: /courses/