User-agent: * Disallow: /rest/ Disallow: /xml/ Disallow: /json/ Disallow: /atom/ Disallow: /opml/ Disallow: /rss/ Disallow: /widget/ Disallow: /search? Disallow: /register Disallow: /review Disallow: /orderconfirmation Disallow: /*? User-agent: msnbot Crawl-delay: 4 User-agent: Slurp Crawl-delay: 4 User-Agent: Googlebot Crawl-delay: 3