# CAPSite Robots.txt # We usually don't want crawlers indexing our content (other than our crawler). # So ban popular crawlers we know about. User-agent: googlebot Disallow: / User-agent: slurp Disallow: / User-agent: msnbot Disallow: / User-agent: teoma Disallow: /