Crawler

IndexMonBot

Our fetcher visits a URL only when someone runs a check or a monitor on it. Per check it reads robots.txt, the page, and the sitemap.