Every month, Hunter robot crawls billions of web pages. It works in a way similar to Google but with a focus on business data. We categorize the information in our tools to allow hundreds of thousands of professionals to connect with other businesses.
The robot only analyzes public web pages and navigates at a slow pace. It never visits more than one page every two seconds for a website to make sure other visitors aren’t slowed down by the crawling.
Yes. The robot strictly respect robots.txt, both disallow and allow rules (we do not, however, support nonstandard extensions unless stated otherwise). If for whatever reason you wish to disable it on your website, add the two following lines into to your robots.txt file:
If you think that Hunter robot is someway misbehaving on your website or if you have any questions about it, please don’t hesitate to contact us at email@example.com.