I recently tried to crawl a client's competitor's website for a 'blink analysis'. Some SEO spider tools (including one open source tool as well as the free version of Screaming Frog) I tried were blocked from crawling the site in question with '403' errors recorded in the tools when I used them. SEO Macroscope wasn't (also got some good info relevant to my analysis at a glance from a crawl with siteliner .com - both are free tools) .