About our crawler

This crawler belongs to an internal SEO auditing tool. It scans websites that its operator manages or audits in order to find technical SEO issues such as broken links, missing metadata, and redirect problems.

It respects robots.txt, crawls politely with bounded concurrency and backoff, and never modifies anything on the sites it visits.

How to block it

Add the following to your robots.txt to opt out entirely:

User-agent: CrawlSpecBot
Disallow: /

If you believe this crawler is misbehaving on your site, block it via robots.txt as above; it checks robots.txt on every run and will stop immediately.

Remove my site

Tell us and we stop. This opens a support request that a person reads.

Stop free scans of my site

Visitors can ask us for a free scan of any site. If you would rather nobody could do that with yours, prove you control the domain and we will refuse every request for it. This takes a minute and needs no account. You do not wait for us.