About our crawler

This crawler belongs to an internal SEO auditing tool. It scans websites that its operator manages or audits in order to find technical SEO issues such as broken links, missing metadata, and redirect problems.

It respects robots.txt, crawls politely with bounded concurrency and backoff, and never modifies anything on the sites it visits.

How to block it

Add the following to your robots.txt to opt out entirely:

User-agent: CrawlSpecBot
Disallow: /

If you believe this crawler is misbehaving on your site, block it via robots.txt as above; it checks robots.txt on every run and will stop immediately.

Remove my site

Tell us and we stop. This opens a support request that a person reads; we reply within two business days.

Stop free scans of my site, without waiting for us

Visitors can ask us for a free audit of any site. If you would rather nobody could do that with yours, prove you control the domain and we will refuse every request for it. This takes a minute and needs no account.