CrawlSpecBot/1.0

A free look at what your templates get wrong

We crawl 25 pages, group what is broken by template and show three fixes in full, each with a prompt for your coding agent.

Pages crawled
25

One polite pass, then it stops

Fixes in full
3

The rest are counted on your report

Accounts and cards
0

Just an address for the report

Scan a site

The report opens in your browser when the crawl finishes, and we email you the link.

25 pages, depth 2, robots.txt honoured. Most scans finish in under a minute, and the report link stays live for 14 days.

What we do not do

  • The free scan only reads your pages. It changes nothing on your site.
  • We always obey robots.txt. No customer can switch that off.
  • Your report goes only to the address you typed. We never sell or share it.
  • We run no AI over your site. The free scan is crawl output and arithmetic.

The crawler's own page shows how to block it with one robots.txt line. About CrawlSpecBot

What a free audit does not include

A real audit of a small sample, not a trial with the answers removed.

2,000 pages per crawl
A sample finds template problems, but not the sections it never reached.
Every fix card, not three
The rest open when the audit moves into a workspace.
Re-crawls that close cards
Cards close when the next crawl confirms the fix. A free scan runs once.
Search Console and content briefs
Clicks and positions per page, keyword maps, and pages written from your own data.

Every one of those ships on the paid plan. See what a plan costs.

Fix prompts for Claude Code, Cursor and Codex

Your report carries a prompt per fix and one for the whole audit, plus spec.md and spec.json downloads. None contains a credential, so they are safe to paste into a shared chat.

What the report looks like

An example report for a fictional store.

Sample report - a fictional store, not a real site

SEO audit of example-shop.test

Pages scanned
25
Issues found
312
Scanned
2026-09-08
Expires in
14d

14 days from the scan

Issues by severity

  • Critical2
  • High24
  • Medium61
  • Low187
  • Info38

The crawl was capped, so these counts describe the pages we reached, not the whole site.

Top 3 fixes

1. Product pages ship without a meta description

Fix the template
P 8.6reach: 19 pagesMISSING_META_DESCRIPTION

Nineteen of the twenty-five pages we read are product pages built from one template, and that template emits no meta description. Google writes its own snippet from whatever text it finds, which on these pages is the shipping notice.

Evidence and affected URLs

How to fix

  • Next.js Return a description from the route segment generateMetadata, composed from the same product record the page renders. A missing description is a data gap, not a copy task.

Evidence

All nineteen pages return an empty <meta name="description">; the tag is present, so the template renders it unconditionally from a field that is never set.

Examples

  • /p/brass-kettle-1l
  • /p/enamel-mug-navy
  • /p/cast-iron-skillet-26cm

Affected URLs

/p/*

  • https://example-shop.test/p/brass-kettle-1l
  • https://example-shop.test/p/enamel-mug-navy
  • https://example-shop.test/p/cast-iron-skillet-26cm
  • https://example-shop.test/p/linen-apron-sand
  • https://example-shop.test/p/copper-measuring-set

2. Paginated category pages all share one title

Fix the template
P 7.4reach: 6 pagesDUPLICATE_TITLE

Six category pages carry the identical title "Kitchen - Example Shop", because the template ignores the page number. Google picks one of them to show and treats the rest as near-duplicates of it.

Evidence and affected URLs

How to fix

  • Next.js Read the page parameter in generateMetadata and suffix the title with " - page N" when it is greater than one. The same value should drive the self-canonical.

Evidence

Six URLs, one title string, one meta description. The only difference between the documents is the product grid.

Examples

  • /c/kitchen
  • /c/kitchen?page=2
  • /c/kitchen?page=3

Affected URLs

/c/kitchen*

  • https://example-shop.test/c/kitchen
  • https://example-shop.test/c/kitchen?page=2
  • https://example-shop.test/c/kitchen?page=3
  • https://example-shop.test/c/kitchen?page=4

3. Product photography carries no alt text

Fix the content
P 6.9reach: 148 resourcesIMAGE_MISSING_ALT

One hundred and forty-eight images across the pages we read have an empty alt attribute. That costs you image search, and it is the single most common accessibility complaint a shop gets. The gallery component sets alt="" on every slide.

Evidence and affected URLs

How to fix

  • Next.js The alt prop on next/image is required but accepts an empty string, which is why this passed review. Pass the product title from the same record that supplies the src.

Evidence

Of 161 images seen, 148 have alt="". The 13 that do not are the site logo and the payment icons in the footer.

Affected URLs

/p/*

  • https://example-shop.test/p/brass-kettle-1l
  • https://example-shop.test/p/enamel-mug-navy

5 more fixes this scan found

  • 4. Filtered category URLs canonicalise to themselves9 pages
  • 5. Journal posts have no internal links pointing at them4 pages
  • 6. The sitemap lists URLs that redirect7 pages
  • 7. No Product structured data on product pages19 pages
  • 8. Category pages carry under 40 words of their own copy5 pages

Ranked by the same score as the cards above. They open when this audit moves into a workspace.

Hand the whole audit to your agent

One prompt with the site facts, the 3 fixes above, the order to do them in and how to verify. It carries no credential, so it is safe to paste anywhere.

Point your agent at this audit

The prompts above are self-contained: an agent can do the work with no account and no credential. The MCP server adds the other half of the loop, so the agent re-reads the audit and re-checks a page itself after it edits one. Its 11 workspace tools wrap the API documented on the API page.

{
  "mcpServers": {
    "crawlspec": {
      "command": "npx",
      "args": [
        "-y",
        "@crawlspec/mcp"
      ],
      "env": {
        "CRAWLSPEC_API_KEY": "<your CrawlSpec token>",
        "CRAWLSPEC_API_URL": "https://crawlspec.dev"
      }
    }
  }
}

The token goes in that placeholder. Tokens come by email, never on this page, and a workspace mints its own from project settings once this audit is claimed.

Not in a free scan

Fixes still locked
5

Ranked by the same score, hidden on the free scan

Pages per crawl on Pro
2,000

25 in a free scan

Re-crawls
0

A free scan runs once and never again

Content briefs
0

Written from the audit plus real search demand

Search Console days
0

Clicks and positions joined onto every page

Agent tools on a workspace
11

Briefs, topics, keywords and the consolidated audit

Frequently asked

Do I have to own the site?
Yes. The checkbox reads: "I own this site or have permission to audit it, and I understand CrawlSpec will crawl up to 25 public pages while respecting robots.txt." We store that sentence with the time and your address. A domain that already belongs to a CrawlSpec workspace is refused.
What does the crawler do to my server?
It fetches at most 25 HTML pages, following links at most two clicks from the address you enter, plus the resources they reference, a few requests at a time, then stops. Every request identifies itself as CrawlSpecBot.
How long do you keep the report and my address?
The report stays readable for 14 days, then is deleted. We keep your address while a report is live; after that we hold only a one-way hash of it. An unclaimed scan project is deleted 90 days after its last scan.
How do I stop this, or delete what you have?
Every report email links to a page with two buttons: stop these emails, and delete my data. Deleting removes your scans, your stored request and the project. Not your site? The crawler page has a removal request.

Ready for the whole site? Founding 50 is 50% off the yearly price, for the first 50 workspaces. See what Founding 50 includes.