Extension: uncapped, sitemap-complete crawl of a Shopify store via Crawler Access signature from a project-local .shopify-env
seo-audit follows links up to 500 pages and Shopify rate-limits storefront crawlers, so a 30k-URL Shopify store is audited from a 500-page sample and the report cannot say how many pages the store really has.
Shopify lets a merchant mint a Crawler Access signature in the admin (Online Store > Preferences > Crawler access): three HTTP headers (RFC 9421 message signature, tag web-bot-auth), bound to one host, valid ~90 days. A crawler replaying them is not rate limited. Every Shopify store also publishes a complete nested sitemap.xml, so URL discovery does not need link following or a cap.
Proposal, as an optional extension extensions/shopify (no MCP, no API keys):
- The signature and per-shop crawl settings live in a project-local
.shopify-env(oneDomain=block per shop, keys before the first block are shared settings). Found in the working directory or up to five parents,$SHOPIFY_ENVoverrides. Git-ignored; the scripts warn when it sits unignored in a work tree. scripts/shopify_env.py: parses the file, validates the signature input (web-bot-auth,@authoritycovered, expiry), and answersprecheck <url>withsignedorfallbackas JSON, never raising.scripts/shopify_crawl.py: walks the sitemap throughurl_safety's pinned session, attaches the signature only to the issuing host, records redirects without following them, backs off on 429/430/503 (exit 5 when that happens on a signed request, which Shopify documents as "signature invalid"), and writessummary.json(aggregate issue counts),sample.json(N pages per template) andpages.jsonl(raw, never for a model).seo-shopifyskill: precheck, crawl, then the unchangedseo-auditpipeline from step 4 on. Without a valid signature for the host,seo-auditruns exactly as today.
Only stores the user administers can be crawled this way; everything else stays capped and unsigned. I have this working and can open a PR if the shape is acceptable, or adjust the placement first.
Source: AgriciDaniel/claude-seo