Extension: uncapped, sitemap-complete crawl of a Shopify store via Crawler Access signature from a project-local .shopify-env

Author: NordaluxCreated Sep 10, 2026Updated Sep 10, 2026

seo-audit follows links up to 500 pages and Shopify rate-limits storefront crawlers, so a 30k-URL Shopify store is audited from a 500-page sample and the report cannot say how many pages the store really has.

Shopify lets a merchant mint a Crawler Access signature in the admin (Online Store > Preferences > Crawler access): three HTTP headers (RFC 9421 message signature, tag web-bot-auth), bound to one host, valid ~90 days. A crawler replaying them is not rate limited. Every Shopify store also publishes a complete nested sitemap.xml, so URL discovery does not need link following or a cap.

Proposal, as an optional extension extensions/shopify (no MCP, no API keys):

  • The signature and per-shop crawl settings live in a project-local .shopify-env (one Domain= block per shop, keys before the first block are shared settings). Found in the working directory or up to five parents, $SHOPIFY_ENV overrides. Git-ignored; the scripts warn when it sits unignored in a work tree.
  • scripts/shopify_env.py: parses the file, validates the signature input (web-bot-auth, @authority covered, expiry), and answers precheck <url> with signed or fallback as JSON, never raising.
  • scripts/shopify_crawl.py: walks the sitemap through url_safety's pinned session, attaches the signature only to the issuing host, records redirects without following them, backs off on 429/430/503 (exit 5 when that happens on a signed request, which Shopify documents as "signature invalid"), and writes summary.json (aggregate issue counts), sample.json (N pages per template) and pages.jsonl (raw, never for a model).
  • seo-shopify skill: precheck, crawl, then the unchanged seo-audit pipeline from step 4 on. Without a valid signature for the host, seo-audit runs exactly as today.

Only stores the user administers can be crawled this way; everything else stays capped and unsigned. I have this working and can open a PR if the shape is acceptable, or adjust the placement first.