Rate limited too soon
Author: mxmzbCreated Apr 26, 2024Updated Aug 7, 2024
I'm running the script locally like this:
yarn index example.com
❯ yarn index example.com
Processing site: sc-domain:example.com
Found 1189 URLs in 2 sitemap
Batch 1 of 24 complete
Batch 2 of 24 complete
Batch 3 of 24 complete
Batch 4 of 24 complete
Batch 5 of 24 complete
Batch 6 of 24 complete
Batch 7 of 24 complete
Batch 8 of 24 complete
Batch 9 of 24 complete
Batch 10 of 24 complete
Batch 11 of 24 complete
Batch 12 of 24 complete
Batch 13 of 24 complete
Batch 14 of 24 complete
Batch 15 of 24 complete
Batch 16 of 24 complete
Batch 17 of 24 complete
Batch 18 of 24 complete
Batch 19 of 24 complete
Batch 20 of 24 complete
Batch 21 of 24 complete
Batch 22 of 24 complete
Batch 23 of 24 complete
Batch 24 of 24 complete
Done, here's the status of all 1189 pages:
• ✅ Submitted and indexed: 410 pages
• Crawled - currently not indexed: 151 pages
• Discovered - currently not indexed: 2 pages
• Page with redirect: 2 pages
• RateLimited: 506 pages
• ❌ Server error (5xx): 9 pages
• ❌ Alternate page with proper canonical tag: 1 pages
• ❌ Duplicate, Google chose different canonical than user: 108 pages
✨ Found 659 pages that can be indexed.
[... list of urls]
Processing url: https://example.com/foo/bar
Indexing already requested previously. It may take a few days for Google to process it.
Processing url: https://example.com/foo/bar1
Rate limit exceeded, try again later.The rate limit exceeds after only around 100-120 urls, and if I rerun it starts from start and again aborts on rate limit aroudn 100-120 urls, so I'm not able to request index for all the URLs that come later.
What am I doing wrong?
Source: goenning/google-indexing-script