Crawling pages not working with specific URL configuration?
Author: nynewcoCreated Apr 2, 2025Updated May 31, 2026
I tried this configuration but it didn't work.
url: "https://midlibrary.io/", match: "https://midlibrary.io/art-styles?filters=photographers&page=**#topshelf", maxPagesToCrawl: 50, outputFileName: "output.json", maxTokens: 2000000,
What am I doing wrong?
Source: BuilderIO/gpt-crawler