Someone will reply to you shortly. In the meantime, this might help:

Hi, I'm using a cheerio crawler and things are generally working well. I occasionally get a Cloudflare 429 page, though, and it manifests itself as an error on

waitForSelector

because I'm getting the Cloudflare response. Should Crawlee be catching these responses and waiting/slowing without intervention? I've had to catch this issue and then pause the autoscale pool (for 10 sec) manually. Should I be tuning other nobs too/instead? I don't have maxRequestsPerMinute configured yet because I'm not sure how to find/tune this setting.

hey, check out these article for the crawler configuration: https://crawlee.dev/js/docs/guides/scaling-crawlers this might also help you with Cloudflare: https://docs.apify.com/academy/anti-scraping and https://docs.apify.com/academy/anti-scraping/mitigation/cloudflare-challenge.md