I immediately get captcha on every URL. Accessing it in a normal GUI browser typing site homepage URL: captcha. Searching this site in google, clicking on the link in google results: browser shows site address and...: captcha. (by the way, they changed it, few months ago this site was not that restrictive) Well... what is our best solution for sites always showing cloudflare captchas ? https://cdn.discordapp.com/attachments/1296401925301862400/1296401925608312842/cloudflare.png?ex=671227e1&is=6710d661&hm=ce0507489e5bae83ad9f46ea275096b8a48148432deb0802c542fa86f3984240&

message has been deleted

Depends on the website, they may require it even for regular users or they use captchas only for suspicious IPs, fingerprints and so on.

Truly often they always check, even for regular users. However, you can tweak your scraper's configuration to try bypassing it. Generally, using Playwright + firefox with a residential proxy tends to work best, although it’s also the most expensive option. btw, Here are some articles that might help: https://help.apify.com/en/articles/7029488-how-to-bypass-cloudflare https://docs.apify.com/academy/anti-scraping/mitigation/cloudflare-challenge.md