Trying to Understand Modern Rate Limiting Before U...
# crawlee-js
c
Been running into aggressive rate limiting while testing some crawlers and automation flows, and I’m trying to understand what modern systems are actually detecting before I jump straight to proxies as the solution. What’s interesting is that sometimes I’m getting blocked even at very low request rates, which makes me think the issue is more about request quality and session behavior than raw traffic volume. I’ve already tried things like: * slowing request frequency * rotating headers/user agents * adding delays
a
message has been deleted
m
Hello! It's exactly what you said: traffic volume is not the only issue. Some servers check your IP address, headers, patterns, and behavior. If your IP is in a blacklist, e.g., you're using well known datacenter proxies, or your request doesn't have the correct headers and cookies, some website will almost certainly block your request. Using better proxies may or may not solve the situation, depending on the reason of blocking.
2 Views