Here are some suggestions to troubleshoot and improve the situation:
1. **Proxy Rotation and Quality**:
- Ensure you’re using high-quality Residential proxies. Sometimes, even Residential proxies can be flagged if they are overused or come from known data centers.
- Rotate proxies more frequently to avoid triggering detection mechanisms.
- Experiment with a mix of IPs from nearby countries instead of exclusively KR (e.g., JP, CN), which might bypass local restrictions.
2. **Request Headers and Payload**:
- Double-check your headers and payloads. Mimic real browser requests by including accurate user-agent strings, cookies, and other headers.
- Tools like [Apify Fingerprint Suite](
https://github.com/apify/fingerprint-suite) can help generate realistic browser fingerprints.
3. **Request Throttling and Timing**:
- Reduce the frequency of requests to avoid looking like a bot. Implement random delays between requests.
- Try spreading requests over time to reduce suspicion.
4. **Alternative Scraping Techniques**:
- Use headless browsers like Puppeteer or Playwright for challenging sites. Even though you’re not currently using them, these tools can mimic user behavior effectively and bypass detection in many cases.
- Implement CAPTCHA-solving services if CAPTCHA blocks are common.
5. **Evaluate Target Sites**:
- Some platforms (like the ones you mentioned) may have advanced bot-detection mechanisms like Akamai or Cloudflare. Scraping these may require tailored solutions or specialized proxies.
Also this section in docs might be useful:
https://docs.apify.com/academy/anti-scraping
In general, if you're having trouble bypassing protection with HTTP requests, consider using Playwright with a Firefox browser configuration. Pairing it with Residential proxies can make your scraper more difficult to detect, though it may increase resource consumption and costs.