https://apify.com/ logo
Join Discord
Powered by
Request works in Postman but doesn't works in craw...
# crawlee-js
l
Hello I'm trying to handle ajax call via got-scraping. I prepare call in postman, where it works fine. But if I want to try it in Actor a got 403 every time. Even if I try i via Puppeteer or Playwrite and click on the button with request I got response with geo.captcha-delivery.com/captcha url to solve it. Please can anybody give me any advice how to handle this issue?
h
Someone will reply to you shortly. In the meantime, this might help: -# This post was marked as solved by Matous. [View answer]().
l
Hi, did you try to fully recreate all the headers that Postman creates automatically for you? It might be that you need to add Content-Type or User-Agent for example. Another thing that could be causing the issue are the proxies, with Postman you are using your home IP, but if you create the actor you might be using different proxies, depending where you run it and if you do use the apify proxy configuration.
l
But I used RESIDENTIAL proxies from particular country.
l
Perhaps you can share some specific code and URLs. Right now it's just guessing what might be going wrong.
l
yes of course I have this website, for example this detailed URL: https://www.idealista.com/inmueble/98545124/ On this detail there is a link to view the mobile phone downloaded using the ajax call "Ver teléfono". https://www.idealista.com/inmueble/98545124/ I am able to scrape this detail but when try to call phone url .. every time got 403 ... even if I start browser, click on that link and catch response... I got 403 status code
I tried to use gotScraping
Copy code
const gotResponse = await gotScraping({
        url: `https://www.idealista.com/${pageCountry}/ajax/ads/${output.id}/contact-phones`,
        headers:{
            accept:'application/json, text/javascript, */*; q=0.01',
            'accept-language':'en-US,en;q=0.9',
            referer: output.url,
            'user-agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36',
            'x-requested-with':'XMLHttpRequest',
            Connection: 'keep-alive',
            // Cookie: cookieHeader,
        },
        proxyUrl: await proxyConfiguration.newUrl(session.id),
    });

    console.log(gotResponse.statusCode);
but the result is same
l
It may be that you are missing some cookies. On the request generated for the phone number I can see there is. SESSION and a userUUID cookie so maybe you have to get those. If you are using Playwright or Puppeteer you can also use 
evaluate
to perform the request in the browser console https://playwright.dev/docs/evaluating https://pptr.dev/api/puppeteer.page.evaluate
PreviousNext
Powered by