I have created a very simple actor that uses the

gotScraping

library to retrieve the HTML from a set of YouTube video pages. The page URLs are passed as an input array.

typescript
await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();

When the actor runs, it correctly retrieves the first page in the sequence, but crashes when attempting to retrieve a different proxy for the second page (see attached screenshot) What puzzles me is that the error does not get caught by the

catch

block in my own code; it seems to crash at a lower level. I am also not clear why I am seeing the warning message

WARN  ProxyConfiguration: Apify Proxy access check timed out.

and how to correct this issue. None of the docs seem to mention it. Lastly, can you please clarify: if I want a different proxy for every request, should my call to

Actor.createProxyConfiguration

be inside the loop, or prior to the loop? Hoping somebody can advise. Many thanks, Tim https://cdn.discordapp.com/attachments/1370672415591501825/1370672415792959569/image.png?ex=682059a9&is=681f0829&hm=bd4ff859ecdcf4b2820d7b859c287a3bc461335fcd403ebb691d0fc397a8558a&

Someone will reply to you shortly. In the meantime, this might help:

message has been deleted

Hey 👋 You should move the proxy declaration outside of the for loop. It's by nature a rotating proxy I believe (you can ping any IP service like https://www.ipify.org/ and see its result to confirm). Also, I'm wondering why do you want to use raw gotScraping instead of the available crawlers (which should lift most of the work?

Thanks for your response (again 🙂). I checked, using ipify, and you're right that the default Apify proxy configuration uses rotating IP addresses. I also switched to using CheerioCrawler and it seems to be stable now (as well as offering lots of additional capabilities). TBH, I had hesitated about using Crawlee, because I assumed that was a heavyweight library, intended for more complex crawling operations (whereas I currently only need to scrape a pre-defined set of URLs with static HTML). But now that I have properly read the Crawlee docs, I can see that it can be used for simple or complex tasks, with no unwanted overhead. It's exactly what I need. Thanks for your guidance.

@tim_coulter_95542 just advanced to level 1! Thanks for your contributions! 🎉