Moving from Playwright to Crawlee/Playwright for Scraping Apify #crawlee-js

Moving from Playwright to Crawlee/Playwright for S...

! Spooky

01/07/2025, 6:18 PM

Are there actually any ressources on building a scraper with crawlee except the one in the docs? Where do I set all the browser context for example?

Copy code

javascript
const launchPlaywright = async () => {
  const browser = await playwright["chromium"].launch({
    headless: true,
    args: ["--disable-blink-features=AutomationControlled"],
  });

  const context = await browser.newContext({
    viewport: { width: 1280, height: 720 },
    userAgent:
      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3",
    geolocation: { longitude: 7.8421, latitude: 47.9978 },
    permissions: ["geolocation"],
    locale: "en-US",
    storageState: "playwright/auth/user.json",
  });
  return await context.newPage();
};

Hall

01/07/2025, 6:18 PM

Someone will reply to you shortly. In the meantime, this might help:

azzouzana

01/08/2025, 12:41 AM

In the launch context, here's an example https://docs.apify.com/sdk/js/docs/examples/playwright-crawler

azzouzana

01/08/2025, 1:17 AM

Or within the pre navigation hook Something like: const crawler = new PlaywrightCrawler({ preNavigationHooks: [ async ({ page, request, browserContext }) => { // Set a specific user agent for the browser context await browserContext.addCookies([ { name: 'session', value: '12345', domain: 'example.com' }, ]); // Emulate a specific device (e.g., mobile) await page.setUserAgent( 'Mozilla/5.0 (iPhone; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Mobile/15E148 Safari/604.1' ); }, ], requestHandler: async ({ page, request }) => { console.log(

Visiting ${request.url}

); const content = await page.content(); console.log(

Content length: ${content.length}

); }, }); await crawler.run(['https://example.com']);

Previous Next