It's always the default HeadlessChrome value. Very easy to detect a bot despite Crawlee's fingerprinting

[crawlee.crawlers._playwright._playwright_crawler] INFO  Crawling URL: https://www.httpbin.org/headers
[crawlee.crawlers._playwright._playwright_crawler] INFO  Response headers: {
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", 
    "Accept-Encoding": "gzip, deflate, br, zstd", 
    "Accept-Language": "en-US", 
    "Host": "www.httpbin.org", 
    "Priority": "u=0, i", 
    "Sec-Ch-Ua": "\"HeadlessChrome\";v=\"143\", \"Chromium\";v=\"143\", \"Not A(Brand\";v=\"24\"", 
    "Sec-Ch-Ua-Mobile": "?0", 
    "Sec-Ch-Ua-Platform": "\"macOS\"", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "none", 
    "Sec-Fetch-User": "?1", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.6 Safari/605.1.15", 
    "X-Amzn-Trace-Id": "Root=1-69495bc0-1b888c117c04745e46ba392f"
  }
}

message has been deleted

@Big_Smoke_420 just advanced to level 1! Thanks for your contributions! 🎉

Post was "too long" so I'm pasting my code as a separate message:

python
import asyncio

from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext


async def main() -> None:
    # The browserforge fingerprints and headers are used by default.
    crawler = PlaywrightCrawler(
        browser_launch_options={
            "args": ["--no-sandbox", "--disable-setuid-sandbox"],
        }
    )

    @crawler.router.default_handler
    async def handler(context: PlaywrightCrawlingContext) -> None:
        url = context.request.url
        context.log.info(f"Crawling URL: {url}")

        # Decode and log the response body, which contains the headers we sent.
        headers = (await context.response.body()).decode()
        context.log.info(f"Response headers: {headers}")

        # Extract and log the User-Agent and UA data used in the browser context.
        ua = await context.page.evaluate("() => window.navigator.userAgent")
        ua_data = await context.page.evaluate("() => window.navigator.userAgentData")
        context.log.info(f"Navigator user-agent: {ua}")
        context.log.info(f"Navigator user-agent data: {ua_data}")

    # The endpoint httpbin.org/headers returns the request headers in the response body.
    await crawler.run(["https://www.httpbin.org/headers"])


if __name__ == "__main__":
    asyncio.run(main())

Hey @Big_Smoke_420 Thank you for your interest in Crawlee. Hmm, the problem with ‘Sec-Ch-Ua’ does not reproduce in my environment with your code.

bash
[crawlee.crawlers._playwright._playwright_crawler] INFO  Response headers: {
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", 
    "Accept-Encoding": "gzip, deflate, br, zstd", 
    "Accept-Language": "en-US", 
    "Host": "www.httpbin.org", 
    "Priority": "u=0, i", 
    "Sec-Ch-Ua": "\"Google Chrome\";v=\"141\", \"Not?A_Brand\";v=\"8\", \"Chromium\";v=\"141\"", 
    "Sec-Ch-Ua-Mobile": "?0", 
    "Sec-Ch-Ua-Platform": "\"Windows\"", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "none", 
    "Sec-Fetch-User": "?1", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-694975dc-29af1cb379e7fbaa3d1a2d75"
  }
}

But I noticed that

brands

in

ua_data

is always for

HeadlessChrome

, it seems that

browserforge

does not patch this data.