Hi, I'm seeing an error after upgrading to Apify Python SDK v4.0.0 and Crawlee v1.9.0. The actor completes successfully, but an error is logged during shutdown after Actor.exit(). To make sure it wasn't related to my code, I also tested the official Crawlee Parsel template (https://apify.com/templates/python-crawlee-parsel), and the same error appears there as well. I also tested the same template with Apify Python SDK v3.4.1 and Crawlee v1.9.0, and the error does not occur there. Here's the run URL where the issue can be seen: https://console.apify.com/view/runs/nuSNKe7T8QcaxDfhs

Hi — I'm trying to use the official apify/facebook-groups-scraper to extract posts from a public Facebook group (https://www.facebook.com/groups/285614534784157). No matter what settings I use, the scraper consistently returns only automated "Let's welcome our new members!" posts. I've tried: Sorting by "New posts" (20 posts) — all welcome posts Sorting by "Most relevant" (20 posts) — all welcome posts The group is public and all posts are visible to logged-out users in a regular browser, so it doesn't appear to be an access issue. Is this a known limitation with this group or this type of group? Is there a setting or workaround that would get me actual discussion posts instead? Happy to provide run logs if helpful. Thank you!!!

Hey apify team. I opened issue tab in instagram scraper and you wanted to share my run ID but i had to make another account for it to make the issue tab becouse my account martinhouha5@gmail.com got disabled for no reason and i didnt got any info why that happened. Could you please enable my account so i can share the run ID to you guys. And i contacted the support email bit only AI responded and i dont know if it got to you. https://cdn.discordapp.com/attachments/1511418236586168481/1511418236854337749/image.png?ex=6a20616c&is=6a1f0fec&hm=3e602f6d8e1eaf4e488f7f9328712afef810bffcb8bd8c69dc6e32fc35a7da49& https://cdn.discordapp.com/attachments/1511418236586168481/1511418237181628426/image.png?ex=6a20616c&is=6a1f0fec&hm=38192fb1c883dd3e3be2148913db29932b30365087a3369e1e7247d8dc09f311&

message has been deleted

Hey! I'm Bevan, a Software Engineer open to full-time roles and freelance work—remote or on-site. I specialize in end-to-end development and high-performance systems. If you know of any open opportunities, let’s connect. Drop me a DM! 📩

message has been deleted

Hello! Yes, depending on what you scraping, having multiple setup for the proxy can be the best way to crawl different pages, even if they are from the same website. There is no rule really but ideally, every time you can avoid doing a request through residential it's always worth considering it.

Hey, I’m pretty new to Apify and currently on the free plan. I still have about $4.20 remaining out of $5, but I’m getting a “free user run limit exceeded” error on one of the actors. Not sure why this is happening, could someone help clarify? Lets say i upgrade to the starter plan, Will it still have some kind of max run limit?

How do I set general settings data that will be accessible to the request handler without passing it to the user data of every request. I thought there will be away to add it to the context when creating the crawler but I can't find any parameter that does this.

There must be some kind of bug with the Request Queue with Crawlee + Apify. Simple ParselCrawler, 8 long running requests (30+ seconds waiting per request) Yet the request queue reads are at 5000+.

I’m looking for a full stack developer who can start working on our project from tomorrow. The frontend (Part 1) was done by a previous developer but not very well, so we had to pause. I need someone who’s up to date with recent tools and best practices and is skilled in backend development (preferably Node.js) and database management. This project is for a game project, and we need to build a solid paywall system and admin page. I’m looking for someone who can take the lead, handle the full development until deployment, and also prepare proper documentation. Let’s talk and see how we can work together.

I just read this amazing blog post about creating WARC files with crawlee: https://crawlee.dev/python/docs/guides/creating-web-archive. I was considering using the wayback proxy to constitute my archive collection. Still I am wondering, if I need to set up some datacenter proxy to bypass some rate limit, is it possible to use it along with the wayback setup? Like, can I use the wayback proxy for archiving on top of other proxies? Does crawlee allow such set up? Thanks 😘

I was wondering what are the best practices to use crawlee with claude code. Looking forward to your inputs folks 🙂

Hello, I find this behavior odd that when

Configuration(purge_on_start=True)

set, then the

storage/request_queues/default/__metadata__.json

file is not purged in method

crawlee.storage_clients._file_system._request_queue_client.FileSystemRequestQueueClient.purge

along with the request files. What is the reason behind this? Which use-case this covers? If you run the repro code (it crawls 2 URLs) multiple times right after another and check the log/metadata json. I see the following:

# first run
[BeautifulSoupCrawler] INFO  Crawled 0/2 pages, 0 failed requests, desired concurrency 1.

# second run
[BeautifulSoupCrawler] INFO  Crawled 0/4 pages, 0 failed requests, desired concurrency 1.

The 2nd run 0/4 is misleading because the only two requests were scheduled and the previous one has been purged. After the 2nd run the manifest.json content contains handled_request_count: 4 and total_request_count: 4 which is printed to the logs. My expectation would be 2 for both values

{
  "id": "7xvLZTJolTixoRk1x",
  "name": null,
  "accessed_at": "2026-01-02 15:10:06.721976+00:00",
  "created_at": "2026-01-02 15:09:48.267199+00:00",
  "modified_at": "2026-01-02 15:10:06.719162+00:00",
  "had_multiple_clients": false,
  "handled_request_count": 4,
  "pending_request_count": 0,
  "total_request_count": 4
}

I've attached repro code as a file due to post length limitation Thank you, CL https://cdn.discordapp.com/attachments/1456667356334919790/1456667356859338823/crawlee_purge_repro.py?ex=695e78b1&is=695d2731&hm=3b75b6e2cb1b412a05d37c860eb06876812be4d1f3dee24f538ccfba406148e1&

If I run a single PlaywrightCrawler that crawls multiple different websites, are concurrency and rate limits shared across all domains, or does Crawlee track them separately per domain? I’m asking because I’m building a general crawler that scrapes schema.org markup from lots of unrelated sites, and the crawl logic is almost identical, so I’m using one crawler instance instead of one per domain. I want to understand whether domains compete for the same concurrency slots or not? Basically, what I'm asking is if I set

minConcurrency

to 3, is it 3 concurrent requests for website_a.com and 3 concurrent requests website_b.com, or is it 3 concurrent requests spread out between website_a.com and website_b.com?

It's always the default HeadlessChrome value. Very easy to detect a bot despite Crawlee's fingerprinting

[crawlee.crawlers._playwright._playwright_crawler] INFO  Crawling URL: https://www.httpbin.org/headers
[crawlee.crawlers._playwright._playwright_crawler] INFO  Response headers: {
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", 
    "Accept-Encoding": "gzip, deflate, br, zstd", 
    "Accept-Language": "en-US", 
    "Host": "www.httpbin.org", 
    "Priority": "u=0, i", 
    "Sec-Ch-Ua": "\"HeadlessChrome\";v=\"143\", \"Chromium\";v=\"143\", \"Not A(Brand\";v=\"24\"", 
    "Sec-Ch-Ua-Mobile": "?0", 
    "Sec-Ch-Ua-Platform": "\"macOS\"", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "none", 
    "Sec-Fetch-User": "?1", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.6 Safari/605.1.15", 
    "X-Amzn-Trace-Id": "Root=1-69495bc0-1b888c117c04745e46ba392f"
  }
}

message has been deleted

without a LinkedIn session cookie (li_at)? I’m on the Free plan and the actor has no field for a cookie. Before I upgrade, I need a yes/no: does this actor support authenticated scraping at all, and if so, where is the cookie input located? If it cannot scrape employee lists without authentication, then upgrading won’t fix anything, because it will still be blocked at the UI level. The results I'm getting are mixed at best and I'm told it's because it is not an authenticated search.

message has been deleted

Hi @H_W For requests like this, there are other threads e.g. #1022804760484659210

My actor tried to call the ZIP file via URL but error. I want to check if the runner can perform an outgoing URL? URL: https://getsamplefiles.com/download/zip/sample-1.zip

I have been struggling with tests for a while, and finally reduced it to a simple test for which I don't understand the behaviour. Is this expected (and I am missing sth) or is this expected? This test fails async def test_failing(): storage_client = MemoryStorageClient() request_queue_client = await storage_client.create_rq_client() req = Request.from_url("https://crawlee.dev") await request_queue_client.add_batch_of_requests([req]) crawler = BasicCrawler( concurrency_settings=ConcurrencySettings(desired_concurrency=1, max_concurrency=1), max_crawl_depth=2, storage_client=storage_client, ) @crawler.router.default_handler async def handler(context: BasicCrawlingContext) -> None: pass stats = await crawler.run() assert stats.requests_finished > 0 but this one passes async def test_success(): storage_client = MemoryStorageClient() req = Request.from_url("https://crawlee.dev") crawler = BasicCrawler( concurrency_settings=ConcurrencySettings(desired_concurrency=1, max_concurrency=1), max_crawl_depth=2, storage_client=storage_client, ) @crawler.router.default_handler async def handler(context: BasicCrawlingContext) -> None: pass await crawler.add_requests([req]) stats = await crawler.run() assert stats.requests_finished > 0 the only difference is that in the first I add requests through a request client or through the crawler. If I add it through this rq = await RequestQueue.open() await rq.add_request(req) it also fails. Thanks in advance

message has been deleted

I followed the official documentation's template code to implement an Apify Actor, built and deployed it on the Apify platform. When I run the Actor and click Abort (graceful abort), the ABORTING event handler handler_foo does not log anything — Actor.log.info(...) inside it never appears. I have no way to confirm whether the handler is actually being triggered, and I need to perform custom actions when the user aborts the run. Questions: How can I guarantee that the ABORTING event handler is called? How can I ensure that logs and operations (like Actor.set_value) inside it execute and appear in the logs? -------------------------------------------------------------------------------------------- from apify import Actor from apify_shared.consts import ActorEventTypes import asyncio def handler_foo(arg: dict): Actor.log.info(f'handler_foo: arg = {arg}') def handler_boo(arg: dict): pass async def main(): async with Actor: # Add event handler Actor.on(ActorEventTypes.ABORTING, handler_foo) await asyncio.sleep(5*1000) # Remove all handlers for a specific event Actor.off('systemInfo') # Remove a specific event handler Actor.off('systemInfo', handler_boo)

https://console.apify.com/view/runs/e65qXc7GXZBNFA8CD My actor suddenly threw an error, and I didn't make any changes to the code. What's more, it works fine when I run it myself, but there's an issue when my users run it.

I am using multiple instances of an AdaptativePlaywright crawler in different servers, using the SQL queue. After running for some time, it looks like the crawler stops working and I only regularly see this logs: │ [crawlee._autoscaling.autoscaled_pool] WARN Task timed out after not set seconds │ │ [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 9; desired_concurrency = 10; cpu = 0.0; mem = 0.0; event_loop = 0.023; client_info = 0.0 │ │ [crawlee._autoscaling.autoscaled_pool] WARN Task timed out after not set seconds │ │ [crawlee.crawlers._basic._basic_crawler] INFO Crawled 0/1157424 pages, 0 failed requests, desired concurrency 10. Restarting the pod where the crawler runs doesn't fix the issue

Lets say i have one session that makes 15 requests. I do not like that it finishes crawl in 25 seconds. How to stretch crawl duration / slow down speed of crawl for small ones? max_requests_per_minute is useless in this situation right? What options i have?

message has been deleted

I am deploying crawlee in my kubernetes cluster and first I tried not setting a maximum (nor a minimum) of concurrent tasks, but crawlee kept on taking more and more memory until the pod was killed by the cluster (took too much memory). Now I set a conservative maximum but I see resources are very underutilised in some moments depending on the page I am crawling. Is there a way I am not seeing on how to do this correctly? Or is it possible that there is a bug when determining a max amount of concurrent tasks where crawlee spawns too many of them?

I am not able to reproduce it in a simple example (it may be a transient error), but I have gotten this error regularly and it kills the crawler completely.

Traceback:
  File "crawlee/crawlers/_basic/_basic_crawler.py", line 1366, in __run_task_function
    if not (await self._is_allowed_based_on_robots_txt_file(request.url)):

  File "crawlee/crawlers/_basic/_basic_crawler.py", line 1566, in _is_allowed_based_on_robots_txt_file
    robots_txt_file = await self._get_robots_txt_file_for_url(url)

  File "crawlee/crawlers/_basic/_basic_crawler.py", line 1589, in _get_robots_txt_file_for_url
    robots_txt_file = await self._find_txt_file_for_url(url)

  File "crawlee/crawlers/_basic/_basic_crawler.py", line 1599, in _find_txt_file_for_url
    return await RobotsTxtFile.find(url, self._http_client)

  File "crawlee/_utils/robots.py", line 48, in find
    return await cls.load(str(robots_url), http_client, proxy_info)

  File "crawlee/_utils/robots.py", line 59, in load
    response = await http_client.send_request(url, proxy_info=proxy_info)

  File "crawlee/http_clients/_impit.py", line 167, in send_request
    response = await client.request(

impit.ConnectError: Failed to connect to the server.
Reason: hyper_util::client::legacy::Error(
    Connect,
    ConnectError(
        "dns error",
        Custom {
            kind: Uncategorized,
            error: "failed to lookup address information: Name or service not known",
        },
    ),
)
exited with code 1

This is my crawler:

crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
  playwright_crawler_specific_kwargs={
      "browser_type": "firefox",
      "headless": True,
  },
  max_session_rotations=10,
  retry_on_blocked=True,
  max_request_retries=5,
  keep_alive=True,
  respect_robots_txt_file=True,
)

I am on version 1.0.4 and I was crawling crawlee.dev (though it doesn't fail in a specific page)