message has been deleted

Hello, I find this behavior odd that when

Configuration(purge_on_start=True)

set, then the

storage/request_queues/default/__metadata__.json

file is not purged in method

crawlee.storage_clients._file_system._request_queue_client.FileSystemRequestQueueClient.purge

along with the request files. What is the reason behind this? Which use-case this covers? If you run the repro code (it crawls 2 URLs) multiple times right after another and check the log/metadata json. I see the following:

# first run
[BeautifulSoupCrawler] INFO  Crawled 0/2 pages, 0 failed requests, desired concurrency 1.

# second run
[BeautifulSoupCrawler] INFO  Crawled 0/4 pages, 0 failed requests, desired concurrency 1.

The 2nd run 0/4 is misleading because the only two requests were scheduled and the previous one has been purged. After the 2nd run the manifest.json content contains handled_request_count: 4 and total_request_count: 4 which is printed to the logs. My expectation would be 2 for both values

{
  "id": "7xvLZTJolTixoRk1x",
  "name": null,
  "accessed_at": "2026-01-02 15:10:06.721976+00:00",
  "created_at": "2026-01-02 15:09:48.267199+00:00",
  "modified_at": "2026-01-02 15:10:06.719162+00:00",
  "had_multiple_clients": false,
  "handled_request_count": 4,
  "pending_request_count": 0,
  "total_request_count": 4
}

I've attached repro code as a file due to post length limitation Thank you, CL https://cdn.discordapp.com/attachments/1456667356334919790/1456667356859338823/crawlee_purge_repro.py?ex=695e78b1&is=695d2731&hm=3b75b6e2cb1b412a05d37c860eb06876812be4d1f3dee24f538ccfba406148e1&

Hey @cdog Yes, the

purge

method does not clear metadata, it only clears requests awaiting processing. To completely delete the queue, use the

drop

method. https://crawlee.dev/python/api/class/RequestQueue

Thank you @Mantisus I'll use that method in the future. But I'm still wondering why purge doesn't clear metadata. Do you have a scenario when that is useful to keep some of the values?

The fact that the storage is not completely deleted can be useful for SQL- and Redis-based storages that support multiple clients. This is because when multiple clients try to create a queue, it causes overhead to avoid race conditions. In some cases, it is also important to keep data on all processed requests for statistical purposes. If purge is performed while the crawler is running, for example when it is running as a web service.

message has been deleted