I see that in crawlee-python we have the option to use Redis for this, anybody figured a way to use Redis on crawlee-js (TS) ?

message has been deleted

Hey, Shortly => No, there's no built in Redis backend for RequestQueue in Crawlee JS. Additinionally even for crawlee-python it's experimental feature for now. But what I see from docs is that Crawlee JS has a pluggable StorageClient (https://crawlee.dev/js/api/core/interface/StorageClient) interface that you could implement yourself for Redis. so it cal look something like this:

import { Configuration } from '@crawlee/core';

// Option A: globally
Configuration.useStorageClient(myRedisStorageClient);

// Option B: per-queue
await RequestQueue.open(queueId, { storageClient: myRedisStorageClient });

Implemented. Leveraging Dragonfly now. It had its ups and downs but now I use it steady with multiple crawlers per day