Can PlaywrightCrawler and CheerioCrawler be used in parallel? For example, can Playwright push a URL (got detail page URL by javascript rendering) to CheerioCrawler request queue for another immediate crawling (get the detailed page json data)? I noticed Request Queue V2, but the official examples seem aimed at speeding up crawling via process forking. I am looking for a more lightweight approach where CheerioCrawler handles the remaining work without the two blocking each other. Do you have any insights on this?

Hello @SirkFun, thank you for reaching out and for the interesting question. Running two crawlers in parallel is possible but there are some constraints. The major one is that you can't on request queue (RQ) for both crawlers because there is no mechanism for the crawler to only handle some of the requests in the RQ. Provided that you have two RQs you can run the PlaywrightCrawler to get the page detail URLs and as you resolve them you can add them to the RQ for the CheerioCrawler to scrape. The last important bit is that you need to make sure that the CheerioCrawler is only allowed to stop once the PlaywrightCrawler has stopped. You can do this by defining

isFinishedFunction

in

autoscaledPoolOptions

of the CheerioCrawler ([docs](https://crawlee.dev/js/api/core/interface/AutoscaledPoolOptions#isFinishedFunction)). Here is an example implementation for this function:

autoscaledPoolOptions: {
    isFinishedFunction: async () => !playwrightCrawler.running && 
            (await crawlerA._defaultIsFinishedFunction()),
},

Thanks for your reply. I have used two crawler with promise.all to make the crawler be parallelized running. The important thing is: 1. add a isSearchDone callback when create a new crawler (at the entrypoint) just like you pasted 2.add a custom request sink function and pass it to first cralwer, in the first cralwer router calling the sink function push the new urls to cherrio crawler. It works perfect with my setup(playwright for js rendering and cheerio for lightweight fast SSR page scrapping) but haven't tested in a large scaled cralwers(I believe this is enough for most cases)

message has been deleted