How to combine the scraping results for Crawlee Pl...
# crawlee-js
j
Hello folks! I'm building an actor with Crawlee and playwright for bbb scraping. I have completed the "detail" content parsing, and I made two booleans inputs to let users decide whether to scrape reviews, complaints or not. Then I realized if they choose both, I'll need to send two more requests and I don't know how to use the router to combine the results(the reviews, complaints and detail results) before using
Dataset.pushData
. https://cdn.discordapp.com/attachments/1413859628948783124/1413859629263228959/image.png?ex=68bd76e3&is=68bc2563&hm=08c8a8d82a2140e8d2dcd815c57293510b770d4a928f6cea77fe1d77191de08d&
a
message has been deleted
e
You need a way to persist state per entity (e.g., one business from BBB) across multiple requests, and only pushData() once all requested pieces (detail + reviews + complaints) are collected. In Crawlee, you do this by:
Copy code
1. Storing a “partial result” in request.userData.
2. Passing it along to subsequent requests.
3. Once all needed data are gathered, pushing it to the dataset.
j
Oh I see, that make sense, thank you! This solves my problem.🫶