Saving scraped data from dynamic URLs using Crawle...
# crawlee-js
m
Hello all. I've been trying to build an app that triggers a scraping job when the api is hit. The initial endpoint hits a crawlee router which has 2 handlers. one for the url-list scraping and the other for scraping the detail from each of the detail-page. (the url-list handler enqueues the next url-list page to url-list handler too btw) I'm saving the data from each of these scrapes inside a KVstore but I want a way to save all the data in the KV store related to a particular job into database. The attached screenshots are the MRE snippets from my code. https://cdn.discordapp.com/attachments/1292736331616616520/1292736331935387709/code-mre-1.png?ex=6704d207&is=67038087&hm=553c5a2bb10e1f9a3b009bbcb7d5909bb818f5779dbe755a8a7d1b26e58ac5aa& https://cdn.discordapp.com/attachments/1292736331616616520/1292736332652740701/code-mre-2.png?ex=6704d207&is=67038087&hm=f7e03757d54a6d6b6abea69d5893b2ec768b1110b509d0700f9cc03d59d5ea47& https://cdn.discordapp.com/attachments/1292736331616616520/1292736333126565981/code-mre-3.png?ex=6704d207&is=67038087&hm=afc77fc8964dc3c426399b8a7cad57ba462bc9946043886ab2db10f961d28e11&
h
message has been deleted
I want to be able to hit the endpoint concurrently for multiple job request. Each job gets routed to handler 1- extracts let's say 10 detail urls and routes each one to detail-url. then each detail url handler saves the detail page results into a kv store
@Oleg V. @Marco