I would just suggestusing apify plus their crawlee platform https://crawlee.dev/