Are airbyte a good choice when implementing stream...
# ask-community-for-troubleshootingd
David Beck
10/29/2021, 10:48 AMAre airbyte a good choice when implementing streaming ingestion?
I'm thinking about receiving events from webhooks or using change data capture.
k
Kamil Breguła
10/29/2021, 1:09 PMNot the best. Have you considered using Apache Beam, Google Dataflow or Apache Spark?
Kamil Breguła
10/29/2021, 1:10 PMWebhooks are quite hard to process, but it will be a lot easier if you set up a small server that will receive your message and forward it to PubSub.
Kamil Breguła
10/29/2021, 1:12 PMYou can find some examples here: https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
d
David Beck
10/29/2021, 2:52 PMThanks! I have looked into dataflow, also datastream (but it's still a bit limited).
I'll try to learn how to use that. And for webhooks maybe cloud functions and pubsub
2 Views