Hi guys! I'm new to Airbyte and I would like to understand if it's right for our data pipeline. Currently we have different systems that push an HTTP request to our integration tools based on an event (ex: an order is created). Can we just set Airbyte to receive HTTP request? Is it feasible to have something like a queue, so that we wouldn't loose any ingoing requests? Thank you so much!

Hey @Fausto if I understand correctly, you want Airbyte to expose a URL endpoint where you would hit a POST request with JSON body as your data. Airbyte would process these records and move them to your destination. This sounds like a webhook connector. We dont have support for now but am sure we have this somewhere on our road map. Let me find the ticket for you

Sure! Anyway would it be possible to just create a flask api as connector?

The problem is that the connector container doesnt keep running. They are trigger only in the job and shut down after finished.

So, probably we should create a webhook configuration where the containers run all time long. I think the main problem could be the container scaling in order to not loose any incoming request. Maybe an easier solution would be to put a queue in front of the api to store the incoming requests. What do you think? Thanks!

Do you need low latency or just is a requirement the webhook process? Other suggestion would use a cloud function to dump data into S3 or GCS and use a connector to sync data store in it batch

Low latency is better but not strictly required. Currently we have many process in different places (db, webservices, powershell/bash...) and we would like to have everything in a single place to be easier to mantain and develop.

Got it. Dumping the data to a storage blob is cheap and can use as a data lake. In the future could be migrate to a webhook connector but at the moment this option is not available