Hi all! Looking for some direction on how one might orchestrate airbyte in the following situation. There is an external system that emits a webhook event with

order_id

and basically we want to build a connection that triggers for the specific

order_id

each time the webhook fires - the connection needs to hit a source at

GET orders/<order_id>/info

Since we can’t pass any parameters in the manual trigger connection api we are struggling to figure out how to set this up. There are an unknown amount of unique orders in the source (could be upwards of 100's of k’s) and there is no need to run a full sync - we only need to sync the specific order that is sent in the webhook event payload. Basically I want to pass a variable/parameter to the connection when I manually trigger it so it can be run for the emitted order. We don’t really want to be creating thousands of connections (one per order) as that would mean we need to programmatically generate them (among other reasons). I do see that we could possibly create some setup using partitions where there is a parent connector that reads a source of unprocessed webhook order_id’s but that seems roundabout. I wanted to ask if anyone can point me to some examples of solving this paradigm or offer some guidance. Thanks in advance!

@Zachary Martinez Hansen - you could look at kestra or dagster or something similar? I have been using kestra as a way to orchestrate DBT after an airbyte run

thanks @Josh! Not familiar with dagster but going to dive in now 🙂