-------------- We have a question for our beloved community! 👇 We will be building out support for declarative configuration in the next couple of weeks. Do you think we should build it 1️⃣ using Terraform, 2️⃣ building our own configuration DSL or 3️⃣ another option? Please vote with emoji reactions! And don’t hesitate to detail your thoughts in the thread.

Also, any context you can add around your vote would be super helpful for us to understand your use case better!

Terraform is the gold standard for infra config. I'd love to have a module for airbyte and plug into our infra

even for configuring Airbyte? (so for connectors/sources/destinations etc.. not just for configuring the Airbyte infra)

Yaay to the initiative! Not experienced enough to vote for an alternative.

for airbyte configuration (not airbyte infra config) I'd go for 2️⃣ to make it more portable. I'm partial to YAML for that kind of thing. that way if I was running airbyte locally, then decided to deploy to EC2, then when if y'all have a cloud SaaS version, I could just export/import the configuration without needing to care about how airbyte is set up to run, but just have to care about what it runs.

@Alex Dovenmuehle I think in all cases (YAML or Terraform) we will definitely support what you are describing 😉 One of the main tradeoffs I see between YAML and TF is: with YAML, it will be very customized to Airbyte and the mental barrier to entry is small. With TF, the user will potentially need to learn a new concept (Terraform) before they can configure their Airbyte instance, so the barrier to entry is higher, but like Sam says, it’s the gold standard for declarative infra configuration

Part of the question seems to be if our users source/destination/sync configuration as “infra configuration”.

Good discussion! I feel the barrier to entry for TF is a low one, and I found HCL to be very intuitive. Many data engineers on our teams have already learned it and are currently leveraging it for DE infrastructure. I'm a big Terraform fan.

@s We don't need to configure airbytes sources/connections with terraform, although that might be useful for people who would like a declarative way of doing that. We, at the moment, don't have that need.

I'd definitely go with your own (YAML-based) DSL, because 1) as you pointed out, no one got to use a new concept 2) you're decoupled from tf, 3) (most important in my opinion) if you use terraform for configuration, you'd nudge people towards a "not-best-practice" in the IaC world (on the fly configuration of running instances/containers). Instead, you should leave the part on "how do I get the config => the system" up to the end-user...

A Terraform provider for at least the initial configuration would be awesome

Btw. where do you want to put the "declarative configuration"? Like in a "repository" which could simply be put up on S3/ or a docker container inside the cluster?

I think my only concern with a TF isn't the format itself, more the signal of who the audience for this tool is. obviously mostly data engineers (where there's some overlap with devops who I would consider the main users of terraform), but I'd think you'd want to be able to also cater to the analytics engineering crowd who are used to dbt who might look at TF format and be scared away because they haven't used it before