Or if you would rather have the go client in the airbyte monorepo, I could make a place for it and open a PR. I don't know how well the versioning will work for the go client within a monorepo, but it might be worth a try.
Augustin Lafanechere (Airbyte)
03/24/2022, 6:08 PM
Hi @Aaron Williams, I investigated the development of an Airbyte terraform provider. I soon hit the problem of declaring terraform resources for each kind of source / destination we have. According to my research the current terraform SDK requires to hardcode the schema of each resource. And this is quite hard to accomplish considering the wide diversity of connectors configuration schema (basically the schema of the connectionConfiguration in our API for source / destination endpoint). The solution to tackle this problem could be to leverage the Dynamic Pseudo Type of terraform, which is was not available in the main terraform SDK at the time of my research. Kubernetes SRDs leverage this type using Terraform-plugin-go, this plugin would make the development of the provider a bit complex and lower level, this is why we decided to not jump into this right now. If you have an alternative solution/approach to the problem I mentioned, feel free to start developing a terraform provider from a fork of our monorepo and open PRs whenever you like. We're also open to further discussions on this topic on Discourse, in our Product Feedback and Ideas forum.
I can't finish this message without mentionning that we're soon releasing a CLI for Airbyte that will help you manage Airbyte configuration in a declarative manner, with YAML file. Feel free to checkout the progress on this project on #public-octavia-cli
03/25/2022, 11:54 PM
Thanks @[DEPRECATED] Augustin Lafanechere for the information. I am aware of the upcoming release of the CLI which can be used to manage Airbyte in a declarative manner. That is exciting and what we will probably end up using.
Our company uses terraform for other services and our preference would be to use terraform over the cli.
I understand the challenges of maintaining many different connectors as terraform resources. I agree it is a difficult task. I am hoping that there will be some way to generate the static terraform resource code from the connectors' spec.json files. I don't know if that will work, but maybe it will. I think the AWS terraform provider autogenerates many of their static terraform resources, but I haven't looked at that repo in a while.
I think my plan is to create a private terraform provider for my company's use. Hopefully someday I can get automatic connector terraform resource code generation based on spec.json working. At that time I will make the repo public and we can talk about how to allow the community to contribute and take ownership.