Greetings everyone. Is there an API wrapper for airbyte that will allow me to store sources/destinations/jobs as code? I realize I could build one but I was hoping someone has done that already!

Not an official one. There is a open issue to discuss about it https://github.com/airbytehq/airbyte/issues/1860. I know some users start doing work on this. @Harshith (Airbyte) had made a incredible work for a CLI https://github.com/harshithmullapudi/airbyte-cli

Thanks @[DEPRECATED] Marcos Marx we have more some features internal as we are still developing so feel free to comment if you need something ere. @Allen FP

Just anything that would allow me to source control and modify sources/destinations/and jobs programmatically would be a huge help. Thanks!

The direction of THIS INCOMPLETE PROJECT is pretty much exactly what I am looking for.

@Robert Stolz 😬

@[DEPRECATED] Marcos Marx even if @Robert Stolz has to implement this he can't dump the secrets into yaml file right as the API wouldn't return the secrets. Do you have any idea here ?

I think the deployment export does the same if you look inside tar file it has yaml files. @Allen FP. Any thoughts here ?

The way i'm doing it is providing a separate secrets.yml that specifies the secrets for each connector. One semi-blocking item here is that each connector uses a different naming convention for its secret.

secret_key

vs

token

vs etc... This could be tweaked in the API.

@Allen FP https://github.com/garden-of-delete/airbyte-tentacle Here's a link to the still WIP repo. The big things i'm missing for the first release are: 1. tests 2. examples 3. Better way to specify connections in the yaml file. I've been working on an idea where you can "tag" sources and destinations in the yml. Then, in the config.yml, connections can be specified in terms of connecting one set of tagged sources to a set of tagged destinations. The missing piece here is a cleaner / more human readable way to have granular control over the configuration of the connections. 4. A better work around for the secrets issue I describe above.

To expand on 3: You can import and export a fully configured connection from the API no problem. The issue is that it is far from easy on the eyes. Very hard to make tweaks in the yml. That's what is motivating me to come up with a better way to manipulate connections in the yml file.

If you want some example config.yml files that work with this tool, DM me and I can provide.

Guys I tried to do it. https://github.com/harshithmullapudi/airbyte-cli here is it. Happy to hear any suggestions. Things to do. Update the entities which will try to finish by the End of week.