https://linen.dev logo
#feedback-and-requests
Title
# feedback-and-requests
a

Allen FP

08/09/2021, 5:58 PM
Greetings everyone. Is there an API wrapper for airbyte that will allow me to store sources/destinations/jobs as code? I realize I could build one but I was hoping someone has done that already!
u

user

08/09/2021, 6:05 PM
Not an official one. There is a open issue to discuss about it https://github.com/airbytehq/airbyte/issues/1860. I know some users start doing work on this. @Harshith (Airbyte) had made a incredible work for a CLI https://github.com/harshithmullapudi/airbyte-cli
a

Allen FP

08/09/2021, 7:08 PM
Thanks @[DEPRECATED] Marcos Marx we have more some features internal as we are still developing so feel free to comment if you need something ere. @Allen FP
u

user

08/09/2021, 7:30 PM
Just anything that would allow me to source control and modify sources/destinations/and jobs programmatically would be a huge help. Thanks!
u

user

08/09/2021, 7:31 PM
The direction of THIS INCOMPLETE PROJECT is pretty much exactly what I am looking for.
u

user

08/09/2021, 7:33 PM
@Robert Stolz 😬
u

user

08/11/2021, 8:07 AM
@[DEPRECATED] Marcos Marx even if @Robert Stolz has to implement this he can't dump the secrets into yaml file right as the API wouldn't return the secrets. Do you have any idea here ?
a

Alderson

08/11/2021, 8:36 AM
I think the deployment export does the same if you look inside tar file it has yaml files. @Allen FP. Any thoughts here ?
u

user

08/11/2021, 4:21 PM
The way i'm doing it is providing a separate secrets.yml that specifies the secrets for each connector. One semi-blocking item here is that each connector uses a different naming convention for its secret.
secret_key
vs
token
vs etc... This could be tweaked in the API.
u

user

08/11/2021, 4:24 PM
@Allen FP https://github.com/garden-of-delete/airbyte-tentacle Here's a link to the still WIP repo. The big things i'm missing for the first release are: 1. tests 2. examples 3. Better way to specify connections in the yaml file. I've been working on an idea where you can "tag" sources and destinations in the yml. Then, in the config.yml, connections can be specified in terms of connecting one set of tagged sources to a set of tagged destinations. The missing piece here is a cleaner / more human readable way to have granular control over the configuration of the connections. 4. A better work around for the secrets issue I describe above.
j

Jeff Crooks

08/11/2021, 4:26 PM
To expand on 3: You can import and export a fully configured connection from the API no problem. The issue is that it is far from easy on the eyes. Very hard to make tweaks in the yml. That's what is motivating me to come up with a better way to manipulate connections in the yml file.
u

user

08/11/2021, 4:26 PM
If you want some example config.yml files that work with this tool, DM me and I can provide.
u

user

08/12/2021, 1:41 PM
Guys I tried to do it. https://github.com/harshithmullapudi/airbyte-cli here is it. Happy to hear any suggestions. Things to do. Update the entities which will try to finish by the End of week.
2 Views