Question for contributors - in cases where you hav...
# connector-contributions
w
Question for contributors - in cases where you have contributed a change that you want to use in your own self-hosted airbyte install, how do you deploy your development while still using the actual connector vs a custom connector. I see how to create a custom connector and that is straight forward, but then when the real connector has the change introduced, switching back seems like a headache. I can also see how you could push the image directly into the airbyte kubernetes cluster just like I do when I'm actually testing in my development environment, but is there a risk underlying cloud upgrade could force a reload and the image wouldn't be found on the public repo. I'm curious what's the best practice everyone else follows for this.
p
What I'm about to write is no best practice at all ⚠️, because it relies on Airbyte internals, but it works in my project. I have some custom versions of existing connectors. In Airbyte's database, in
actor_definition_version
table there are
docker_repository
and
docker_image_tag
columns, and you can update them to "switch" between official version and version you have stored in public/private registry (I use AWS ECR as a registry for Docker images). As long as there is compatibility in handling connection's state, it shouldn't be any issues with "switching".
b
thanks @Przemysław Dąbek that worked for us! I ended up creating a new record in
actor_definition_version
rather than updating the existing one
n
I think we should make a guide with a best practice for this, once we figure out what works well.
So when this happens on Cloud, we have tools to build a dev build, or a release candidate build, and pin only some workspaces to use it. The way dev builds work is: 1.
airbyte-ci connectors build
with flags for a pre-release build. This would build the image from the current work tree code, mark it with the version in metadata +
-dev.{sha}
, and it would push the metadata of that version to metadata registry, just not mark it as latest. This in turn allows Airbtyte-server to ingest and use it when we later pin it. 2. Then, we use internal tools to pin the particular actor (connection) to the particular dev build, if we need to experiment with it (or customer requests early access to a patch) 3. Then, when the patch is mainlined in the next public connector release, we remove the version pin. Now, this is very close to what @Przemysław Dąbek suggests. Because by “we pin” I mean that we make a DB record mostly how he describes. What would be good here: 1. Document this flow and describe how to pin and unpin 2. Document that airbyte-server has an api endpoint for this (allegedly?) and describe how you could use it instead of editing the database directly. 3. Perhaps provide a retool dashboard template that users could use to pin, or make the UI for this.