Can you run airbyte on multiple instances behind a load balancer? The database will run on AWS RDS Postgres. Is this architecture possible?

Hi Afam!

Can you run airbyte on multiple instances behind a load balancer

i'm not sure what you mean by this piece? are you trying to run multiple instance of airbyte at the same time?

The database will run on AWS RDS Postgres.

yes. If you explore the

.env

file in the root of the project you can replace the database credentials there with whatever database you want. If you do that, you can then also remove the database container from the K8s or docker compose setup. We have more instructions on that here.

Does this mean that the Airbyte server does leader election and doesn't inadvertently race between the servers, to schedule workers?

Right now Airbyte supports one server 😛 so we don't have race problems.

Ok, I just looked into this on k8s and the 'lease' CRD:s. Seems like the "lease" is nor available in GKE (GCP) since it is "too close" to etcd (security wise). Are you doing to continue building the server in Java?

Are you doing to continue building the server in Java?

Yes, @haf curious about this question.

@[DEPRECATED] Marcos Marx Nothing particular; it's just that having different programming languages for the ELT makes operating it much more expensive, because you need programmers with two competencies to extend it, rather than programmers with just Python. Furthermore, I can imagine use cases for destinations/outbound sync where global locks would make sense for connectors too (not just the main server), so if I'd solve it in python it would apply across the Airbyte project.