https://linen.dev logo
#ask-community-for-troubleshooting
Title
# ask-community-for-troubleshooting
a

Afam Uzoka

08/05/2021, 1:21 PM
Can you run airbyte on multiple instances behind a load balancer? The database will run on AWS RDS Postgres. Is this architecture possible?
c

charles

08/05/2021, 5:41 PM
Hi Afam!
Can you run airbyte on multiple instances behind a load balancer
i'm not sure what you mean by this piece? are you trying to run multiple instance of airbyte at the same time?
The database will run on AWS RDS Postgres.
yes. If you explore the
.env
file in the root of the project you can replace the database credentials there with whatever database you want. If you do that, you can then also remove the database container from the K8s or docker compose setup. We have more instructions on that here.
h

haf

08/08/2021, 9:04 PM
Does this mean that the Airbyte server does leader election and doesn't inadvertently race between the servers, to schedule workers?
c

charles

08/09/2021, 4:31 PM
Right now Airbyte supports one server 😛 so we don't have race problems.
We are going to add for multiple servers and have to do as you mentioned, some sort of leader election /locking scheme.
h

haf

08/10/2021, 2:20 PM
Ok, I just looked into this on k8s and the 'lease' CRD:s. Seems like the "lease" is nor available in GKE (GCP) since it is "too close" to etcd (security wise). Are you doing to continue building the server in Java?
u

[DEPRECATED] Marcos Marx

08/11/2021, 2:23 AM
Are you doing to continue building the server in Java?
Yes, @haf curious about this question.
h

haf

08/11/2021, 6:34 AM
@[DEPRECATED] Marcos Marx Nothing particular; it's just that having different programming languages for the ELT makes operating it much more expensive, because you need programmers with two competencies to extend it, rather than programmers with just Python. Furthermore, I can imagine use cases for destinations/outbound sync where global locks would make sense for connectors too (not just the main server), so if I'd solve it in python it would apply across the Airbyte project.
I was also thinking of looking into a raft algo https://raft.github.io/#implementations that would solve leader election in-proc which would be compatible across nodes / languages.
As a last bit; as I understand it the stdout "problem" as it pertains to leaking data in GCP / GKE / k8s hasn't really been solved and we're working on a solution to this as we speak, but it sort of doesn't include java 😉
🥲 1
4 Views