https://linen.dev logo
m

Muhammad Imtiaz

02/18/2022, 10:40 AM
Hell team, I'm trying to install Airbyte on EKS using helm chart. I want to use my RDS postgres db instead of default one which came through helm charts. Documentation says that I can install it. All I've to do is disable default postgres helm chart and provide external db creds in the values.yaml. But the issue is, secret doesn't created with external db creds. Can you please let me know what else I've to do? https://docs.airbyte.com/deploying-airbyte/on-kubernetes#:~:text=planned%20logging%20improvements.-,U[…]0an%20external%20DB,-After%20Issue%20%233605 https://github.com/airbytehq/airbyte/blob/f927dc501d05dc1030d130c3eb5544e5430af6c2/charts/airbyte/values.yaml#L559
👀 2
1
h

Harshith (Airbyte)

02/21/2022, 5:01 PM
@Davin Chia (Airbyte) is it possible to help here ?
d

Davin Chia (Airbyte)

02/23/2022, 8:01 AM
Hmm I haven’t had time to explore how this works with external DBs
is it possible to use the kustomize charts for now? I believe they are easier to configure
m

Muhammad Imtiaz

02/23/2022, 2:03 PM
Thank you @Davin Chia (Airbyte) for your response. I've deployed it via Kustomize & configured external RDS postgres. But I''m facing an issue with Temporal service... It would be highly appreciated if you can help here. It keeps on restarting.
d

Davin Chia (Airbyte)

02/23/2022, 2:08 PM
does Temporal have the right database configuration?
m

Muhammad Imtiaz

02/24/2022, 9:54 AM
@Davin Chia (Airbyte) I've provided the following db URL to temporal service. Pod is up now, but If I see its logs, it is not able to connect to db
Copy code
jdbc:<postgresql://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com:5432/postgresql>
postgresql
database is created. Temporal service is not able to connect. Following are the tempral service logs
@Davin Chia (Airbyte) If I give the following DB URL without
jdbc
Copy code
<http://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com|cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com>
d

Davin Chia (Airbyte)

02/24/2022, 10:23 AM
can you try and pass in the ip?
m

Muhammad Imtiaz

02/24/2022, 10:57 AM
@Davin Chia (Airbyte) RDS postgres has no IP address. It has public endpoint.
d

Davin Chia (Airbyte)

02/24/2022, 10:59 AM
your RDS is open to the internet?
m

Muhammad Imtiaz

02/24/2022, 11:01 AM
No, It is not exposed as public. It is in private subnets.
But, EKS & RDS are in same VPC.
@Davin Chia (Airbyte) My RDS instance & EKS are deployed in private subnets but in same VPC. I'm able to connect through EC2 in the same VPC.
d

Davin Chia (Airbyte)

02/24/2022, 11:19 AM
all DNS names can be converted to ip - e.g.
nslookup <http://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com|cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com>
. you can google to find out more information you can look at the the original temporal chart to see what is passed in. either the dns name or the ip should work. I’m sorry but we don’t offer infrastructure debug help - each company’s infrastructure is set up too differently for this to be practical. my general tips are to double check the env vars are given the right values and that the pod is receiving them! I’d also check that there are no networking restrictions on your RDS instance
m

Muhammad Imtiaz

02/24/2022, 11:56 AM
@Davin Chia (Airbyte) I've cross checked my ENV variables, they seems to be alright. As whatever database I provide, when airbyte spins up, it creates all the tables into that datbase. But the same database is not accessible by Temporal service. Following are the env to temporal service:
Copy code
DATABASE_USER=<user>
DATABASE_PASSWORD=<password>
DATABASE_HOST=<http://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com|cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com>
DATABASE_PORT=5432
DATABASE_DB=postgresql
DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT/${DATABASE_DB}
DATABASE_URL=jdbc:<postgresql://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com:5432/postgresql>
@Davin Chia (Airbyte) I've also checked correct env variables are feeded to the temporal pod
Anyone wants to add something here?
m

Marcos Marx (Airbyte)

03/01/2022, 10:53 PM
@Muhammad Imtiaz if you don’t try to use an external db for temporal; are you able to run Airbyte?
m

Muhammad Imtiaz

03/02/2022, 7:33 AM
@Marcos Marx (Airbyte) yes, I'm able to use external db for other components. The only issue is temporal doesn't able to communicate with external db.
m

Marcos Marx (Airbyte)

03/04/2022, 12:22 AM
@Muhammad Imtiaz Airbyte uses the default image of temporal.io so if you want to edit and use an external database for temporal you need to check their docs/configuration, check some discussions here
m

Muhammad Imtiaz

03/07/2022, 7:42 AM
@Marcos Marx (Airbyte) I figured out that
temporal
looks for database named
postgresql
. It doesn't work with anyother db names. Thank you very much for all your help.
👍 1
2 Views