https://linen.dev logo
r

Ryan N

12/14/2021, 1:08 AM
maybe the better question is how to set up this parameter of .env under kube/overlays/stable-with-resource-limit
Copy code
# Worker pod tolerations and node selectors
JOB_POD_TOLERATIONS=
JOB_POD_NODE_SELECTORS=
u

user

12/14/2021, 1:13 AM
I’ve worked on this setup. 1. I strongly recommend using helm charts. 2. set
Copy code
extraEnv:
    - name: JOB_POD_TOLERATIONS
      value: "key=airbyte-worker,operator=Equal,value=true,effect=NoSchedule"
r

Ryan N

12/14/2021, 1:14 AM
Note there was a breaking change between versions, prior to 0.33.x it was called WORKER_POD, now it is called JOB_POD
u

user

12/14/2021, 1:15 AM
Also, another think to keep in mind is that you need to taint all of your node groups, otherwise they can get scheduled on non-tained nodes.
u

user

12/14/2021, 3:57 AM
@Jags (Kloud.io) thanks mate, really appreciate your help. I’m have switching to helm and running into some issue regarding to externaldb. for some reason, I have set postgres.enabled = false and fill parameters for external db ( except existingSecret and existingSecretPasswordKey ) but get the issue of not found externaldb secret. have you ran into the same issue when deploying with helm ?
u

user

12/14/2021, 3:58 AM
yes I did, I had to add a new secret and apply in my EKS cluster and use the name in the values
Copy code
apiVersion: v1
kind: Secret
metadata:  
  name: airbyte-externaldb
type: Opaque
data:
  password: <base-64-encoded>
u

user

12/14/2021, 3:59 AM
And you add them in the values.yaml
Copy code
externalDatabase:
  ...
  existingSecret: airbyte-externaldb
  existingSecretPasswordKey: password
u

user

12/14/2021, 5:10 AM
@Jags (Kloud.io) thank you a lot, not sure if I do anything wrong in the secret but I quickly run into the issue of connection refuse to external database of worker pod and server pod. I tried few workaround but nothing avail here is how i did it I created a new file name ‘db-external-secret.yaml’ with this values
Copy code
apiVersion: v1
kind: Secret
metadata:  
  name: db-external-secret
  namespace: airbyte
type: Opaque
data:
  password: my-base-64-encode
and I changed the values.yaml to this
Copy code
postgresql:
  enabled: false
  postgresqlUsername: airbyte
  postgresqlPassword: airbyte
  postgresqlDatabase: db-airbyte
  existingSecret: ""

externalDatabase:
  host: my-host
  user: myusername@myhost
  password: mypassword
  existingSecret: db-external-secret
  existingSecretPasswordKey: password
  database: postgres
  port: 5432
then i got this logs not sure if it has anything to do with svc configuration
u

user

12/14/2021, 5:16 AM
Hmm that doesn't seem to be an issue with connecting to the database, but the temporal. Did you change any SVC settings? Are your ports open for incoming traffic?
3 Views