https://linen.dev logo
Join Slack
Powered by
# kubernetes
  • a
    Hi, I followed https://docs.airbyte.com/deploying-airbyte/on-kubernetes/ to deploy airbyte into our kubernetes cluster by running
    Copy code
    kubectl apply -k kube/overlays/stable
    I am trying to create a Postgres source by connecting to the Postgres instance deployed as airbyte db, but got 
    Check Connection Failed!
    , I checked airbyte-server log and got
    Copy code
    2022-08-02 19:27:06 INFO i.a.s.RequestLogger(filter):112 - REQ 172.16.41.115 POST 200 /api/v1/scheduler/sources/check_connection - {"connectionConfiguration":"REDACTED","sourceDefinitionId":"decd338e-5647-4c0b-adf4-da0e75f5a750"}
    which made me feel like the connection is established successfully, what should I check next to proceed?
    k
    • 2
    • 3
  • d
    Copy code
    ➜ kc describe clusterrole airbyte-postgres-access
Name:         airbyte-postgres-access
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  pods       []                 []              [get watch list]
  services   []                 []              [get watch list]
  • d
    Copy code
    ➜ kc describe clusterrolebinding airbyte-postgres-access
Name:         airbyte-postgres-access
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  airbyte-postgres-access
Subjects:
  Kind            Name           Namespace
  ----            ----           ---------
  ServiceAccount  airbyte-admin  airbyte
  • d
    so to work around, i deployed the airbyte resources in the same namespace where a postgres instance lives.
  • d
    still not able to connect.
  • d
    here's the postgres service:
  • d
    Copy code
    ✗  kc -n bridge describe svc postgres
Name:              postgres
Namespace:         bridge
Labels:            <http://kustomize.toolkit.fluxcd.io/name=bridge|kustomize.toolkit.fluxcd.io/name=bridge>
                   <http://kustomize.toolkit.fluxcd.io/namespace=flux-system|kustomize.toolkit.fluxcd.io/namespace=flux-system>
Annotations:       <none>
Selector:          app=postgres
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.100.116.241
IPs:               10.100.116.241
Port:              postgres  5432/TCP
TargetPort:        5432/TCP
Endpoints:         192.168.73.88:5432
Session Affinity:  None
Events:            <none>
  • d
    it's up
    Copy code
    ✗  kc -n bridge get ep postgres -o wide
NAME       ENDPOINTS            AGE
postgres   192.168.73.88:5432   168d
  • d
    in airbyte webapp, i've tried the following for hostname: 
    postgres
    , 
    postgres.svc.cluster.local
    , 
    postgres.bridge.svc.cluster.local
    , and even direct ip 
    192.168.73.88
    -- none of them is able to connect.
  • d
    airbyte worker logs always indicate:
    Copy code
    2022-08-05 19:55:13 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-05 19:55:13 INFO i.a.i.s.r.AbstractDbSource(check):84 - Exception while checking connection:
Log4j2Appender says: 2022-08-05 19:55:13 INFO i.a.i.s.r.AbstractDbSource(check):84 - Exception while checking connection:
2022-08-05 19:55:13 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 60001ms.
Log4j2Appender says: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 60001ms.
    a
    • 2
    • 1
  • g
    Hi All, I am trying to deploy Airbyte on AWS EKS using the helm chart and by connecting to a Postgres RDS instance instead of running it in a container. The bootloader container times out and I don't get any logs either. I tested for connectivity from the nodepool, and I can interact with the RDS instance allright. It must be something AWS-specific that I am missing as the helm chart worked fine on minikube. Is there a step by step guide I can follow? The most helpful would be an idotproof explanation on how do I get from an empty AWS account to a working Airbyte on EKS + RDS setup atep by step. It could also be used to update the documentation online. Thanks in advance!
  • g
    Hi Davis, Do you have some infrastructure as code you could share to get a working EKS-deployment?
  • d
    Hi all , in the pod-sweeper service i edited the script so now it will delete completed state pods every two minuteds , and failed state pods every 15 minutes dose it wipe the logs of that job with that ?
  • g
    Hi all, Trying to upgrade to the 1.0.0 charts, now published at 
    <https://airbytehq.github.io/helm-charts/>
    ... and I'm getting this error:
    Copy code
    Helm upgrade failed: template: airbyte/charts/server/templates/deployment.yaml:240:15: executing "airbyte/charts/server/templates/deployment.yaml" at <eq .Values.deploymentMode "oss">: error calling eq: incompatible types for comparison
    Any thoughts?
    m
    • 2
    • 5
  • c
    Hi, I just made a fix on kube and made a PR, if anyone can review it thx : https://github.com/airbytehq/airbyte/pull/15876 Without this fix I cannot deploy anymore, airbyte/cron tag is on an old version.
  • j
    Chart: 0.45.1 Image tag: 0.40.1
  • j
    Hi all, how can I contribute to Helm chart? I found some small bugs but I notice the one in the main repo is a bit different than the one in helm chart repo and the version convention is bit different too.
    s
    m
    • 3
    • 3
  • n
    Hello Everyone, I am facing an issue with the temporal pod on GKE when I configure an external database instead of a built-in. It keeps on crashing. I never encountered this issue in our pre-prod environments. I tried multiple versions(0.38.4,0.39.35,0.39.36). Are there any updates that happened to the temporal image?
    z
    • 2
    • 9
  • a
    Hi yall! Performing an airbyte upgrade and moving up to the latest. Wanted to confirm that the scheduler pod has been removed/replaced ? Does that functionality live with temporal now ?
    • 1
    • 1
  • a
    Noticing the scheduler pod has been removed from the helm chart.
  • l
    Hello guys ! Yesterday I was working with the 0.45.1 chart version, but today we have an upgrade of the charts and now the latest release is 0.42.0, but my server and webapp pods dont deploy, could anyone help me with this ? Im using the most default command and even with this Im not able to deploy PS: helm install --debug airbyte airbyte-oss/airbyte -n airbyte-helm Ps2: This is my unique log at server pod: 
    INTERNAL_API_HOST environment cannot be null
    m
    c
    j
    • 4
    • 13
  • d
    Since our chart upgrades from 
    0.45.1
    to 
    0.42.0
    , I am seeing this error when attempting to deploy :
    Copy code
    Error: template: airbyte/charts/webapp/templates/ingress.yaml:53:33: executing "airbyte/charts/webapp/templates/ingress.yaml" at <.Release.Name>: nil pointer evaluating interface {}.Name
    l
    d
    • 3
    • 32
  • m
    It looks like all the historical Helm chart releases have been removed from https://github.com/airbytehq/helm-charts without warning, meanwhile there are multiple open issues (16587, 16586) with the latest Helm chart release (which is now the only release available)
  • n
    Hello, Our security team detected the “CVE-2022-27664” vulnerability with temporal and airbyte-worker container images. These two images use golang 1.15.8(tctl, sql-tool, and temporal-server packages), 1.10(dockerize package), and 1.17.2(airbyte-worker). They are asking us to upgrade the golang version to 1.18.6. Could you please suggest how to upgrade these images? Currently, I deployed airbyte-0.39.35-alpha on gke.
  • d
    I’ve opened a PR to address the broken Ingress template value ---> https://github.com/airbytehq/airbyte/pull/16931
    l
    • 2
    • 1
  • l
    Hey guys ! In the new version 0.41.1 it seems to have duplicated the minIO resource and an error with micronaut in the worker pod. Anyone could deploy this version, if so how?
    j
    b
    • 3
    • 11
  • j
    Hey, I can’t follow the helm chart versioning anymore.
  • j
    helm chart 0.41.1 is the latest version and not 0.41.2?
  • l
    Today when I checked the artifact hub page I had this question too haha
  • j
    Hi Everyone! which recent version of the helm chart is working without error (out of the box ) ? I'm having some errors deploying latest ones
    j
    • 2
    • 2