https://linen.dev logo
Join Slack
Powered by
# kubernetes
  • h
    Hi team , After upgrading airbyte to the latest version 
    0.39.27-alpha
    , Incremental sync fails with the below error, where as the full | refresh overwrite replication works
    Copy code
    2022-06-28 09:03:58 source > 	at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:76) ~[debezium-core-1.9.2.Final.jar:1.9.2.Final]
2022-06-28 09:03:58 source > 	at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:155) ~[debezium-core-1.9.2.Final.jar:1.9.2.Final]
2022-06-28 09:03:58 source > 	at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:137) ~[debezium-core-1.9.2.Final.jar:1.9.2.Final]
2022-06-28 09:03:58 source > 	at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109) ~[debezium-core-1.9.2.Final.jar:1.9.2.Final]
2022-06-28 09:03:58 source > 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
2022-06-28 09:03:58 source > 	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
2022-06-28 09:03:58 source > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
2022-06-28 09:03:58 source > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
2022-06-28 09:03:58 source > 	at java.lang.Thread.run(Thread.java:833) ~[?:?]
2022-06-28 09:03:58 replication-orchestrator > Source has no more messages, closing connection.
    m
    • 2
    • 2
  • u
    Hey guys, thanks for your wonderful product! I would like to clarify if you have any plans to change back "kubectl cp" to "kube java client"? I'm asking because "kubectl cp" is just a wrapper around "kubectl exec", and we have very strict security policies on our local kubernetes cluster, and in particular the "pods/exec", "pods/attach" roles are prohibited. More details in this thread.
    • 1
    • 6
  • g
    I have been trying to do a helm upgrade of airbyte from v0.39.1-alpha to v0.39.34-alpha via fluxcd and getting helm upgrade failed messages, apparently from the minio subchart secrets template. Apparently it can't locate the root-password from an existing secret?? Airbyte seems to have jumped from version 7.3.2 of bitnami minio subchart (which is no longer available??) to version 11.6.2 so it's possible a lot has changed in minio. Should I manually delete the airbyte-minio secret and try the upgrade again? Should I set `minio.auth.rootUser`/`rootPassword` to the same defaults as Airbyte (`minio`/`minio123`)? Do I have to manually delete the minio pods?
    f
    • 2
    • 4
  • v
    Hi all, We’ve deployed Airbyte in Google GKE, and we need to whitelist Airbyte’s IP to our MySQL database. Which IP address should I be looking at? Thanks !
    m
    f
    • 3
    • 2
  • m
    👋 Can I use IRSA to connect to S3 ? I saw some issues about it but I'm not sure if it's done or not
    m
    g
    • 3
    • 5
  • m
    Hi Team, Can someone please guide I am deploying airbyte via helm chart. I want my Logs to write on S3 bucket, which parameter I should pass for AWS Credentials? Previously i did Kubernetes deployment, i can pass 
    AWS_ACCESS_KEY_ID
    and 
    AWS_SECRET_ACCESS_KEY
    to the 
    kube/overlays/stavle/.env
    it was working fine But via helm chart, I pass the following param values
    Copy code
    logs:
  accessKey:
    password: #AWS ACCESS KEY ID"
  secretKey:
    password: #AWS Secret Access KEY"
    But it seems it is not working. I am wondering am I passing the wrong parameter to the helm chart? Which one do I have to use to pass the AWS credentials in these parameters https://artifacthub.io/packages/helm/airbyte/airbyte#logs-parameters Airbyte Chart Version 0.3.5" Many Thanks
    f
    • 2
    • 1
  • f
    Hi! Does anyone deployed Airbyte using the official helm chart? I did it, but when i try to connect an external data source it doesnt work (image attached). Otherwise, when i use docker-compose it works.
    m
    • 2
    • 1
  • m
    👋 I'm trying to use IRSA to access S3 bucket but AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID can't be set to null or removed I guess it block the usage of a role Does anyone manage to use IRSA ? PS : I'm using the latest release
  • f
    @Marcos Marx (Airbyte) I am using Kustomize to deploy airbyte on GKE. Where can I adjust the workers’s resources (CPU and Memory)? And how to increase parallelism?
    m
    • 2
    • 3
  • l
    Hello guys, I'm trying to install airbyte in my k8s using helm chart, but I got this error, could anyone help me, please ? I'm using all default configs and this command `helm install --generate-name . -n airbyte`:
    Copy code
    Error: template: airbyte/templates/worker/deployment.yaml:63:23: executing "airbyte/templates/worker/deployment.yaml" at <include "airbyte.database.secret.name" .>: error calling include: template: airbyte/templates/_helpers.tpl:90:50: executing "airbyte.database.secret.name" at <.Subcharts.postgresql>: nil pointer evaluating interface {}.postgresql
helm.go:81: [debug] template: airbyte/templates/worker/deployment.yaml:63:23: executing "airbyte/templates/worker/deployment.yaml" at <include "airbyte.database.secret.name" .>: error calling include: template: airbyte/templates/_helpers.tpl:90:50: executing "airbyte.database.secret.name" at <.Subcharts.postgresql>: nil pointer evaluating interface {}.postgresql
    m
    g
    • 3
    • 6
  • a
    hi team, looking to get some advice here on deploying to k8's. We cloned the Git Airbyte repo and will add our custom connectors so we will decouple from the git repo to our private airbyte repo. given that how do we upgrade airbyte versions since git pull wont work for us here. I saw this git line in chart.yaml and just changing it to new version should work or should I change the helm version too?
  • a
    any updates on this issue https://github.com/airbytehq/airbyte/issues/1868 thanks team!
  • j
    Hey everyone! I’m trying to install airbyte through helm and got the following error: ➜ 
    airbyte helm install airbyte airbyte-oss/airbyte
    Error: INSTALLATION FAILED: parse error at (airbyte/charts/worker/templates/deployment.yaml:381): unexpected EOF
    could someone give me a ✋ , thank you!!
    h
    • 2
    • 2
  • s
    Hi folks, how are y’all doing? Quick troubleshooting question… I’m running Airbyte (AIRBYTE_VERSION: 0.35.5-alpha) on AWS EKS and I have it configure to connect to a AWS RDS Postgres instance. Things seems to start up correctly, but when I attempt to upgrade one of the destinations connectors I’m getting the following error:
    Copy code
    Internal Server Error: Get Spec job failed.
    I’ll post on the thread a more complete log error from the 
    airbyte-server
    Any tips here to help figure this out?
    u
    • 2
    • 3
  • h
    Hi! I have had a working deployment of Airbyte 0.39.14 on GKE k8s. A couple of days ago all connections started failing and everything pointed to minIO local storage (Cannot publish to S3: Storage backend has reached its minimum free disk threshold). Therefore I have deployed the (almost) latest version v0.39.39 and created a GCS bucket and configured the key in the secret file. I can confirm that the logging has been directed to GCS bucket. However, the airbyte-worker seems to be throwing errors on sync as I probably cannot get the .env file configured properly. I am using kustomize based off of stable template. Here are the relevant variables in the .env file based on docs https://docs.airbyte.com/deploying-airbyte/on-kubernetes/#configure-logs but I have found no info on the STATE_STORAGE_MINIO_ variables:
    Copy code
    # S3/Minio Log Configuration
# S3_LOG_BUCKET=airbyte-dev-logs
# S3_LOG_BUCKET_REGION=
# S3_MINIO_ENDPOINT=<http://airbyte-minio-svc:9000>
# S3_PATH_STYLE_ACCESS=true
S3_LOG_BUCKET=
S3_LOG_BUCKET_REGION=
S3_MINIO_ENDPOINT=
S3_PATH_STYLE_ACCESS=

# GCS Log Configuration
GCS_LOG_BUCKET=airbyte-logging-prod

# State Storage Configuration
STATE_STORAGE_MINIO_BUCKET_NAME=airbyte-dev-logs
STATE_STORAGE_MINIO_ENDPOINT=<http://airbyte-minio-svc:9000>
    The errors from the log are:
    Copy code
    2022-07-27 11:50:07 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable DEPLOYMENT_MODE: 'OSS'
2022-07-27 11:50:07 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable CONNECTOR_SPECIFIC_RESOURCE_DEFAULTS_ENABLED: 'false'
2022-07-27 11:50:07 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable STATE_STORAGE_MINIO_BUCKET_NAME: ''
2022-07-27 11:50:07 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable STATE_STORAGE_MINIO_ENDPOINT: ''
2022-07-27 11:50:07 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable TEMPORAL_HISTORY_RETENTION_IN_DAYS: '30'
2022-07-27 11:50:07 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable ACTIVITY_MAX_ATTEMPT: '5'
2022-07-27 11:50:07 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS: '30'
2022-07-27 11:50:07 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS: '600'
2022-07-27 11:50:07 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable TEMPORAL_CLOUD_ENABLED: 'false'
2022-07-27 11:50:07 [32mINFO[m i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):232 - Waiting for temporal server...
2022-07-27 11:50:07 [33mWARN[m i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):243 - Waiting for namespace default to be initialized in temporal...
2022-07-27 11:50:11 [32mINFO[m i.t.s.WorkflowServiceStubsImpl(<init>):188 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal-svc:7233}}
2022-07-27 11:50:16 [32mINFO[m i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):260 - Temporal namespace default initialized!
2022-07-27 11:50:16 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable TEMPORAL_CLOUD_ENABLED: 'false'
2022-07-27 11:50:16 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):968 - Using default value for environment variable TEMPORAL_CLOUD_ENABLED: 'false'
2022-07-27 11:50:16 [32mINFO[m i.a.w.t.TemporalUtils(configureTemporalNamespace):140 - Workflow execution TTL already set for namespace default. Remains unchanged as: 30 days
2022-07-27 11:50:16 [1;31mERROR[m i.a.w.WorkerApp(main):540 - Worker app failed
java.lang.IllegalArgumentException: null
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.0.1-jre.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:38) ~[io.airbyte.airbyte-config-config-models-0.39.39-alpha.jar:?]
	at io.airbyte.config.storage.MinioS3ClientFactory.validate(MinioS3ClientFactory.java:33) ~[io.airbyte.airbyte-config-config-models-0.39.39-alpha.jar:?]
	at io.airbyte.config.storage.MinioS3ClientFactory.<init>(MinioS3ClientFactory.java:27) ~[io.airbyte.airbyte-config-config-models-0.39.39-alpha.jar:?]
	at io.airbyte.workers.storage.S3DocumentStoreClient.minio(S3DocumentStoreClient.java:39) ~[io.airbyte-airbyte-workers-0.39.39-alpha.jar:?]
	at io.airbyte.workers.storage.StateClients.create(StateClients.java:21) ~[io.airbyte-airbyte-workers-0.39.39-alpha.jar:?]
	at io.airbyte.workers.WorkerApp.getContainerOrchestratorConfig(WorkerApp.java:354) ~[io.airbyte-airbyte-workers-0.39.39-alpha.jar:?]
	at io.airbyte.workers.WorkerApp.launchWorkerApp(WorkerApp.java:448) ~[io.airbyte-airbyte-workers-0.39.39-alpha.jar:?]
	at io.airbyte.workers.WorkerApp.main(WorkerApp.java:537) [io.airbyte-airbyte-workers-0.39.39-alpha.jar:?]
    If I set the STATE_STORAGE_MINIO_ variables to empty then I also get errors similar to this:
    Copy code
    022-07-27 11:32:43 [32mINFO[m i.a.w.p.AsyncOrchestratorPodProcess(copyFilesToKubeConfigVolumeMain):372 - Waiting for kubectl cp to complete
2022-07-27 11:32:43 [32mINFO[m i.a.w.p.AsyncOrchestratorPodProcess(copyFilesToKubeConfigVolumeMain):379 - kubectl cp complete, closing process
2022-07-27 11:32:48 [32mINFO[m i.a.w.t.TemporalUtils(withBackgroundHeartbeat):322 - Stopping temporal heartbeating...
2022-07-27 11:32:48 [32mINFO[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):158 - Completing future exceptionally...
java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:320) ~[io.airbyte-airbyte-workers-0.39.39-alpha.jar:?]
	at io.airbyte.workers.temporal.sync.LauncherWorker.run(LauncherWorker.java:90) ~[io.airbyte-airbyte-workers-0.39.39-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.39.39-alpha.jar:?]
	at java.lang.Thread.run(Thread.java:1589) [?:?]
Caused by: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
	at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:181) ~[io.airbyte-airbyte-workers-0.39.39-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:315) ~[io.airbyte-airbyte-workers-0.39.39-alpha.jar:?]
	... 3 more
Caused by: io.airbyte.workers.exception.WorkerException: Non-zero exit code!
	at io.airbyte.workers.temporal.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:165) ~[io.airbyte-airbyte-workers-0.39.39-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:315) ~[io.airbyte-airbyte-workers-0.39.39-alpha.jar:?]
	... 3 more
    • 1
    • 1
  • n
    Hello Team, We are deploying Airbyte on multiple namespaces and use one cloud SQL instance as an external database for the metadata. I have created a separate database for each instance. Since Airbyte creates a temporal database automatically, I am getting errors. Is there a way to change the temporal database name to something else through helm charts or kustomize? Thanks for your help.
    u
    • 2
    • 4
  • a
    Hi, I followed https://docs.airbyte.com/deploying-airbyte/on-kubernetes/ to deploy airbyte into our kubernetes cluster by running
    Copy code
    kubectl apply -k kube/overlays/stable
    I am trying to create a Postgres source by connecting to the Postgres instance deployed as airbyte db, but got 
    Check Connection Failed!
    , I checked airbyte-server log and got
    Copy code
    2022-08-02 19:27:06 INFO i.a.s.RequestLogger(filter):112 - REQ 172.16.41.115 POST 200 /api/v1/scheduler/sources/check_connection - {"connectionConfiguration":"REDACTED","sourceDefinitionId":"decd338e-5647-4c0b-adf4-da0e75f5a750"}
    which made me feel like the connection is established successfully, what should I check next to proceed?
    k
    • 2
    • 3
  • d
    Copy code
    ➜ kc describe clusterrole airbyte-postgres-access
Name:         airbyte-postgres-access
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  pods       []                 []              [get watch list]
  services   []                 []              [get watch list]
  • d
    Copy code
    ➜ kc describe clusterrolebinding airbyte-postgres-access
Name:         airbyte-postgres-access
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  airbyte-postgres-access
Subjects:
  Kind            Name           Namespace
  ----            ----           ---------
  ServiceAccount  airbyte-admin  airbyte
  • d
    so to work around, i deployed the airbyte resources in the same namespace where a postgres instance lives.
  • d
    still not able to connect.
  • d
    here's the postgres service:
  • d
    Copy code
    ✗  kc -n bridge describe svc postgres
Name:              postgres
Namespace:         bridge
Labels:            <http://kustomize.toolkit.fluxcd.io/name=bridge|kustomize.toolkit.fluxcd.io/name=bridge>
                   <http://kustomize.toolkit.fluxcd.io/namespace=flux-system|kustomize.toolkit.fluxcd.io/namespace=flux-system>
Annotations:       <none>
Selector:          app=postgres
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.100.116.241
IPs:               10.100.116.241
Port:              postgres  5432/TCP
TargetPort:        5432/TCP
Endpoints:         192.168.73.88:5432
Session Affinity:  None
Events:            <none>
  • d
    it's up
    Copy code
    ✗  kc -n bridge get ep postgres -o wide
NAME       ENDPOINTS            AGE
postgres   192.168.73.88:5432   168d
  • d
    in airbyte webapp, i've tried the following for hostname: 
    postgres
    , 
    postgres.svc.cluster.local
    , 
    postgres.bridge.svc.cluster.local
    , and even direct ip 
    192.168.73.88
    -- none of them is able to connect.
  • d
    airbyte worker logs always indicate:
    Copy code
    2022-08-05 19:55:13 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-05 19:55:13 INFO i.a.i.s.r.AbstractDbSource(check):84 - Exception while checking connection:
Log4j2Appender says: 2022-08-05 19:55:13 INFO i.a.i.s.r.AbstractDbSource(check):84 - Exception while checking connection:
2022-08-05 19:55:13 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 60001ms.
Log4j2Appender says: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 60001ms.
    a
    • 2
    • 1
  • g
    Hi All, I am trying to deploy Airbyte on AWS EKS using the helm chart and by connecting to a Postgres RDS instance instead of running it in a container. The bootloader container times out and I don't get any logs either. I tested for connectivity from the nodepool, and I can interact with the RDS instance allright. It must be something AWS-specific that I am missing as the helm chart worked fine on minikube. Is there a step by step guide I can follow? The most helpful would be an idotproof explanation on how do I get from an empty AWS account to a working Airbyte on EKS + RDS setup atep by step. It could also be used to update the documentation online. Thanks in advance!
  • g
    Hi Davis, Do you have some infrastructure as code you could share to get a working EKS-deployment?
  • d
    Hi all , in the pod-sweeper service i edited the script so now it will delete completed state pods every two minuteds , and failed state pods every 15 minutes dose it wipe the logs of that job with that ?
  • g
    Hi all, Trying to upgrade to the 1.0.0 charts, now published at 
    <https://airbytehq.github.io/helm-charts/>
    ... and I'm getting this error:
    Copy code
    Helm upgrade failed: template: airbyte/charts/server/templates/deployment.yaml:240:15: executing "airbyte/charts/server/templates/deployment.yaml" at <eq .Values.deploymentMode "oss">: error calling eq: incompatible types for comparison
    Any thoughts?
    m
    • 2
    • 5
12Latest