Hey, I'm trying downgrade Airbyte using helm and

values.yml

file. What should be change there in order to deploy Airbyte with the desired version (in my case 0.40.27)? I found

version

section but it does not help at all, Airbyte is deployed with the most recent version

I face this error when I Airbyte tries to git clone the dbt repo for transformation: 2023-06-23 041948 normalization > fatal: unable to access 'xxxxxxxxxxxxxxxxxxxxxx.git/': server certificate verification failed. CAfile: none CRLfile: none It's a GitLab repo and it uses self signed certificate. How can I trust the self signed certificate to make the git clone command work? Or are there any alternatives to clone repos that use self-signed certificates for https?

🔥 Community Office Hours starts in 15 minutes 🔥 At 1pm PDT click here to join us on Zoom!

Could you let me know how to build repor-metrics image as I am facing java cast issue and i want to fix it however, there is no document how to build the image from source. it would be helpful if you can shed some light on this ?

Does anyone have a nice dashboard example of otels collector metrics?

Hi everyone - I'm trying to deploy on EC2 using a deployment script that initiates the run-ab-platform.sh file - but I get this error. Anyone seen this before?

Error response from daemon: manifest for airbyte/proxy:dev not found: manifest unknown: manifest unknown

Keep getting this error when I try to start airbyte on an EC2 instance that uses the Amazon Linux AMI. Docker Compose version 2.19.0 is what I have installed. How do I get airbyte to recognize the docker compose installation?

🔥 Community Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 1pm PDT click here to join us on Zoom!

Hi, I am trying to set a pipeline between Mysql and Redshift when I enable base normalisation I get bellow error. Airbyte version is 0.44.12 which is setup in EC2 instance ARM64 machine type

2023-06-22 09:30:22 INFO i.a.w.p.DockerProcessFactory(create):192 - Preparing command: docker run --rm --init -i -w /data/196/2/normalize --log-driver none --name normalization-redshift-normalize-196-2-miizf --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.44.12 airbyte/normalization-redshift:0.4.3 run --integration-type redshift --config destination_config.json --catalog destination_catalog.json
2023-06-22 09:30:22 normalization > [FATAL tini (7)] exec /airbyte/entrypoint.sh failed: Exec format error
2023-06-22 09:30:22 INFO i.a.w.n.DefaultNormalizationRunner(close):196 - Terminating normalization process...
2023-06-22 09:30:22 ERROR i.a.w.g.DefaultNormalizationWorker(run):86 - Normalization failed for job 196.
io.airbyte.workers.exception.WorkerException: Normalization process did not terminate normally (exit code: 1)
	at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:207) ~[io.airbyte-airbyte-commons-worker-0.44.12.jar:?]
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:84) ~[io.airbyte-airbyte-commons-worker-0.44.12.jar:?]
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:37) ~[io.airbyte-airbyte-commons-worker-0.44.12.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.44.12.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

If I downgrade the version of a source that has live connections using it, does the downgrade propagate across all the connections?

I need help. Why is my “_airbyte_data” field now “{}” when it used to have data before upgrading to 0.50.4?

hello, I am trying to install Airbyte using official helm chart values provided. I am seeing that the bottloader is in error status:

shubhamsingh@shubhams-MacBook-Pro airbyte % kubectl get pods                                                            
NAME                         READY   STATUS    RESTARTS   AGE
airbyte-airbyte-bootloader   0/1     Error     0          8m7s
airbyte-db-0                 1/1     Running   0          8m8s
shubhamsingh@shubhams-MacBook-Pro airbyte %

When i try to check the bootloader logs it shows the following:

2023-06-27 06:44:47 ERROR i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):78 - Failed to verify database connection.
org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-2)
	at org.jooq_3.17.8.POSTGRES.debug(Unknown Source) ~[?:?]
	at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:90) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.DefaultExecuteContext.connection(DefaultExecuteContext.java:560) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:296) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:301) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.AbstractResultQuery.fetchLazyNonAutoClosing(AbstractResultQuery.java:322) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.SelectImpl.fetchLazyNonAutoClosing(SelectImpl.java:2861) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.ResultQueryTrait.fetchOne(ResultQueryTrait.java:509) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.Tools.attach(Tools.java:1526) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.DefaultDSLContext.fetchOne(DefaultDSLContext.java:4662) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.DefaultDSLContext.lambda$fetchValue$49(DefaultDSLContext.java:4682) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.Tools.attach(Tools.java:1526) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.DefaultDSLContext.fetchValue(DefaultDSLContext.java:4682) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.DefaultDSLContext.fetchValue(DefaultDSLContext.java:4696) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.DefaultDSLContext.fetchExists(DefaultDSLContext.java:4778) ~[jooq-3.17.8.jar:?]
	at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$0(DatabaseAvailabilityCheck.java:76) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
	at io.airbyte.db.Database.query(Database.java:23) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
	at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$1(DatabaseAvailabilityCheck.java:76) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
	at io.airbyte.db.check.DatabaseAvailabilityCheck.check(DatabaseAvailabilityCheck.java:47) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
	at io.airbyte.db.init.DatabaseInitializer.initialize(DatabaseInitializer.java:45) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
	at io.airbyte.bootloader.Bootloader.initializeDatabases(Bootloader.java:200) ~[io.airbyte-airbyte-bootloader-0.50.1.jar:?]
	at io.airbyte.bootloader.Bootloader.load(Bootloader.java:101) ~[io.airbyte-airbyte-bootloader-0.50.1.jar:?]
	at io.airbyte.bootloader.Application.main(Application.java:22) ~[io.airbyte-airbyte-bootloader-0.50.1.jar:?]
Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 - Connection is not available, request timed out after 30000ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) ~[HikariCP-5.0.1.jar:?]
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181) ~[HikariCP-5.0.1.jar:?]
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146) ~[HikariCP-5.0.1.jar:?]
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) ~[HikariCP-5.0.1.jar:?]
	at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:87) ~[jooq-3.17.8.jar:?]
	... 21 more
Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "postgres"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:693

From the EKS nodes i was able to confirm that the RDS deployed is reachable and the RDS and EKS are in the same VPC. Can you tell me what am i missing here.? Attaching the values file for your reference. I am using S3 and AWS RDS for storage.

Hello, yesterday I deployed Airbyte via GKE using Helm. Here's the documentation link: https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm/. We used the Default deployment. The deployment was successful. However, Airbyte can be accessed without a username and password. Could you provide me with guidelines on how to configure the username and password when accessing Airbyte via GKE? Thank you.

Hello Team, I am deploying airbyte on EKS for the first time..The helm installation got passed suceffully.. but i am seeing that the server and worker pods are in CrashLoopBackOff :

airbyte-airbyte-bootloader                          0/1     Completed          0                5m33s
airbyte-connector-builder-server-58b4d88c59-fj758   1/1     Running            0                61m
airbyte-cron-686c7c8b7-bbmhd                        1/1     Running            0                61m
airbyte-db-0                                        1/1     Running            0                150m
airbyte-pod-sweeper-pod-sweeper-75d4cc5d75-snggp    1/1     Running            0                61m
airbyte-server-7f7f97f569-f766x                     0/1     CrashLoopBackOff   15 (4m18s ago)   61m
airbyte-temporal-6dfc6f69cb-m9x6p                   1/1     Running            0                61m
airbyte-webapp-bc78bcc77-q9ms5                      0/1     Running            0                61m
airbyte-worker-7c8bd697d8-n7ltg                     0/1     CrashLoopBackOff   15 (3m57s ago)   61m
airbyte-worker-7c8bd697d8-pdn9q                     0/1     CrashLoopBackOff   15 (4m16s ago)   61m

On checking the logs i am seeing the following:

2023-06-27 09:08:28 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable METRIC_CLIENT: ''
2023-06-27 09:08:28 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable METRIC_CLIENT: ''
2023-06-27 09:08:28 WARN i.a.m.l.MetricClientFactory(initialize):74 - MetricClient was not recognized or not provided. Accepted values are `datadog` or `otel`. 
2023-06-27 09:08:28 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):248 - Waiting for temporal server...
2023-06-27 09:08:28 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):259 - Waiting for namespace default to be initialized in temporal...
2023-06-27 09:08:31 INFO i.t.s.WorkflowServiceStubsImpl(<init>):86 - Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal:7233}}
2023-06-27 09:08:36 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):276 - Temporal namespace default initialized!
2023-06-27 09:08:36 INFO i.a.w.ApplicationInitializer(initializeCommonDependencies):191 - Initializing common worker dependencies.
2023-06-27 09:08:36 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: null
java.lang.IllegalArgumentException: null
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:36) ~[io.airbyte.airbyte-config-config-models-0.50.1.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.validate(DefaultS3ClientFactory.java:31) ~[io.airbyte.airbyte-config-config-models-0.50.1.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.<init>(DefaultS3ClientFactory.java:24) ~[io.airbyte.airbyte-config-config-models-0.50.1.jar:?]
	at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:51) ~[io.airbyte.airbyte-config-config-models-0.50.1.jar:?]
	at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:226) ~[io.airbyte.airbyte-config-config-models-0.50.1.jar:?]
	at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:213) ~[io.airbyte.airbyte-config-config-models-0.50.1.jar:?]
	at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:194) ~[io.airbyte-airbyte-workers-0.50.1.jar:?]
	at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:161) ~[io.airbyte-airbyte-workers-0.50.1.jar:?]
	at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:63) ~[io.airbyte-airbyte-workers-0.50.1.jar:?]
	at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262) ~[micronaut-inject-

Can someone let me know what can be causing this please.?

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom!

@Sunny Hashmi - do you have more information on metrics tracking or this article https://airbyte.com/blog/airbyte-monitoring-with-dbt-and-metabase?

Hi there, been looking to quickly do a proof of concept for my company and run airbyte locally, create some test sources and connections, create a test custom source, etc. I was wondering if there was a simpler way to use airbyte than start the whole lot of containers in the compose file:

init bootloader db worker server webapp airbyte-temporal airbyte-cron airbyte-connector-builder-server airbyte-proxy

Maybe only some of them are absolutely necessary, but i'm not sure. Should airbyte work just fine with only

db worker server webapp

?

Looking to setup

otel-collector

to make metrics available in Prometheus. Does anyone know how/why this was out of date? And has it been replaced by anything? https://github.com/airbytehq/airbyte/pull/27637

Is there any good documentation around the airbyte otel metrics available? And does anyone have a nice dashboard in grafana to share?

Hi Team, Looking to report a bug in the latest helm release

0.45.40

When deploying the latest helm chart with app

0.50.5

I get this error on my eks cluster:

POST 500 /api/v1/web_backend/check_updates

This error shows an "Oops something went wrong message" on the Connections, Source, and Destinations, and Settings page. I've eliminated networking issues as when I revert back to helm chart

0.45.38

I get a functioning webapp deployment with no issues. Where can I report his bug to the airbyte-platform team?

hi, team I created an issue yesterday, can anyone check it out? https://github.com/airbytehq/airbyte/issues/27750 Thanks

Hey. https://github.com/airbytehq/airbyte/pull/27637 removed documentation about Airbyte OTEL (open telemetry) metrics. Is there any more up-toda-te mechanism for extracting metrics from Airbyte?

Hi team, Is it possible to set the webhook for notification with the helm chart ?

(I’m having problems with memory. My Airbyte is deploied on AWS EC2)

Is there a way to set Pod Disruption Budget on the helm chart

hello, I have a new airbyte setup running on AWS EKS, when i am trying to setup google search console as a source i am getting the following error as shown in the image.. from the EKS nodes i am able to connect to the outside internet without any issues. Can someone tell me what can be causing this.?

is it possible to add to kubernetes chart the nodeSelector and tolerations for airbyte-db.yaml and minio.yaml ?

Hello. Has anyone faced issues • Migrating from 0.40.7 to 0.40.26 • Using docker compose I am currently getting flyway migration exceptions (see file) • Is there a configuration for cascade updates of the key

(workspace_id, actor_definition_id)

?

I'm trying to deploy AIrbyte on EC2 - but I receive this error when I try to run the shell script. Anyone seen (and soved) this before?

error response from daemon: failed to initialize logging driver: template: log-tag:1:25: executing "log-tag" at <':'>: expected string; found ':'

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom octavia loves