We’re having some issues with the Airbyte worker, as detailed in issue #24269, and am curious if there will be movement on resolving it. I am forced to run multiple workers with aggressive memory limits so they are cycled frequently, to work around the deficiency.

Hi All, I'm testing the waters with airbyte. I downloaded the docker image and spun up the containers. I see there's a

temporal

container running but I'm unable to reach it (it's not exposing any port). Is this normal behavior? What are the steps to test the temporal integration?

Hi, I followed the steps by the documentation and am pretty sure everything is set, and I could open http://localhost.8000 yesterday and this morning. But it is not working since then, and keeping receiving the error 'channel 2: open failed: connect failed: Connection refused'. I googled and troubleshooted it , still failed.https://docs.airbyte.com/deploying-airbyte/on-aws-ec2/

Hey, just recently added a proxy to our Airbyte webapp so it's secure but we now saw that the webapp docker image is still hosted on " 0.0.0.0/Port" instead of 127.0.0.1. I have looked around in the Configs of the docker-build etc but I have not seen where I can change that IP adress. Could you give me some pointers as to where to change this? Much appreciated.

Wanted to post this in the right channel! Was curious if anyone had advice about this?

Is there some reason why Airbyte team decided to move forward with Helm and not Kustomize? I probably missed the memo when the decision was made, so if anyone can send me a link to any docs / PRs that explain this, I will be thankful.

Hi everyone, I’m about to upgrade from 0.34 to 0.40. How can I get reassurance that my existing connections and sync states will not be affected? I’ve thought of pausing all syncs before deleting my deployments and performing the upgrade, but is there anything else that can I do? (More context: We’re using an external DB (postgres), so theoretically… so as long as airbyte is still pointed to this external DB, everything should be fine?)

Hello, I've followed the steps in the documentation https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine/ but for some reason am unable to get the local db setup working upon running

docker compose up -d

password authentication failed for user "docker"

and

Role "docker" does not exist.

This is especially strange because I have gotten the same exact setup working in another vm instance, I have verified that there are no differences between

flags.yml

,

docker-compose.yaml

, and

.env

in the two instances, yet it is still failing on one. Any suggestions? In the failing instance, I did have to take extra steps to allow

docker compose up -d

to be run without sudo, whereas I did not have to do this in the first so that could be a clue as to what is going wrong.

Hello. Is it possible to set the domain of Airbyte (example: airbyte-prod.mycompany.com) instead of

localhost

in Slack-sync-notifications? The documentation doesn't mention about it.

Hi. I've deployed Airbyte on EKS (under a namespace

airbyte

) and working with API to create multiple workspaces. I believe these workspaces are mapped to the same Kubernetes namespace (as other workspaces). I wanted to know if there's a way to map Airbyte workspace to a Kubernetes namespace. My goal is to support multi-tenancy.

Hi, I've deployed Airbyte on AWS. If it is disconnected and I re-connect to Airbyte, I need to create first connection, how can I avoid it?

I'm running into a situation of ballooning memory usage on connector syncs. It had initially been causing my instance to lock up and get replaced by the auto-scale group and now seems to be causing an individual sync to fail. It's running the MySQL source and S3 Glue destination.

I'm deploying Airbyte via Helm with an external database (have already done this without issue in 4 other environments) and this time my

temporal

pod keeps crashing with the error

2023-05-04T17:27:33.813Z	ERROR	Unable to create SQL database.	{"error": "pq: database \"temporal\" already exists", "logging-call-at": "handler.go:97"}
2023/05/04 17:27:33 Loading config; env=docker,zone=,configDir=config
2023/05/04 17:27:33 Loading config files=[config/docker.yaml]
Unable to load configuration: config file corrupted: yaml: line 17: found unknown escape character.

I'm deploying chart 0.45.11 / airbyte version 0.44.0

Hello Everyone, I’m facing exactly the same issue as @Ben Konicek. I’m running helm chart 0.45.8 and trying to connect to GCP SQL.

+ temporal-sql-tool --plugin postgres --ep 172.27.5.3 -u airbyte -p 5432 create --db temporal                                                                                                                            2023-05-04T18:47:04.590Z    ERROR    Unable to create SQL database.    {"error": "pq: database \"temporal\" already exists", "logging-call-at": "handler.go:97"}                                                         ││ 2023/05/04 18:47:04 Loading config; env=docker,zone=,configDir=config                                                                                                                                                    2023/05/04 18:47:04 Loading config files=[config/docker.yaml]                                                                                                                                                            Unable to load configuration: config file corrupted: yaml: line 29: did not find expected key.                                                                                                                           Stream closed EOF for airbyte/airbyte-temporal-77846d4b7d-b54ks (airbyte-temporal)

Hello Everyone, I am deploying Airbyte via Helm on Azure Kubernetes. I have setup an external database, but I am wondering about Minio. Is anything other than logs are stored in Minio that should be persisted more long term? I have found mixed answers regarding this on the forum and elsewere, and I was hoping someone here could clear things up for me. I want to be able to completely delete airbyte from the cluster (or even delete the cluster itself) and re-deploy it without losing any airbyte configuraton, so it can more or less pickup where it left off again.

Hi there, I have opened a Issue related to upgrading the dockers. If you have some time, would you mind taking a look at it and giving me your feedback? I would really appreciate it! Thanks!

Hi there, I am still running into this issue deploying on GKE with Helm and I'm unable to proceed with deploying Airbyte. https://airbytehq.slack.com/archives/C01AHCD885S/p1683221371096399 I've tried deploying it manually, recreating the entire database, etc and the

temporal

pod continues to log the error "temporal database already exists" and fail to configure the docker config file. As far as I can tell,

temporal

is successfully creating the temporal DB, so I'm at a loss for why it is then erroring out saying it failed to create it.

Hi Airbyte Community 👋 , I'm a bit frustrated by the number of solutions available and would like to have some feedback / help I saw a pretty good diagram of Modern Data Architecture with many.. MANY solutions I'm on AWS on currently have a few ETL process build on my own with lambda functions and would like to scale the entire process with real ETL solution I would like to know if I can mix Airbyte with some AWS solutions and maybe some other solutions in order to achieve a pretty good base architecture that can scale by adding more tools I would like to start with Airbyte as ETL base product with dbt for more advanced transformation Is a workflow manager like airflow necessary to start? Can I use it with lake formation for example as an alternative to proposed solutions in Lakehouse in this diagram? Thanks! P.S: is there a design help channel? This question is more about design than deployment

is there a way to deploy app version 0.44.3 via helm yet?

Hi friends, I’m looking to contribute to a source connector, would be my first time, is there any good documentation or is there anyone here who could help me undersatnd how I might do something like this? For some context, I’m looking to add more streams to the Auth0 connector

Hi all, while deploying airbyte on Kubernetes using Helm, I am getting the following error:

Error: UPGRADE FAILED: template: airbyte/charts/webapp/templates/secrets.yaml:14:60: executing "airbyte/charts/webapp/templates/secrets.yaml" at <.Values.global.secrets>: wrong type for value; expected map[string]interface {}; got interface {}

I see that there are some configs missing in values.yaml could someone show me how a working values.yaml looks like! thanks in advance.

Hi all Looking to deploy airbyte on prem and wanted to ask if the docker container running airbyte inherits the VPN of the host? I need to read data from a "local" database on the same network and would like to get some input on the possibility of doing so?

Hi! I've got Airbyte deployed on our VPC in AWS, and I've allocated a

t2.xlarge

instance, but during a sync the instance starts to fail connectivity checks, and eventually becomes unreachable. This is happening consistently and seems to be airbyte-related since I'm not running anything else on the box. Any clues or places I should look for support on something like this?

Hello everyone, i'm trying to deploy Airbyte on Kubernetes with HELM using GCP, after the deployement i don't know how to access to Airbyte UI. where can i find the port and the IP that redirect me to the webapp ?

Hello everyone, I was able to deploy Airbyte using docker and connect to it with Octavia CLI as well as the python SDK. I noticed that the python SDK can't communicate with self-hosted Airbyte. Octavia works well but it uses a different API. The python SDK seems to have the correct endpoint urls and models for Airbyte Cloud while Octavia has the correct endpoint urls and models for Airbyte Open Source. I was able to write some custom scripts by importing the API client from the Octavia docker image. Are there any plans to make the API client from Octavia available on pypi or to add Airbyte Open Source support to the python SDK? I'd be happy to help wherever I can. I would have forked the python SDK and updated it for open source but I don't know how the auto generation works.

Hey team, I ran this command but I'm not sure how to close the SSH tunnel. Any pointers?

gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME -- -L 8000:localhost:8000 -N -f

Hi I just followed the instructions on the documentation page to install airbyte locally on windows. When I run this command bash run-ab-platform.sh I get this error run-ab-platform.sh: line 2: $'\r': command not found run-ab-platform.sh: line 7: $'\r': command not found any ideas on how to address this?

I am currently working on testing airbyte internally using the helm-chart. Firstly I tried with postgresql.enabled = true which starts a postgres container. It worked in that case and I was able to access the webapp by port-forwarding. Then I tried using a dedicated postgres instance as data source for airbyte. In that case, I am facing some issue due to which temporal pod is not able to come up. Here is the log: ___________________________________ Waiting for Temporal server to start... + tctl cluster health + grep SERVING Waiting for Temporal server to start... + echo 'Waiting for Temporal server to start...' + sleep 1 _Unable to start server. Error: sql schema version compatibility check failed: pq: no pg_hba.conf entry for host "10.6.103.216", user "airbyte_admin_user", database "temporal", no encryption_ ___________________________________ Here is my values.yaml content. _______________________________________________________ postgresql: enabled: false temporal: extraEnv: _- name: "SKIP_SCHEMA_SETUP"_ value: "true" _- name: "SQL_TLS"_ value: "true" _- name: "SQL_TLS_DISABLE_HOST_VERIFICATION"_ value: "true" externalDatabase: _host: {{ $airbyte_db_application_secret.data.pg_host }}_ _user: {{ $airbyte_db_application_secret.data.pg_user }}_ _password: {{ $airbyte_db_application_secret.data.pg_password }}_ _database: {{ $airbyte_db_application_secret.data.db }}_ _port: {{ $airbyte_db_application_secret.data.pg_port }}_ _jdbcUrl: "jdbcpostgresql//{{ $airbyte_db_application_secret.data.pg_host }}:{{ $airbyte_db_application_secret.data.pg_port }}/{{ $airbyte_db_application_secret.data.db }}?stringtype=unspecified"_ _____________________________________________________________ Not able to figure out what's the problem. Please advise.

Hello - I have installed docker according to the instructions. I am running on Windows 11. I get this error message. docker compose v2 not found! please install docker compose!

Hi, me again. Maybe I'm missing some docs but it took quite a lot of research (and a little help from Speakeasy) to create demo that interacts with the API for a self-hosted Airbyte. https://github.com/aroxby-wayscript/airbyte-python