Hello could you help me on this ? https://discuss.airbyte.io/t/airbyte-server-not-healthy-exception-with-prefect-airbyte-starting-or-api-call-with-0-40-28/3792/8 Still stuck with airbyte/helm 0.40.27

Hi - I am ingesting data from a mongodb collection. I used raw json as normalization because we have hundreds of nested fields and the normalized tablular data option just doesnt work. Is there an easy way to collect only specific fields from the mongo collection? Ideally without having to modify the underlying mongo connector?

❤️ Airbyte, but handling schema changes is a huge 👎

Hello team 👋 I set up slack notifications yesterday in order to get alerted when a sync fails. However, I also keep getting alerts on schema changes 😕 Is there a way for me to turn these schema change alerts off?

In a low-code connector, is it possible to implement a CustomTransformation that takes the stream schema as an input parameter? If so, what would that look like in the manifest.yaml?

I had an airbyte instance lock up today because it ran out of space on the filesystem. It looks like the

airbyte_workspace

volume is responsible - is there anything I should/can do to clean that up? I'm running on 0.39.37-alpha (yeah, I should upgrade, but I want to make sure I don't run out of space again first). I'm limping along with a small amount of free space, but I'm afraid the next sync that happens might fill it up again. Thoughts? Thanks!

I upgraded (git pull) Airbyte yesterday. All day today I've been getting this screen when clicking "+ Source" . Rebooting Airbyte fixes it for a little while.

Hey all, I see Airbyte OSS has a DataDog connector, but our instance does not have it, do we need to fork the main repo again and upgrade to bring in? Or is there another way to bring this connector into play?

Hello! I am trying get a tickets endpoint from zendesk Support and send to Big Query, but i receive a error in normalization I am using a airbyte version 0.40.32 and Zendesk Support version 0.2.25 (editado)

Having trouble with the final step connecting to localhost via Google Cloud terminal on my machine for the Google Cloud Platform integration.

gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME -- -L 8000:localhost:8000 -N -f

https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine/ I've tried killing that port and trying again, and trying other ports. First time there is no error (after killing the port) but then when I run the command again I get this error again.

bind [127.0.0.1]:8000: Address already in use
channel_setup_fwd_listener_tcpip: cannot listen to port: 8000
Could not request local forwarding.

Greetings. I have Airbyte deployed to an EKS cluster in AWS. I find that I need to manually cycle the deployments every week or so, else new pods aren’t able to be placed. This cluster’s nodes do not have any sort of horizontal pod autoscaling configured, and I’m limited to 2 nodes. Does Airbyte try to do its own scaling in a way that it would get stuck if the cluster didn’t allow it to grow? Has anyone else run into periodic pod placement failures of this nature?

How do I customize airbyte deploy using helm? I had been using my own

values.yaml

file, but now the

charts

directory has been removed from the repo. I can no longer use the sample values file for reference.

Hi, Have a basic question. When Airbyte is deployed in Kubernetes, if i want to create a connection that pulls 200 tables, will it spin up a pod per table to ingest the data or do i have to create multiple connections with subset of tables and parallelize it?

Hi everyone. I'm working on a project with my team using the open source code. When I cloned the project down from GitHub a few months ago, I was able to run it with

docker compose up

successfully. However, my newer teammates pulled down the repo and are getting this error when running the same docker command:

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"

I created a new folder on my local machine and was able to reproduce the error. Is there a known bug causing this? When I first cloned the project, I did not do anything special to run the docker command.

Hi everyone, I want to automate some tasks after airbyte sync jobs run and transformations are done. Does airbyte open source and/or cloud send webhook for such purposes?

hey all – i'm looking to use the Gong connector to track call interactions. Gong exposes information about call participants in an endpoint that the connector is currently not using. i've never coded at the level that would be needed for this, but wanted to see if there's a way to start tinkering

Hi all! Am new to airbyte! Just wanna get your suggestions on the following, 1. Is Kubernetes the best place to deploy airbyte? 2. If so, I see two options for the same, a. Kubernetes using Kustomize - Deprecated as per doc. b. Kubernetes using Helm - Beta as per doc. Is it okay to go ahead and use Helm even though it's in beta? Thanks in advance!

Deployed on k8s using helm chart version 0.41.0. Webapp pod is looping in back-off restarting showing the following error:

[emerg] 1#1: host not found in upstream "$CONNECTOR_BUILDER_API_HOST" in /etc/nginx/conf.d/default.conf:6

How do I get past this error? I’ve tried several other helm chart versions which show other errors. If there a stable helm chart version that I can use?

Hi everybody! I am new in Airbute and I need to sync data from Hubspot to Postgres. I did initial sync using my local machine with docker-compose and now I have 2 problems 1. How I can setup this sync programmatically using my own docker-compose configuration? There is Postgres DB there so I want to pull all the data there. 2. I see some unwanted things in resulting DB: no assiciation table, bad type for IDs e t.c. How can I tune sync process?

Hi all, I´m stuck on the Facebook MArketing source. I´ve created the app and I manage to call it through airbyte and sync some data. I know I need to ask for advanced access and for that to have the app reviewed. BUT I don´t know which credentials I should give Facebook to review the app, as I can´t give them credentials to access my clients Airbyte. Any ideas which credentials I should give?

Hi All, I am trying to sync the data from source mongo to s3 destination with parquet format as file format the sync is failing with below error, I am using airbyte open source version 0.39.35 spinned up in an EC2 instance using docker compose.

2023-02-28 12:54:17 - Additional Failure Information: java.lang.NullPointerException: Array contains a null element at 0. Set parquet.avro.write-old-list-structure=false to turn on support for arrays with null elements.

I am looking to understand where can I change this parameter

parquet.avro.write-old-list-structure=false

in the destination connector code or any alternate approach to resolve this issue

Is there a way to parallelize a connection for example from one MySQL db to another, maybe from the command line, so that each table is uploaded in parallel? It's fine if the approach sacrifices a bit of consistency. Currently using airbyte a single connection takes +4 hours.

hey guys i am having this error while setting up google ads connector

Hey! I am facing this issue in the process of local deployment. Can you please help resolve.

Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /Users/richar/IdeaProjects/backbase/airbyte/flags.yml

Hi. A lot a questions related to this topic already, but couldn't find a good answer. I've developed a (working) custom connector and pushed the Docker image to Google Artifact Registry. Now I've hosted Airbyte on a GCE instance. I've authenticated to the Artifact registry on the GCE instance as described here (https://docs.airbyte.com/operator-guides/using-custom-connectors/#for-docker-compose-airbyte-deployments) using gcloud. However, I can't seem to pull the image directly from the Airbyte interface when I enter the Docker repo. It does seem to work when I add a custom connector using a public Docker Hub repository. I manually pulled the Docker image from the private Artifact registry on the GCE instance, and after that I could add it in the interface. Is this expected behavior? Or is there a way I can pull the image directly from a private repository through the Airbyte interface?

I’m looking to load data from AWS DocumentDB using the mongodb connector. By default, DocumentDB requires you use a certificate to connect, which seems to be addressed in this PR here. Any plans on merging this to main soon? Or should I hack it into my code 😞

Hi everyone, does someone have a tutorial about Incremental sync on MongoDB source connector: We don't intend to update the schema frequently, but it's a big database and it's a shame we have to overwrite ALL THE DATA every day (and it costs a lot). We have

createdAt

and

updatedAt

fields for every collection, I guess we should use

updatedAt

as a cursor, am I right ?

those dbt are from airbyte. My database is dying. 😭

2023-03-03 16:46:02 normalization > 277 of 696 OK created incremental model airbyte.facebook_us_ads_insights_region_scd..................................... [INSERT 0 14958 in 485.76s

I feel like it shouldn’t be this slow 🤔 (And also that it didn’t use to be last week..) Can I add indexes or something?

Hi, I have been running Airbyte Open Source on Google Compute Engine and something weird happen few days ago. I manually ran sync, where the first one “failed” and the sync successful on its second try. Both failed and successful syncs ran for 4 hours each, but for some reason even the failed sync seem to have printed something to BigQuery. And the Slack message sent also indicated the total ran time was around 8 hours. Does anyone know what happened? or had something like this happen before? The log file attached is also for both failed and successful syncs. For some reason they were all printed together. Notice how the BigQuery tables contains same information on the same day, but in different hours. The 18 hours one was the ‘failed’ one.