Does anyone know how real-time Airbyte can get? Is it possible to stream or can we only use CDC/Incremental loads

Hi everyone, Dit anyone happen to run into this problem? If so, how did you resolve it? I file like there is a configuration sample file missing in airbyte

config/dynamicconfig/development.yaml

https://github.com/airbytehq/airbyte/issues/24581

Hi all, how can we debug the root cause of

Incremental | Deduped + history

sync not working for a shopify connection to postgres. We can clearly see that even though the order is not modified its inserted into the raw table:

select array_agg(aro."_airbyte_data"->>'updated_at') as updated_at_arr,  array_agg(aro."_airbyte_emitted_at") as emitted_at_arr, count(*)   from 
client_shopify."_airbyte_raw_orders" aro
group by aro."_airbyte_data"->>'id' order by 3 desc;

any pointers highly appreciated!

Does anyone from airbyte team really care about the questions ?

Hi all, recently updated to 0.42.0 and am getting the following response with a Postgres connector that was previously working on 0.41.0:

The connection tests failed.

Internal Server Error: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 1 path $

Hello Airbyte community octavia wave My PostgreSQL sync take more than 32 hours to discover the source db and initialize before actually starting to replicate data 🤔 Has anyone faced the issue? I've described it more thoroughly in this Github issue Thanks in advance for your help! 🙏

What does this mean in a PR and how can I resolve it? "Add labels to github PRs based on context / Add Labels to PRs. Safe to Merge on fail (pull_request_"

Hey all! I'm trying to deploy EKS with helm via Terraform. Version 0.42.0 The issue i face is that the only pods that get deployed are the bootloader /db + minio:

josh@Joshs-MacBook-Pro / % kubectl get pods -n airbyte
NAME                         READY   STATUS    RESTARTS   AGE
airbyte-airbyte-bootloader   1/1     Running   0          3m16s
airbyte-db-0                 1/1     Running   0          3m20s
airbyte-minio-0              1/1     Running   0          3m17s

using helm_release tf resource with the below attributes set:

resource "helm_release" "airbyte" {
  name             = "airbyte"
  namespace        = "airbyte"
  create_namespace = true
  description      = "Airbyte"
  chart            = "airbyte"
  version          = "0.42.0"
  repository = "<https://airbytehq.github.io/helm-charts/>"
}

TF apply times out, with the below error:

failed pre-install: timed out waiting for the condition

I presume the above is causing the issues, has anyone come across anything similar?

Is there a way to have custom logic for incremental loads? I see a lot of connectors have the ability to select a column that’s incremental, but was wondering if there was a way to have a custom definition checking multiple columns or even a json parameter.

Hi guys, i have a question, can airbyte support permission transfer on Postgres source<>target replication for tables? for some reson the data was copied but permissions are not..

Bump: https://airbytehq.slack.com/archives/C021JANJ6TY/p1679794303985569

Hello Airbyte team. I am running several instances of the Facebook Marketing connector (for different clients), all writing out to BigQuery. I am seeing some very long sync times (>24 hours) for initial syncs. For some of the larger insights tables I am using

Incremental | Deduped + history

. I don’t specifically need the deduping and I wonder if this sync mode is potentially part of the issue. I am testing this out myself but am curious if you have any thoughts on that sync mode vs others from a performance standpoint.

Hello Airbyte team, I am trying to send data from mysql to mysql, is there anyway to not create table in the destination by Airbytes, I could like to pre create those table with exact case.

hello Airbyte team. I am running AirByte instance on K8S cluster. I ran into a problem when i try to set up new connection. It always show "Error: non-json response" or "502 bad gateway". I had double check sources and destinations connect and all fine. My connection is from Mongodb to S3 bucket. It happened when i try to setup new connection, all connection same before still working.

Hello lovely Airbyte team. I currently run 2 self-hosted Airbyte on GCP: one in our Dev environment and one in our Prod environment. I would like to change the color of sidebar in order to distinguish between the two envs more easily. Any suggestions? I couldn't find any configurable option for this. Thanks a lot in advance

Hi! We have recently started a POC of Airbyte open source to ingest data into BigQuery. It works great so far. airbyte rocket However, I have noticed that schema fields are seemingly reordered somewhere in the process, so the ordering of columns differ between source and destination. We have MySQL sources and use basic normalization. Is that to be expected? I would assume Airbyte knows about the source schema?

Hello, is there an option to change the default timezone from UTC to a different one using the docker compose method? Adding the timezone to the environment doesn't seem to work.

Hi everyone. Has anyone successfully connected the Amazon SP API Connector? We are self-hosted and run into the error: HTTPError(‘400 Client Error: Bad Request for url: https://sellingpartnerapi-eu.amazon.com/reports/2021-06-30/reports’) I dont quite understand why its calling a report endpoint in 2021 but also cant find a place to change this

I’m having issues with airbyte 0.42.0 and the startup migrations completing successfully. We are deploying it in Kuberentes and the bootloader is failing to come up healthy. Its failing to complete the schema upgrade for

0.32.8.001 - AirbyteConfigDatabaseDenormalization

with the error that

public.actor_type

does not exist yet. Even more baffling is that there is a log line saying it is created right before the error:

2023-03-28 16:41:47 [32mINFO[m i.a.d.i.c.m.V0_32_8_001__AirbyteConfigDatabaseDenormalization(createEnums):85 - actor_type enum created

HI All, is there anyway to retain the latest connector version after a new deployment ? This gets reset everytime I do an install/upgrade via helm

I'm looking to update Airbyte MySQL connector to use parallel connections for reading with JDBC I've read, one can use subconnections to read data in parallel from server. This will increase performance considerably. Looking for some help in doing so, someone a bit better than I in Java

Can anyone give a recommended size for AirByte hosted? 16vpcu and 64G ram?

or is that overkill?

I'm really confused with the S3 connector, do we need to set up a connection for each file we want to replicate if the files are having different columns? For example, I have /file_1.csv and file_2.csv with different columns, do I need to set 2 connections? It appears to be that way since we are using one output stream name.

.

good day @everyone ... i create connections from postgresql to Typesense. somehow if datatype integer it cannot create in typesense. even the Airbyte status completed. anyone have workaround for this, advice ? 🙇

Hi all, there is no Firebase Realtime database source(https://docs.airbyte.com/integrations/sources/Firebase-Realtime-Database) on my local machine. Does anyone know what and how I can use the Firebase Realtime database source?

anyone had any luck connecting to MongoDB’s Atlas “Data Federation”?

Hi all, I am having a problem using airbyte within a private network. I explain the situation. When I deploy the docker stack without being connected to the private network, airbyte works correctly. However, as soon as I connect to the VPN to be on the private network, airbyte is not accessible anymore. I have timeout errors regarding Temporal. Have you ever encountered this problem? Thanks for your help.

Hi Team, does airbyte support 1. spark integration 2. source side filtering of data assets 3. support for joins