Hi, is it possible to keep two tables in Sync bi-directionally i.e. Data can be updated in each table and the other needs to be updated based on the latest updated date.

I am unable to fork the repo to submit a PR for a bug fix, can anyone help me with this? Thanks

Normalize to MySQL/Mariadb destination Hi all, I'm trying to get data from a local Postgresql database to a local MySQL v 8.0.26 OR MariaDB -V 10.3 server. Regardless of the source I use (Postgres in this case, but also Adwords, Google Analytics), I get the following error:

io.airbyte.workers.WorkerException: Normalization Failed.

at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:61) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]

at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:19) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]

at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]

at java.lang.Thread.run(Thread.java:833) [?:?]

Caused by: io.airbyte.workers.WorkerException: Normalization Failed.

at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:58) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]

... 3 more

Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn't successful

at io.airbyte.workers.normalization.DefaultNormalizationRunner.close(DefaultNormalizationRunner.java:160) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]

at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:46) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]

at io.airbyte.workers.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:19) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]

at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.36.0-alpha.jar:?]

at java.lang.Thread.run(Thread.java:833) [?:?]

Has anyone got normalize to work with MySQL/MariaDB? {edit} AND some version info would be useful. 0.36.0-alpha running on Rocky8.4

Hi, all I am trying to run airbyte locally on my k8s cluster. I have everything set up and running, but when I go to the webapp I get "Unknown error" Also the server logs don't show anything

Setting up Facebook Marketing, anyone seen this? FacebookAPIException('Error: 100, (#100) Missing permissions') 0.36.0-alpha on Rocky8.4 NB - credentials allow data extract via Python

Hi all, i wanna understand the backend queries for full_refresh and Incremental Sync - Deduped History sync modes. And is there a way to customise these queries as per our requirements. Thanks in advance!!

Hi, I want to do something like a MYSQL to MYSQL replication with some timestamp conversions as transformations in between. Can i do that using Airbyte using CDC similar to Streamsets?

is there a way to support auto-sync in airbyte

Hi Everyone Actually, I want to do certification in airbyte. Is there any course for learning airbyte tool?

Hi Team, I am setting up Airbyte from docker but getting the below error. Please help me

airbyte-bootloader | 2022-04-20 07:45:07 ERROR i.a.b.BootloaderApp(assertNonBreakingMigration):189 -                ___ ___________________  ________________ _  __
airbyte-bootloader |               /  |/_ __/_ __/ ____/ | / /_ __/ _/ __ \/ | / /
airbyte-bootloader | ________________________  / /| | / /  / / / __/ / |/ / / / / // / / / |/ / ________________________
airbyte-bootloader | /_____/_____/_____/_____/ / ___ |/ /  / / / /___/ /| / / / _/ // /_/ / /| / /_____/_____/_____/_____/
airbyte-bootloader |  __ _______ __________ /_/ _|_/_/ _/_/_/_____/_/_|_/_/_/ /___/\____/_/_|_/ __ __________ __________
airbyte-bootloader |  / / / / ___// ____/ __ \  / _/ | / / __ \/ / / /_ __/ / __ \/ ____/ __ \/ / / / _/ __ \/ ____/ __ \
airbyte-bootloader | / / / /\__ \/ __/ / /_/ /  / // |/ / /_/ / / / / / /  / /_/ / __/ / / / / / / // // /_/ / __/ / / / /
airbyte-bootloader | / /_/ /___/ / /___/ _, _/ _/ // /| / ____/ /_/ / / /  / _, _/ /___/ /_/ / /_/ // // _, _/ /___/ /_/ /
airbyte-bootloader | \____//____/_____/_/ |_| /___/_/ |_/_/  \____/ /_/  /_/ |_/_____/\___\_\____/___/_/ |_/_____/_____/
airbyte-bootloader | 
airbyte-bootloader | -------------------
airbyte-bootloader |  See details below
airbyte-bootloader | -------------------
airbyte-bootloader | 
airbyte-bootloader | 2022-04-20 07:45:07 ERROR i.a.b.BootloaderApp(assertNonBreakingMigration):197 - Cannot upgrade from version 0.27.1-alpha to version 0.36.1-alpha directly. First you must upgrade to version 0.32.0-alpha. After that upgrade is complete, you may upgrade to version 0.36.1-alpha

hi team, can someone confirm with me how to load a custom connector image on the Airbyte cloud? I don't see the the option to add a new connector in my airbyte cloud . Thanks!

I Built a new source connector and built an docker images and its working fine. Now some other server running airbyte on kube, how to send/use my docker image on their server(airbyte on kube). Kindly help me.

Hi, I would appreciate help in upgrading an existing airbyte solution I inherited. Airbyte installed on GCP VM ubuntu 20.04 Current version: https://github.com/airbytehq/airbyte/releases/tag/v0.30.16-alpha The docs https://docs.airbyte.com/operator-guides/upgrading-airbyte instruct to checkout v0.32.0-alpha-patch-1 first if I am understanding it correctly : "If your current version of airbyte is < v0.32.0-alpha-patch-1, you first need to upgrade to this version before upgrading to any later version." but this has failed miserably for some reason and I spent 2 hours getting back to previous state. The UI was not accessible, airbyte-server logs had no errors, but the DB server logs showed sql stmt errors. This leads me to believe the upgrade doc is incorrect. I can try installing the same prod version locally on my dev environment and try upgrading from there but would like to avoid a lot of trial and error if possible. So, again, I'd really appreciate help on getting my instance up to date without losing existing connections.

hey folks, I want to contribute to Airbyte and followed the quick start guide from the Github. when I run ``docker-compose up` I get a network error:

WARNING: The RUN_DATABASE_MIGRATION_ON_STARTUP variable is not set. Defaulting to a blank string.
WARNING: The SECRET_PERSISTENCE variable is not set. Defaulting to a blank string.
WARNING: The WORKER_ENVIRONMENT variable is not set. Defaulting to a blank string.
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

can you navigate me the direction I need to debug?

Hello everyone ! I am having some trouble setting the DBT transformation. As far as I know, we should have a custom transformation option at the bottom of the connection form. However, for some reason, I have only: "Raw data (JSON)" and "Normalized Tabular Data". What am I missing ? I am using the Cloud Version by the way. The connection is from RS to Snowflake. Thank you guys so much for your help. Have a great day !

Hi - I’m trying to update an existing connector and run the whole stack locally.

Hi - I’m trying to update an existing connector and then run the whole stack. I’ve updated the connector’s

spec.json

,

Source.java

and Dockerfile. I ran

./gradlew :airbyte-integrations:connectors:<connector>:build

and then

./gradlew build
SUB_BUILD=PLATFORM ./gradlew build

But I cannot see my change in the UI (should have a dropdown). The changes I’ve made basically mirror https://github.com/airbytehq/airbyte/pull/10995 Can anyone provide some insights please?

hey, I made my first PR to airbyte. I am confused how where can I read more about test codestyles? couldn't find any in contributing guide. currently, I change on my data but are there any test samples for the connector exists?

Hey all, I want to create a kind of mega Dockerfile that contains airbyte and friends to showcase a few tools to run with a data lake on mybinder.org. It's ideal if I keep the Docker image that runs on binder as small as possible to load faster. So I'm hoping to find a way to install Airbyte that wouldn't require installing Docker in Docker. Any options here?

I found

34.106.109.131

as the IP necessary to whitelist. Am I missing something, is this in any docs anywhere? I found it in an archived thread here.

Hello Team , We are trying to evaluate airbyte and testing out the airbyte load features... have some questions.... we have large volume of data load (full load refresh) from data source , we see issues that it fails to load large volumes in one full load using airbyte feature ( full load) ...any suggestions or any ideas how to deal with the large volume ( full refresh ) for large volumes on airbyte level. Thank you.. Please find log message....error message...what is the real issue of the error ...is it source connection timeout or anything else ....2022-04-19 204401 [32mINFO[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 24294000 (72 GB) 2022-04-19 204401 [32mINFO[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):301 - Records read: 24295000 (72 GB) 2022-04-19 204401 [32mINFO[m i.a.w.p.KubePodProcess(close):708 - (pod: airbyte-dev / source-mssql-sync-649-2-labsr) - Closed all resources for pod 2022-04-19 204401 [32mINFO[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):305 - Total records read: 24295634 (72 GB) 2022-04-19 204401 [32mINFO[m i.a.w.p.KubePodProcess(close):708 - (pod: airbyte-dev / destination-s3-sync-649-2-emnch) - Closed all resources for pod 2022-04-19 204401 [33mWARN[m i.a.c.i.LineGobbler(voidCall):86 - airbyte-destination gobbler IOException: Socket closed. Typically happens when cancelling a job. 2022-04-19 204401 [1;31mERROR[m i.a.w.DefaultReplicationWorker(run):169 - Sync worker failed. java.util.concurrent.ExecutionException: io.airbyte.workers.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 143 at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?] at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:162) ~[io.airbyte-airbyte-workers-0.35.67-alpha.jar:?] at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:57) ~[io.airbyte-airbyte-workers-0.35.67-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.67-alpha.jar:?] at java.lang.Thread.run(Thread.java:833) [?:?] Suppressed: io.airbyte.workers.WorkerException: Source process exit with code 143. This warning is normal if the job was cancelled. at io.airbyte.workers.protocols.airbyte.DefaultAirbyteSource.close(DefaultAirbyteSource.java:136) ~[io.airbyte-airbyte-workers-0.35.67-alpha.jar:?] at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:126) ~[io.airbyte-airbyte-workers-0.35.67-alpha.jar:?] at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:57) ~[io.airbyte-airbyte-workers-0.35.67-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.67-alpha.jar:?] at java.lang.Thread.run(Thread.java:833) [?:?] Suppressed: io.airbyte.workers.WorkerException: Destination process exit with code 143. This warning is normal if the job was cancelled. at io.airbyte.workers.protocols.airbyte.DefaultAirbyteDestination.close(DefaultAirbyteDestination.java:119) ~[io.airbyte-airbyte-workers-0.35.67-alpha.jar:?] at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:126) ~[io.airbyte-airbyte-workers-0.35.67-alpha.jar:?] at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:57) ~[io.airbyte-airbyte-workers-0.35.67-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.67-alpha.jar:?] at java.lang.Thread.run(Thread.java:833) [?:?] Caused by: io.airbyte.workers.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 143 at io.airbyte.workers.DefaultReplicationWorker.lambda$getReplicationRunnable$5(DefaultReplicationWorker.java:312) ~[io.airbyte-airbyte-workers-0.35.67-alpha.jar:?] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]

Hello Team, I am trying airbyte for first time and hence asking it here -> Not able to create source / destination in docker + mac silicon chip Environment • Airbyte version: 0.36.1-alpha • OS Version / Instance: mac 12.3.1 (Silicon chip) • Deployment: Docker • Source Connector and version: PokeAPI • Destination Connector and version: LocalCSV • Severity: High • Step where error happened: Deploy Current Behavior Tell us what happens. I am trying airbyte for first time in my local machine with docker, and trying to setup a sample source and destination. But when I create it, its been running for more than 30 mins and source and destination are not getting created. It just says

Please wait a little bit more…

Hi everyone! Is it possible to make changes in the source of the connection? This is my first time to make a connection and I only setup 1 project in my Jira source. Now I'd like to add more projects.

Hello! Quick question, is there a way to duplicate a source and a destination? Just so we don't need to do the setup from scratch.

Does anyone know what this error mean in the beginning of the sync?

is incremental append sync mode available for local_json destination ?

Flexing my first 3 connections... Not sure why they are failing...

Hi Team Airbyte, I am trying to sync data from zoho CRM to postgres, but facing some issue. Here is the screenshot.

I'm getting this error for my mongodb to s3 connection

errors: $.server_addresses: is missing but it is required, $.instance: does not have a value in the enumeration [replica]

Please help

Hi Team Airbyte, i startet to create first connections but I'm facing everytime the issue, that all files in S3 (Destination) are without file extensions. I'm using airbyte 0.36.1-alpha (Docker Image). Is that a known issue? I have attached some configs and tried to follow official documentation. I have this issue with all output options and not only with parquet.