Hey! I’m syncing from SurveyMonkey to BigQuery. It seems to sync completely (finishes normalization), but the whole sync fails with an error (larger stack trace in 🧵):

ERROR i.a.w.g.DefaultReplicationWorker(run):196 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!

Additional info: OSS v0.40.17, running on an Autopilot GKE Cluster _SurveyMonkey source_: v0.1.14 _BigQuery destination_: v1.2.13 Takes about 1.5 hours, 132k records/211MB

Hello everyone. I’m syncing from MongoDB et BigQuery. Any help will be appreciated 🙃

2023-02-01 01:29:55 ERROR i.a.w.g.DefaultReplicationWorker(run):177 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:172) ~[io.airbyte-airbyte-workers-0.39.2-alpha.jar:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:62) ~[io.airbyte-airbyte-workers-0.39.2-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.2-alpha.jar:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
	Suppressed: io.airbyte.workers.exception.WorkerException: Destination process exit with code 1. This warning is normal if the job was cancelled.
		at io.airbyte.workers.internal.DefaultAirbyteDestination.close(DefaultAirbyteDestination.java:119) ~[io.airbyte-airbyte-workers-0.39.2-alpha.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:134) ~[io.airbyte-airbyte-workers-0.39.2-alpha.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:62) ~[io.airbyte-airbyte-workers-0.39.2-alpha.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.2-alpha.jar:?]
		at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getDestinationOutputRunnable$7(DefaultReplicationWorker.java:400) ~[io.airbyte-airbyte-workers-0.39.2-alpha.jar:?]

Additional info: v0.39.2-alpha

Hi (Airbyte Team) I just had a quick question. I have a Custom API as a source, Postgres as a destination, and JsonSchema Type as an array of nested objects. I want to put this nested object and array into a separate table. I'm employing the Custom DBT strategy. I was wondering whether there was anything else except Custom DBT?

how to solve the below error?, my model file is not even having 48 lines

Does anyone else split their airbyte databases into multiple Connections? I'm finding it more efficient to split large tables into a separate Connection so that multiple Connections can sync at the same time

Hello everyone i'm actually working with the instagram connector. I'd love to get the comments of all media we recuperate. I saw that a ticket had been opened few months ago (https://github.com/airbytehq/airbyte/issues/11115) but it is still without solution. So if anyone know how to add this feature it would be so powerful in my job ! Thanks a lot.

Hey. I've found an issue in source, debugged it and I know what's required to fix it but I'm not so familiar with code so need someone from airbyte team to point me.

Hey, If a record is deleted in source defined in airbyte, how can we identify in the destination after Synch has happened. Dont see any flag post -synch that indicates a delete for the column.

Hi there, I'm trying to use Xero connector, but it seems that I have to use Xero Custom Connections which has a monthly subscription. Is this the only way to connect with Xero?

Hi, I have a mysql table, normalization fails with the following warnings when using

airbyte/normalization-mysql:0.3.0

but I do not see the warnings with

airbyte/normalization-mysql:0.2.3

. what changed between the versions that can be related to the warnings? Thanks

WARN: Unknown type string for column name at otswcir
WARN: Unknown type string for column dob at otswcir

I am testing importing data from salesforce into BigQuery using local Airbyte for my test setup. I am having a number of tables being created, which are not the original data object

Like above, the original object was account, but now there are 3 objects in the destinations, the billing and the shipping are having a _`_airbyte_Account_hashid` field added to it. Where is this object coming from. Also getting a bunch of

_airbyte__raw_*

tables. Can I not create them, is there something I can set. There is a whole bunch of unwanted noise

Hello team! I'm having problems with the airbyte platform (version 0.40.28) for a while now. The main bug is that there are containers that are not deleted when source replication fails. The error I get from airbyte is "Something went wrong within the airbyte platform". Any idea how to fix it?

AlwaysPull latest image of source

Hello there 👋, I'm currently running Airbyte self-hosted (deployed via Helm on an EKS cluster). I've developed a custom source with the Python CDK, package it in a Docker image hosted on GitHub Container Registry. It runs fine but I have a question: Is it possible to tell Airbyte to always pull the Docker image before launching a source? My usecase would be to only have the

latest

tag on my image and as soon as my CI pushes a new version of this

latest

image, Airbyte would automatically pull it at next run. N.B.: today, we need to tag each image with a different tag, and then go to the Settings -> Source to manually update the version for Airbyte to pull the image

hey, I’m running airbyte self hosted k8s (kustomize) getting this error:

.AsyncOrchestratorPodProcess(computeExitValue):149 - State Store missing status, however orchestrator pod orchestrator-repl-job-9950-attempt-2 in terminal. Assume failure.
2023-02-01 13:19:13 INFO i.a.c.t.TemporalUtils(withBackgroundHeartbeat):316 - Stopping temporal heartbeating...
2023-02-01 13:19:13 INFO i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$5):198 - Completing future exceptionally...
java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
	at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:314) ~[io.airbyte-airbyte-commons-temporal-0.40.27.jar:?]
	at io.airbyte.workers.sync.LauncherWorker.run(LauncherWorker.java:116) ~[io.airbyte-airbyte-commons-worker-0.40.27.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.40.27.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

happens on every attempt to sync. can anyone tell the reason for the issue? I’m afraid this could be caused because I removed the volume

airbyte-volume-configs

as a result of this issue.

Could anyone help me with this?

Hi, is there an estimation of when future operations to run after sync will be available -> https://docs.airbyte.com/understanding-airbyte/operations/#future-operations? I mean other than dbt as the docker script or webhooks mentioned in the documentation

hey. anyone from tech team available? I'd love to take a minute

Hey guys, I tried connecting the amazon seller api. However I realised, that the keys (column names) sended by Amazon are coming in german (since it is a german AWS account), however in airbyte the keys (column names) are expected in english. Therefore the keys (column names) matching does not work. Does someone have an idea how to solve this? (I’m quite new to airbyte, please be not angry if I missed something ovious 🙂 )

Hi, I am trying to fix a bug for connector source-google-analytics-v4, but get some issue when trying to commit the pull requests. If anyone could help me? Thanks

Hi teams, I'm doing a poc with airbyte from Postgres prod -> Postgres datawarehouse for my company in order to replace stitch. I'm using hourly replication frequency and principal sync mode is "*incremental append*". The synchronisation takes about 20mins each time. But after 10 days testing, I found there are data missing during the synchronization: for ex at 2023-01-27, I lost 2 rows of data. When I look into details, the missing data are generated at 2023-01-27 094458.782365+00 and 2023-01-27 094510.105975+00, while my sync slots are marked succeeded. Same problem happened also in 2 others tables I tracked, in different days. Could you pls tell me is that a normal problem for the tools? Any solution to avoid/correcte this problem? I'm willing to provide all the information if needed.

hello everyone! has anyone tried to use a sftp-json destination that can help me understand these errors? when i pass in a password (public key) i get an error that the port could not be cast to integer, which is a bit confusing

Hi everyone, quick question regarding the Google Analytics 4 connector - is it possible to use it with the open-source version of Airbyte, or is it available just for cloud version users? I'm asking since the connector's documentation has 'For Airbyte Cloud' instructions only.

I recreated the PR of fixing google-analytics-v4 segments issue https://github.com/airbytehq/airbyte/pull/22223 but still getting add lable fail issue. I don't know how to fix it. Will leave it for now.

Dear AirByte Team, I am a first-time user of Airbyte and have set up a connection between MySQL (source) and ClickHouse (destination). The syncing process is fetching tables and writing temporary tables to the destination. However, I am encountering issues with the “normalization” routines, leading to the failure of the whole syncing operations. The following exceptions are causing the failures: (in the thread, to avoid clutter)

After running Airbyte in prod successfully for several months, I’m seeing significant stability issues recently--basically, it appears to be crashing the server. The airbyte instance (0.40.32) is deployed on EC2 on a t3a.2xlarge with a 120GB attached volume. We’re using airbyte to move 2-3TB across 10s of tables from PostgreSQL into Snowflake. Based on https://discuss.airbyte.io/t/crashing-freezing-corruption-lost-data/3715, I suspected we’d need to bump the disk size up, but when I check disk usage, it’s not super high and

df

shows about 40% free space. Any suggestion welcome…

Did anyone implement mixpanel to redshift pipeline using airbyte?

Can we configure Airbyte notifications to be received to a custom end point (Not slack)?

Hi , I am installing AIrbyte on AWS EKS and see the airbyte-webapp erroring with... though I am using EKS, it seems using docker files

airbyte-webapp-5b7b558b5-g7mdd 0/1 Error