Issues connecting to Azure Synapse [Troubleshooting] connectors Is this your first time deploying Airbyte?: Yes OS Version / Instance: Ubuntu Memory / Disk: 8Gb / 100Gb Deployment: Docker Airbyte Version: 0.40.4 Source name/version: Destination name/version: MS SQL Server 0.1.20 Step: Setting destination. Description: Hi, I’m triying to set Azure Synapse as destination but I got Could not connect with provided configuration. Enforced unique constraints ar…

What Airbyte tables can I delete after a sync? [Frequently Asked Questions] normalization For this question, there are usually three tables that most users ask about: the _tmp , _raw and _scd tables. _tmp is safe to delete. If there’s a sync in-progress, it might fail (but should work fine on a retry). scd is not safe to delete. It’s required for incremental normalization to continue working. _raw is technically safe to delete (assuming normalization is completing successfully aft…

Airbyte failling after reading msql messages [Troubleshooting] normalization I have tried multiple times to get the msql - BQ connector working. It connects well enough and then shows it consuming the data from the MySQL data source but then fails at the last minute in normalisation. 2022-09-27 101108 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):103 - HikariPool-1 - Starting… Is this your first time deploying Airbyte?: No / Yes OS Version / Instance: Ubuntu,…

We are not able to configure DB2 and Oracle as source using Airbyte [Connector Development] Hello, We have installed Airbyte in our local system. While trying to configure DB2 and Orcale we are getting the error provided in the attached document. Thanks Tapas Db2_to_local_Json.doc (163.6 KB) Oracle_Source_creation_error.doc (283.2 KB)

JSON not being copied on incremental sync [Troubleshooting] data-loading, source-postgres, destination-snowflake https://discuss.airbyte.io#environment-1|Environment Airbyte version: 0.40.6 OS Version / Instance: Amazon Linux 2 / AWS EC2 Deployment: Docker Source Connector and version: Postgres 1.0.7 Destination Connector and version: Snowflake 0.4.35 Step where error happened: After initial sync https://discuss.airbyte.io#current-behavior-2|Current Behavior We have a column which contains json data, but of type text in postgres. We’re using wal2json plugin for logical replication. …

Postgres-destination: specify target table [Q&A] connectors Hello, I am fetching the same "ads_reports" (from airbyte/source-tiktok-marketing:0.1.15) stream with 2 different angularities (daily and lifetime), and I need to write the data into 2 different tables, but they currently get written to the same table. I would like to know if it is possible to achieve that. I am using airbyte/destination-postgres:0.3.20 Thank you in advance Imane

MySQL Source: sporadic

java.io.EOFException: Failed to read remaining bytes

errors [Troubleshooting] source-mysql Is this your first time deploying Airbyte?: No OS Version / Instance: Linux Ami 2 Memory / Disk: you can use something like 16Gb / 90 Gb Deployment: Docker Airbyte Version: 0.40.4 Source name/version: MySQL 0.6.8 Destination name/version: Redshift 0.3.47 (S3 Staging) Step: sync Description: Sync sporadically starts to fail with the following error: 2022-09-28 001225 e[1;31mERRORe[m i.d.p.Err…

Airbyte read data from the beginning without a reset [Troubleshooting] source-postgres Hi After a failure Airbyte starts reading all the data from an incremental table from the beginning even if I did not reset the data. This is an unexpected and annoying because the table is veeeery large and we don’t want to re read it again Do you have any explanation ?

Struggling with S3 to Snowflake connection [Troubleshooting] data-loading, destination-snowflake, connectors, source-s3 Is this your first time deploying Airbyte?: No OS Version / Instance: Linux Memory / Disk: you can use something like 128GB / 1TB Deployment: Docker Airbyte Version: 0.40.9 Source name/version: S3 - 0.1.21 Destination name/version: Snowflake 0.4.38 Step: During sync Description: Currently unable to get a connection from an S3 bucket to Snowflake to sync. It has worked in the past. We have thr…

Reminder: Office Hours Half Hour today (and every Wednesday)! [Q&A] getting-started Hey all! A quick reminder that every Wednesday we have the Airbyte Community Office Hours Half Hour – 11AM Pacific time. This is a great chance to see the Community Assistance team, ask direct questions, and get timely answers!

Facebook marketing v15 [Troubleshooting] source-facebook-marketing, connectors Hi guys, We are trying to replicate data from facebook using Airbyte and get the following error You are calling a deprecated version of the Ads API. Please update to the latest version: v15.0.') We have tried downgrading to < v0.15 in our facebook ads account but no other version seem to be available to us. We have seen several posts already with similar error messages on older API versions,…

MySQL to Snowflake incremental loading fails [Troubleshooting] source-mysql, destination-snowflake Is this your first time deploying Airbyte?: No OS Version / Instance: EC2 Linux AMI 2 Memory / Disk: M6a.xlarge instance - 4 vCPUs, 16 GiB memory, 60 Gb storage Deployment: Docker Airbyte Version: 0.40.9 (latest as of writing) Source name/version: MySQL 0.6.14 (latest as of writing) connected through SSH tunnel Destination name/version: Snowflake 0.4.38 (latest as of writing) internal staging S…

Airbyte-worker exited with code 1 [Troubleshooting] getting-started, deploy Is this your first time deploying Airbyte?: Yes OS Version / Instance: Ubuntu Memory / Disk: 100gb Deployment: Docker Airbyte Version: 0.40.9 Source name/version: NA Destination name/version: NA Step: The problem is at the initialization of the service Description: I am following the GCP setup guides but after I run the docker-compose up step the process keeps failing because of the worker (air…

Missing data when connecting from Shopify to Big Query [Troubleshooting] destination-bigquery Is this your first time deploying Airbyte?: Yes OS Version / Instance: Ubuntu, Windows or MacOS… are some example remove it with yours Memory / Disk: 4Gb / 1 Tb Deployment: Docker Airbyte Version: 0.40.8 Source name/version: Shopify Destination name/version: Big Query Step: The issue is happening during sync Description: We are getting data from shopify and sending it to Big Query. We are fetc…

How do I implement a source defined cursor to a full refresh connector [Connector Development] connector-development, normalization Hi can anyone please guide me how to go about implementing incremental connector with a source defined cursor.

Improving Connector Speed [Troubleshooting] normalization, data-loading Hi All, We are trying to improve sync speed for Facebook connector by playing with parameters provided in airbyte/.env at master · airbytehq/airbyte · GitHub There are two areas of focus. Sync speed. For facebook marketing, it takes 5 minutes to download even small amounts of data. Normalization is taking many minutes - with just kilo bytes of data. Can you please help us how to improve…

A deep dive into Airbyte's CDC replication [Guides] getting-started, data-loading, source-postgres, connectors The third tutorial in Airbyte’s series on synchronization modes is now live. This article gives an explanation and overview to help you understand Airbyte’s Change Data Capture (CDC) data synchronization. [image] airbyte.com

Support incremental sync "overwrite" mode [Product Feedback and Ideas] We have to index millions of records from Snowflake into an Elasticsearch index. Our expectation is that an update of a record in Snowflake will update the correspondent document in the Elasticsearch index. This behaviour is currently available for the fullrefresh-overwrite mode only (the table primary key is used as the document id). We cannot afford to always do a full refresh of the index as…

NullPointerException: "schedule" is null [Troubleshooting] data-loading, kubernetes Is this your first time deploying Airbyte?: No OS Version / Instance: CentOS Linux 7 (Core) Memory / Disk: 12Gb / 1Tb+ Deployment: Kubernetes Airbyte Version: 0.40.4 (recently updated from 0.35.10 alpha) Source name/version: MSSQL 0.4.20 (but similar problems with other connectors as well) Destination name/version: Snowflake 0.4.38 Step: during sync Problem Some of the connections running on ou…

Custom path for DBT's profiles.yml [Troubleshooting] transformation, normalization, data-loading Hey, I would like to add a custom dbt transformation for my replication to Bigquery. I also need to specify custom destination dataset in Bigquery (default one writes to airbyte_raw ). To do that I had to put my file profiles.yml into my dbt project github repo in order to specify my dataset. Following Airbyte doc, I have now to specify the profiles.yml path into Airbyte GUI Entrypoint arguments f…

has renamed the channel from "airbyte-for-beginners" to "get-help"

set the channel topic: This is a great place for general questions, advice and to spark discussions. If you haven’t yet, check out our Forum, where there are lots of answers to problems that are easily searchable -> https://discuss.airbyte.io

has renamed the channel from "get-help" to "airbyte-help"

Airbyte seems to cancel after running records I wrote a custom connector. As part of the "read_records" method I am writing to the log of when it starts and stops as well as the APIs that it's hitting. Airbyte seems to run all read_records, then fail after seemingly finishing. Attached is the log file (ids have been changed).

This is a screenshot of error generated

image.png

I fall into below error in login page of Airbyte..Can someone please help with it

Let me know if the above helps, Shivam!