Suddenly after two days some of the syncs are failing [Troubleshooting] source-mysql, destination-bigquery is this your first time deploying Airbyte: No OS Version / Instance: AWS EC2 Memory / Disk: 32Gb/50 GB Deployment: Docker Airbyte Version: 0.40.7 Airbyte Source name/version: MySQL(0.6.13) Airbyte Destination name/version: BigQuery(1.2.1) Source name/version: AWS Aurora MySQL Serverless v2 (Compatible with MySQL 8.0.23) Destination name/version: Bigquery Step: Create a connection from MySQL to Bi…

Marketo Connector Does Not Work At All [Troubleshooting] source-marketo Is this your first time deploying Airbyte: No OS Version / Instance: EC2 t2.large Memory / Disk: 16Gb Deployment: Docker Airbyte Version: 0.40.7 Source name/version: Marketo 0.1.7 Destination name/version: Snowflake 0.4.36 Step: Source connector fail Description: Marketo sync does not work at all, seems like there is something wrong with the source connector logs-14100 (1).txt (202.9 KB)

Deploy Kubernetes(EKS): Unable to launch airbyte using helm charts [Troubleshooting] deploy, getting-started Is this your first time deploying Airbyte?: Yes OS Version / Instance: Ubuntu 18… are some example remove it with yours Memory / Disk: you can use something like 16Gb / 1 Tb Deployment: Kubernetes Airbyte Version: v0.40.6 Source name/version: Destination name/version: Step: The issue is happening while setting up airbyte using helm chart Description: Reason for failure : Error: INSTALLATION FA…

Hi everyone, I am trying to connect our Mysql server, I am getting the error 'Unsecured connection not allowed' Can anybody help me with the issue and fix

Error while syncing the Clickhouse to Google BigQuery [Troubleshooting] source-clickhouse, destination-bigquery Hi Guys We have as issue while Syncing Clickhouse data to Google BigQuery. How could we fix this issue? Details Is this your first time deploying Airbyte?: No OS Version / Instance: Ubuntu, self-hosted Airbyte instance on AWS EC2 instance of t2.2xlarge with an initial 30 GB of storage Memory / Disk: initial 30 GB of storage Deployment: Docker Airbyte Version: 0.40.7 Source name/ver…

MongoDB Failed to fetch schema [Troubleshooting] connectors Is this your first time deploying Airbyte?: No OS Version / Instance: Ubuntu 22.04.1 LTS Deployment: Docker Airbyte Version: 0.40.06 Source name/version: MongoDB 0.1.17 Destination name/version: Google Bigquery 1.2.1 Step: Creating new connection, and refreshing source schema on existing connections Description: When refreshing existing connections (that have not had issues for weeks), the scr…

Incremental Sync acceptance test failure [Connector Development] connectors Hi Team, I have an issue where I would like to use the date field and name filed in record in combination as a cursor field. My records are like this eg below:

{"type": "RECORD", "record": {"stream": "states_history_cases", "data": {"name": "Schleswig-Holstein", "state": "SH", "cases": 167, "date": "2022-09-18T00:00:00.000Z"}, "emitted_at": 1663690865497}}

`{"type": "RECORD", "record": {"stre…

"invalid input syntax for type" normalization ASP -> Postgres/Mysql [Troubleshooting] normalization Is this your first time deploying Airbyte?: Yes OS Version / Instance: Windows Memory / Disk: Deployment: Local Docker Containers Airbyte Version: 0.40.4 Source name/version: Amazon Seller Partner Destination name/version: Postgres 15.0 or MYSQL 8.0.30 Step: The issue is happening during sync Description: I tried to normalize the amazon seller partner data in a mysql database first and t…

Twilio connector and subaccounts [Connector Development] data-loading, connectors Deployment: Docker Airbyte Version: What version are you using now? 0.40.7 Source name/version: Twilio version 0.1.8 Destination name/version: Snowflake 0.4.36 Step: During sync While using the twilio connector it appears that only the main account data is being pulled. We have a lot of subaccounts. Would it be possible to adjust the implementation to automatically get data from subaccounts or…

Stack Syncs and Automated Full Resyncs on CDC [Troubleshooting] destination-snowflake, source-postgres, connectors Is this your first time deploying Airbyte?: No OS Version / Instance: Ubuntu 18.04.4 LTS Memory / Disk: you can use something like 16Gb / 50GB Deployment: Docker Airbyte Version: 0.40.3 Source name/version: Postgres Connector / 1.0.7 Destination name/version: Snowflake Destination / 0.4.34 Step: Sync Description: The behaviour that I am about to describe is a little bit complex but I will try …

The version 0.39.14-alpha keeps failing to update any connectors [Troubleshooting] connectors Is this your first time deploying Airbyte?: No OS Version / Instance: Plural Memory / Disk: you can use something like 15Gb Deployment: Are you using Docker or Kubernetes deployment? Plural Airbyte Version: What version are you using now? 0.39.14-alpha Source name/version: * Destination name/version: * Step: The issue is happening during sync, creating the connection or a new source? Updating …

Error with MySQL Connector [Troubleshooting] connectors Is this your first time deploying Airbyte?: Yes OS Version / Instance: Windows 10 Memory / Disk: 16Gb / 255 Gb Deployment: Docker Airbyte Version: 0.40.7 Source name/version: MySQL (Alpha) Destination name/version: BigQuery Step: Sync Description: When I try to retrieve the information it prompts me with the following error. 2022-09-22 061650 e[32mINFOe[m i.a.a.Loggi…

Powered by Airbyte connectors [Q&A] connectors Hi, does powered by Airbyte support all the connectors / sources listed or is there connector support list for powered by Airbyte? Thank you

Unable to setup Mysql Source [Troubleshooting] connectors When setting up a local machine instance of a MySQL connector the source tests run and I can use the source. When I try to use the same credentials for our GCP instance of Airbyte when I test my connection it returns a non-Json response error. Is this your first time deploying Airbyte?: No / Yes OS Version / Instance: Ubuntu, Windows or MacOS… are some example remove it with yours Memory / Disk…

Need some guidance on customizing oauth token refresh [Connector Development] connectors, oauth I’ve built a custom connector for a private API that we use, and it’s tested and working. The last step is to automate the generation of an access token. I have to provide a client_id and client_secret to an endpoint, and in return I get an access token (no refresh token) that expires in X amount of time. How can I automate this into my connector such that it will provide the client_id and client…

MySQL source connector: slow CDC replication [Troubleshooting] source-mysql Is this your first time deploying Airbyte?: No OS Version / Instance: Linux Ami 2 Memory / Disk: you can use something like 16Gb / 90 Gb Deployment: Docker Airbyte Version: 0.40.4 Source name/version: MySQL 0.6.8 Destination name/version: Redshift 0.3.47 (S3 Staging) Step: sync Description: Sync using CDC from MySQL to Redshift is very slow: it takes around 18 mins to sync just 9.9k records or …

Netsuite source connector Date Issue [Troubleshooting] connectors Is this your first time deploying Airbyte?: No OS Version / Instance: MacOS Memory / Disk: you can use something like 4Gb / 1 Tb Deployment: Docker Airbyte Version: 40.7 Source name/version: netsuite Destination name/version: Snowflake Step: The issue is happening in all the scenario of interface. Incremental or Full. Description: Date format is not matching with Netsuite recent updates. 2022…

New Postgres source has no incremental option [Troubleshooting] source-postgres Is this your first time deploying Airbyte?: Yes OS Version / Instance: Amazon Linux Memory / Disk: t3a.2xlarge Deployment: Docker Airbyte Version: 0.40.7 Source name/version: source-postgres 1.0.10 Destination name/version: destination-snowflake 0.4.36 Step: The issue is happening creating the connection and selecting how to replicate Description: When configuring the replication between postgr…

Starting the Airbyte API on a self-hosted instance [Connector Development] I need to be able to change the state of a particular connector, I’m aware you can do that through the Airbyte API as described here But when I try to do it on our self-hosted instance (via postman) the API request just returns a 405 Not Allowed header. I assumed that the and when I click through the web interface I don’t see anything related to the API anywhere. Does it even start when you run d…

Passing stream results from into a secondary stream's API path [Connector Development] connectors I have three streams. accounts |=statement list |==statements A field from accounts is used to populate the path for statement list. Two fields from statement list are needed to populate the path for statements. I have the first two streams working the way I want by defining a class-specific stream slice. class AccountRelatedStream(ParentClassStream, ABC): def stream_slices( …

Postgtres to BigQuery with GCS staging - Sync worker failed [Troubleshooting] data-loading Is this your first time deploying Airbyte?: Yes OS Version / Instance: AWS EC2 instance type t2.medium Memory / Disk: you can use something like 4Gb / 30 Gb Deployment: Docker Airbyte Version: 0.40.8 Source name/version: Postgres 0.4.6 (I made downgrade becouse the timestamp error) Destination name/version: Bigquery 1.2.3 Step: The issue is happening during sync and source and connectors are cre…

image.png

image.png

Unable to Creating the Source for building a custom connector using Python CDK [Connector Development] connectors, getting-started Hi, I try to build a custom connector using Python CDK However, I’m stuck at the very first step at running generate.sh file. The error is: Use ‘docker scan’ to run Snyk tests against images to find vulnerabilities and learn how to fix them Running generator… the input device is not a TTY. If you are using mintty, try prefixing the command with ‘winpty’ While trying to generate a connector…

Support for Databricks destination on Azure [Connector Development] destination-databricks Right now my understanding is that the Databricks Lakehouse destination is only for AWS deployments (S3 requirement). Is there a roadmap for Azure support? My organisation has strict rules on how data is moved and our platform of choice is Azure. Thanks!

Mixpanel connector provides no data [Troubleshooting] connectors Hello, Is this your first time deploying Airbyte?: No Source name/version: Mixpanel 0.1.23 Destination name/version: Snowflake Step: The issue is happening during sync Description: Since last Mixpanel connector update, my sync has no record in it. Can you help me to identify where the issue comes from please ? Available to provide any additional informati…

Postgres source Incremental mode on views [Q&A] source-postgres Hello, we have noticed that in the Postgres source version 1.0.3 there was a change implemented, which hides Incremental mode on tables with no cursor field. Here is the respective PR. With this change there is now no possibility to have Incremental mode on views. In our case, we use views on source to filter out for sensitive information which should not go to the DWH, but we still want to crea…

Airbyte log sweeping [Q&A] Hi. Can’t seem to continue on this thread: I would like to sweep Airbyte logs more frequently as our tables are quite big and hence the respective logs are also bigger than usual, causing disk space to become full. Using the above link as a guide, I’ve added to the .env file: MAXIMUM_WORKSPACE_RETENTION_DAYS=1 I expect any logs older than a day to be cleared, but that doesn’t seem to happen. …

Support for Parquet format in AzureBlobStorage destination [Product Feedback and Ideas] connectors, azure I would love if the AzureBlobStorage destination connector could save files in Parquet format. This would allow us to pass metadata about field data types to the destination. Thanks! Martin