Ga4 api to bigquery: problem with incremental data [Troubleshooting] connectors Hi, 😄 I succeeded in getting a custom report data from GA4 API , then send it to bigquery. Now, I wanna update only incremental data from the source but there is nowhere to setup custom cursor field or set up the end date is yesterday’s date. Is there any way to do so? Let’s me explain a little bit: So, here is some of my setup info: Custom reports: [{“name”: “Revenue by ad events”, “dimen…

Hi All, I am trying to create a connector “Microsoft OneDrive and Microsoft OneNote” for Airbyte. I followed the Airbyte docs and was able to successfully run a code to fetch users details from Microsoft app. I created an Application in Microsoft Azure portal and used Application permissions for accessing data. Though I was able to fetch users stream, I am getting following error while fetching drives, sites and OneNote details. “Tenant Id does not have SharePoint Online (SPO) License” Can anyone guide me on this issue? Does anyone know if Airbyte team will be able to provide a trial app with SPO license for checking the code and if so whom to contact? Thanks!

"Tenant Id does not have SharePoint Online (SPO) License" Error while trying to read OneDrive details via Microsoft App using Azure Portal [Connector Development] I am trying to create a connector “Microsoft OneDrive and Microsoft OneNote” for Airbyte. I followed the Airbyte docs and was able to successfully run a code to fetch “users” details from Microsoft app. I created an Application in Microsoft Azure portal and used Application permissions for accessing data. Though I was able to fetch users stream, I am getting following error while fetching drives,…

image.png

Connection to Google Sheets fails when google sheets has a Data Connector to Big Query [Troubleshooting] source-google-sheets, connectors Is this your first time deploying Airbyte?: Yes OS Version / Instance: Linux on AWS EC2. 2 vCPU. Memory / Disk: 8 GBs memory Deployment: Docker Airbyte Version: 0.40.0-alpha Source name/version: Google Sheets / 0.2.17 Destination name/version: BigQuery / 1.1.16 Step: An old source, while loading and while retesting the connection Description: We had this running sync from Google Sheets to BigQ…

How to speed up api responses [Troubleshooting] deploy Is this your first time deploying Airbyte?: No OS Version / Instance: Ubuntu Memory / Disk: you can use something like 64Gb / 500 gb Deployment: Docker Airbyte Version: 0.40.0-alpha Source name/version: Destination name/version: Step: All API responses Description: Just want advice on how to improve API response time. what environment variables can be set to get everything done faster? Anythin…

Source halted midway, nothing in log (UI) [Troubleshooting] connectors Hi, as the title says, the connection stopped mid way. The logs aren’t updating for almost a day. The connector is doing an initial sync, and it shows “syncing”. Other connectors working fine. I see nothing in logs on the Airbyte UI - it simply doesn’t update anymore, no errors, nothing. Question: what is the strategy I could use to dive deeper into what’s happening? Any command line options to …

Cannot access webapp as call to list_latest times out [Troubleshooting] kubernetes, deploy Is this your first time deploying Airbyte?: No OS Version / Instance: linux rhel 7 Memory / Disk: 8Gb/30Gb Deployment: Are you using Docker or Kubernetes deployment? Kubernetes Airbyte Version: What version are you using now? 0.39.36-alpha Source name/version: Destination name/version: Step: The issue is happening during sync, creating the connection or a new source? When opening up webapp Descr…

Java Destination SPI: start v.s. accept [Connector Development] connector-development Hi, I’m trying to develop a Java-based Destination, and looking into AirbyteMessageConsumer. My question here is: What is the difference between start v.s. accept? I saw some connectors have comment in start (or startTracked) like: // todo (cgardens) - move contents of #write into this method. So I guessed implementing everything in accept might somewhat work, but I was not sure of their dif…

Contributing a Connector implementation from out of Airbyte [Connector Development] connector-development Hi, wanted to confirm a couple of points about contributing a new Connector implementation from out of Airbyte. Q1. Looking in other Connectors under GitHub - airbytehq/airbyte: Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases., most (or all?) of them are boilerplated like Copyright (c) 20XX Airbyte, Inc., all rights reserved. Are…

Incremental Models Failing [Troubleshooting] destination-snowflake, data-loading Is this your first time deploying Airbyte?: No OS Version / Instance: amazon_linux_2_ami_id Memory / Disk: volume size 100gb Deployment: Docker Airbyte Version: 0.40.3 Source name/version: Postgres 1.0.4 Destination name/version: Snowflake 0.4.34 Step: My sync is failing for incremental deduped + history models on the second run (incremental run) Description: The error message I’m getting is Ca…

Sync suddenly fails [Troubleshooting] connectors, data-loading Is this your first time deploying Airbyte?: Yes OS Version / Instance: Amazon Linux - EC2 t2.large Memory / Disk: 8Gb / 30Gb Deployment: Docker Airbyte Version: 0.40.6 Source name/version: MySQL - 0.6.11 Destination name/version: S3 - 0.3.15 Step: Issue happens during sync Description: Error occurs during sync: Table rows count: 42m

The requested stream categories was not found in the source [Connector Development] Hello Airbyte community, I’m having some problems when creating my own Airbyte Connector. I got an error message saying: The requested stream categories was not found in the source. Available streams: dict_keys(['accounts']). What does this mean and how can I fix this problem?

Hello, I have two questions regarding a sync of 6 very small test tables with basic normalization enabled that I am running postgres > redshift dbs on my docker deployment hosted on ec2. First, I am noting two ~5-7 min stalls just as dbt is getting started. I am unsure what these lags are as they do not happen in a dbt project that I have written to test locally on the same database. What is happening here? This greatly affects our performance Second, I would like to increase the number of dbt threads in the future as we scale our workloads. Is there a way to override the default number of threads without specifying my own dbt project. I know thread count is configured in the dbt profiles.yml file which I can't include in my custom dbt project gitlab project anyway since it contains credentials.

Configuring state storage to s3? [Troubleshooting] deploy Is this your first time deploying Airbyte?: No OS Version / Instance: AWS EKS Memory / Disk: 1TB Deployment: Kubernetes Airbyte Version: 0.40.2 I am trying to configure the state storage to push data to s3 instead of storing them on the local minio pod. I’ve tried doing something similar to configuring the custom s3 bucket where I set the STATE_STORAGE_MINIO_BUCKET_NAME to the same bucket name …

Source Google Sheets connection tests failed [Troubleshooting] source-google-sheets → Is this your first time deploying Airbyte?: Yes OS Version / Instance: Windows Memory / Disk: 8 GBs memory Deployment: Docker Airbyte Version: 0.40.6 Source name/version: Google Sheets Destination name/version: Step: Connection with a new source Description: Connection tests failed. I’ve tried to connect via “Account service key” and “OAuth” but the error is the same. I made this same conn…

Create a new connector automatically importing OpenAPI specification of API REST compatible product [Product Feedback and Ideas] connectors One of the main stoppers for people to use the product is if they need to get data from a product that don’t have already a connector builded. Not every body has the neccesary Python skills to build a connector, and that do that some people go away from Airbyte, to find another product that is more easy to get data into from any source. I saw in other products of this kind that is common to have…

Error on Normalizatrion at Source MYSQL [Troubleshooting] normalization, destination-mysql Hello, I’m using an instance in oracle. Processor: ARM Ampere A1 RAM: 24GB I connected Facebook as source and MySQL as destination, but the sync does not end when I select the option: Normalized tabular data. How can I resolve this error? I have already connected another PostgreeSQL instance from Amazon AWS and the same error also occurs. Attached are the error logs logs-111.txt (158.9 KB)

Airtbyte in a crashloop on fresh helm install [Troubleshooting] kubernetes, getting-started Is this your first time deploying Airbyte?: Yes OS Version / Instance: Linux Deployment: K8S Airbyte Version: helm chart v0.40.8 Description: Installing the chart for the first time. Multiple containers in a crashloop. Getting: _ _ _ _ / | (_)____/ /_ __ __/ /____ / /| | / / ___/ __ \/ / / / __/ _ \ / _ |/ / / / /_/ / /_/ / /_/ __/ /_/ |_/_/_/ /_._…

Shopify connectors having problems [Troubleshooting] connectors Is this your first time deploying Airbyte?: No OS Version / Instance: Ubuntu running on GCP Compute Instance Memory / Disk: Mem - 16GB / Storage - 100GB Deployment: Docker deployment Airbyte Version: 0.39.39 Alpha Source name/version: Shopify - 0.1.37 Destination name/version: BigQuery - 1.1.11 Step: The issue is happening during the retrieval of data from the source, (I presume) Description: C…

Airbyte workers registration on task queues [Q&A] Hello everyone! I was seeking some guidance regarding the airbyte-workers task registration. Based on this I can see that there are provisions to have multiple task queues for SYNC jobs, whereas, for the other jobs, they have only one task queue (static). I wanted to understand what is the reason to have multiple task queues for data sync in particular. Secondly, only the queue name for data syn…

Octavia Apply Disable automatic sync [Troubleshooting] octavia-cli Is this your first time deploying Airbyte?: No / OS Version / Instance: Ubuntu Memory / Disk: 4GB Deployment: Kubernetes Airbyte Version: 0.40.6 Source name/version: Postgres Destination name/version: 1.0.7 Step: During octavia apply Description: Sometimes when running octavia apply if there is a change the connection is automatically triggered and starts a sync. Is it possible to disable thi…

State is saved for 1 stream (out of 3) only (after Airbyte upgrade) [Connector Development] connectors Pretty much as titled. I had this connector developed, which was working fine for more than a month now. After upgrading Airbyte, however, it started saving state for 1 out of 3 streams - for others it does full resync everytime. The state message I see in the UI is this: However, using python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.js…

Docker Service Completely Frozen for Days [Troubleshooting] data-loading, getting-started, connectors Is this your first time deploying Airbyte: No OS Version / Instance: EC2 t2.large Memory / Disk: 16Gb Deployment: Docker Airbyte Version: 0.40.7 Description: A few days ago our Postgres connector started to fail, we were running 0.4.40 and so we upgraded to 1.0.10 and subsequently upgraded from Airbyte 0.40.0-alpha to 0.40.7. After these two upgrades one of our Postgres runs simply says running …

Datetime field saved as dictionary in S3 parquet [Troubleshooting] data-loading, connectors Is this your first time deploying Airbyte?: Yes OS Version / Instance: Amazon Linux - EC2 t2.large Memory / Disk: 8Gb / 30Gb Deployment: Docker Airbyte Version: 0.40.6 Source name/version: MySQL - 0.6.11 Destination name/version: S3 - 0.3.15 Step: Issue happens during sync Description: While synching a MySQL database to S3 Parquet, timestamp fields are beying converted to a dictionary structure: …

Correctly handling private key file as a secret [Connector Development] The API I’m working with today requires that I sign a JWT with a private key that I’ve generated locally. I’ve got a script that loads it and can sign locally for testing in Postman, but is there a better way to do it in Airbyte? For one, I couldn’t get the key into the config.json file correctly as it doesn’t handle being shoved into a single line very well and while I’m pretty sure that there’…

Security vulnerability with temporal and airbyte-worker images [Q&A] kubernetes Hello, Our security team detected the “CVE-2022-27664” vulnerability with temporal and airbyte-worker container images. These two images use golang 1.15.8(tctl, sql-tool, and temporal-server packages), 1.10(dockerize package), and 1.17.2(airbyte-worker). They are asking us to upgrade the golang version to 1.18.6. Could you please suggest how to upgrade these images? Currently, I deployed airbyte…

This is how micronaut detects match or not on @Require. It seems this is the issue which result in failure of multiple

JobErrorReportingClient

instances.

Hubspot to SQL Server replication failed (after 18 hours) [Troubleshooting] getting-started, data-loading Is this your first time deploying Airbyte?: Yes OS Version / Instance: Windows 10 Pro (Azure VM), Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz 2.59 GHz Memory / Disk: 16GB RAM, 130GB Deployment: Docker (Docker Desktop 4.12.0 (85629)) Airbyte Version: 0.40.4 Source name/version: Hubspot 0.1.83 Destination name/version: SQL Server 0.1.2 StepSync (first time) Description: The sync fails aft…