Hello everyone ! I would like to know how airbyte could scale to a high number of connection (~200). My team is using mainly GCP. So we are looking for either Compute Engine or GKE. Is one compute engine VM is enough to handle 200 connections with daily sync ?

Hello everyone, Iam a beginner, Iam following the airbyte doc quickstart section, I tried to deploy Airbyte using Docker its asking for authentication what can i do? This is when i cloned and ran "docker_compose up" command... I dont even get to see the Airbyte Banner yet. Guide me through next steps please . FYI Iam running this on windows, Thanks.

Hello, we're currently in the process of setting up the Netsuite connector in our airbyte. But we have some questions on access level. https://discuss.airbyte.io/t/why-does-netsuite-connector-role-require-full-access/2946 Specifically around why it needs

FULL

access the biggest concern is will airbyte be doing writes into our netsuite data or even possibly deleting?

Using the latest helm deployment on k8s GKE. When deploying the

worker

pod is CrashLoopBackoff, any idea what can be causing this? The logs are showing a Micronaut error and then shutting down

[m i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: null

with stacktrace

java.lang.IllegalArgumentException: null at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?] at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:36) ~[io.airbyte.airbyte-config-config-models-0.40.15.jar:?] at io.airbyte.config.storage.MinioS3ClientFactory.validate(MinioS3ClientFactory.java:33) ~[io.airbyte.airbyte-config-config-models-0.40.15.jar:?] at io.airbyte.config.storage.MinioS3ClientFactory.<init>(MinioS3ClientFactory.java:27) ~[io.airbyte.airbyte-config-config-models-0.40.15.jar:?] at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:48) ~[io.airbyte.airbyte-config-config-models-0.40.15.jar:?] at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:164) ~[io.airbyte.airbyte-config-config-models-0.40.15.jar:?] at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:151) ~[io.airbyte.airbyte-config-config-models-0.40.15.jar:?] at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:163) ~[io.airbyte-airbyte-workers-0.40.15.jar:?] at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:136) ~[io.airbyte-airbyte-workers-0.40.15.jar:?] at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:59) ~[io.airbyte-airbyte-workers-0.40.15.jar:?] at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262) ~[micronaut-inject-3.7.1.jar:3.7.1] at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60) ~[micronaut-inject-3.7.1.jar:3.7.1] at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229) ~[micronaut-inject-3.7.1.jar:3.7.1] at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:587) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1] at java.util.Optional.ifPresent(Optional.java:178) ~[?:?] at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:581) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1] at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:298) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1] at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104) ~[micronaut-http-server-netty-3.7.1.jar:3.7.1] at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81) ~[micronaut-context-3.7.1.jar:3.7.1] at java.util.Optional.ifPresent(Optional.java:178) ~[?:?] at io.micronaut.runtime.Micronaut.start(Micronaut.java:79) ~[micronaut-context-3.7.1.jar:3.7.1] at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.7.1.jar:3.7.1] at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.7.1.jar:3.7.1] at io.airbyte.workers.Application.main(Application.java:12) ~[io.airbyte-airbyte-workers-0.40.15.jar:?]

Update: Fixed by setting these in values.yaml https://github.com/airbytehq/airbyte/issues/17390#issuecomment-1274716117

Hello. When deploying Airbyte via Helm chart, is there a way to configure the root path? For example, if I have the webapp service deployed at my-url/airbyte, and I need it to look for resources with paths like my-url/airbyte/index.css rather than my-url/index.css, how do I configure that?

I have been trying to send some data from S3 to Postgres which is in a Jsonl format. I am having a lot of JSON parsing errors specifically "JSON parse error: Column() changed from object to array in row 0" . Is there any option that I can restructure my data so that I can transfer it seamlessly . Please suggest me any steps to achieve the same.

Is there a setting to include data types of attributes for the ingested source json data? I receive the data as “colName”:”value” but there is no data type info with it. My source is Zuora and destination is Snowflake. Thanks.

Hi everyone! Can you help me about the airbyte connector? I'm facing right now an issue using a pipedrive connctor.

Hi all! Has anyone faced an issue where Airbyte OS continually loads when you connect to the instance? It started doing this a few days ago and I'm unable to access the navigation to the connections or destinations for an instance that has been running for over a month.

Hi folks, any tip to improve performance for sync big tables in mysql ? Tables around 300GB to 1TB of data.

Installing Airbyte locally, are my choices really "Give non-anonymized data" or "Give anonymized data"? Sour taste at startup, not much of a choice. If that's not what you mean, please update the copy on that form.

For first setup, to make sure everything works, I chose PokeAPI and SQLite I created lol.db , and the connector was set up, and synced, and reports success. There is no data in the lol.db file. What do?

Seeing that the SQLite instructions are "No Setup Guide found for this connector." I'm left with little guidance

Actually, no matter what I put in I get "All connection tests passed!"

While I'm calling things out that don't work, the slack invite link on https://airbyte.com/connector-contest is https://join.slack.com/share/enQtNDEzMjUzNTQ5OTg0Ny0xN2I2MDU4NGFkOTNlYzZlN2JmMGIzOTgyMjU1ODk1MDIzZWM5YTAzMThkMDFkNzgzNWNjNzljOGU1MmRhMDIw and does not work.

I tried to build the specified image version, but an error was reported`VERSION=0.40.6 ./gradlew airbyte-server:build -x test`

Hi Folks, I have a private docker repo and am trying to make new connectors that pull from images from there. I am receiving a

Internal Server Error: Get Spec job failed.

error and I think it has to do with airbyte not able to pull images from the private repo. Any tips here? I am using airbyte open source on an EC2 and was looking at this for context but am not sure how to implement: https://docs.airbyte.com/operator-guides/using-custom-connectors/#on-dockerhub---repositories

Hi, Setting up the Environment on Google Cloud (Cloud VM + SSH access). Following the documentation to deploy airbyte on gcp, i installed docker,docker-compose and airbyte on vm. How can i access webapp of airbyte without creating ssh tunnel?

Now that the databricks connector also supports Azure I decided to give her a spin, with mixed results. The cluster Airbyte is going to use is not on 24/7, and this causes timeouts in various parts of the sync (either at the start or at the end). Is there anything I can do to prevent this? Increase the time Airbyte waits for the cluster to come online (in Azure this takes 5min easily, sometimes 10)

Hey guys! Just cloned Airbyte from GitHub, did

docker-compose up

, and went to

localhost:8000

. Instead of seeing some user set up I only see alert window for signing in. I have not setup any user yet so obviously I cannot proceed. Any ideas why this happens?

Hi guys, I am using Airbyte to import data from Google Search Console. However, I can get approximately 15 months historical data. I tried to set the start date in the source but it does not help. Any ideas? For example, even though I set the start date as

2019-01-01

in the source setup (below), the earliest date I can get is around June last year

Hi, I notice that on

env

file of version

v0.40.15

, the version variable is still

v0.40.14

Will it be able to pull image of version

v0.40.15

? https://github.com/airbytehq/airbyte/blob/v0.40.15/.env#L13

Hi, I am trying to use the search source api to search for sources by name. But even after trying all the things I always get blank sources for my query. Is there anything wrong I am doing here. Request

{

  "workspaceId": "3057872d-237f-4f05-85eb-d1a7a5431157",
  "name": "Test-google-ad",
}

Hello team, Can anyone tell me what's the sourceCatalogId? How to get one while creating a connection through the Airbyte Open API?

Hi team. I am trying to move from datacoral to airbyte. Any ideas how to do that without breaking anything like not have airbyte in the name of the tables?

Hi team, I am using airbyte to create a datapipeline from to trasfer data from Postgres to Clickhouse. At that time, one table sync is getting failure due to dbt normalization. It is throwing the error like, id column is having duplicate values, due to that, dbt couldn't able to create the incremental model. But checked in the source DB, there is no duplicates in that particular column. Also that column is auto incremented column. Requesting airbyte team and others to fix this issue. Can anybody have suggestions and inputs, please share? @Andy Yeo (Airbyte) @Davin Chia (Airbyte) @Davin Chia (Airbyte) @Andy Jih (Airbyte)

Hi I am trying to sync parquet files from S3 to Postgres but for the multi level objects, There is no data visible after the sync, It is showing NULL values for all the multi level object columns.

Hey team, I’m posting here just to check if I’m doing the setup correctly. I have a connection *Postgres <> Redshift, 1 table, 100k record*s. The extraction took 11 minutes on the first run, the subsequent ran took 16 minutes (0 records). It seems Airbyte is having a hard time during the DBT phase. Do you have any performance tips? Deployment: k8s Version: 0.40.15 Worker(ram/cpu limits): 2Gi/2500m Worker replicas: 3 Using S3 as staging for redshift COPY

im replicating postgres -> snowflake, all on the latest versions. 1 databases has much longer normalization seteps than others its replicating only a few thousand records every hour across like 50 tables, but normalization is taking like 20 minutes. i see that the snowflake warehouse has much higher load during the normalization step. my question is what are the main factors to consider for how long normalization will take? Since i have databases with more records/data/tables replicating hourly with noticeable faster normalization and lower snowflake warehouse load? (the syncs are spread out over hour as to not overlap)

hello, I'm back. We've upgraded airbyte to version “0.40.18" from "0.35." and now a lot of jobs get stuck for over 30 mins. Last log entry is something like.

2022-10-20 13:18:21 INFO i.a.c.i.LineGobbler(voidCall):114 - 
2022-10-20 13:18:21 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK -----
2022-10-20 13:18:21 INFO i.a.c.i.LineGobbler(voidCall):114 -

and then nothing for over half an hour, but the sync is still running. we're using kubernetes, deploying with helm. what can we do to improve performance? increase job resource limits?