i'm currently using airbyte(version 39.1) for ingesting table from postgrsql(Version 9.5) to clickhouse( 22.4.4.7).on airbyte replication page when i try to search stream name in search bar nothing is there and also try to defined sourc name,sync mode,destination name is not add on that option .how can i solve this problem and which is the airbyte latest version?

Is it possible to create a custom Source/Connection on the cloud version of AirByte? Or will I have to switch to self-hosted to accomplish that?

Hello Everyone ! I have deployed Airbyte in kubernetes where most of the components are deployed as "Deployments". Now if the Airbyte deployment gets deleted it deletes the persistent volumes also which deletes all the connections and other states of connections as well. To avoid this situation, what does Airbyte recommend please ?

Hi ,I'm currently using airbyte (version 39.1)for ingesting table from postgresql (version 13) to click house(22.4.4.7). Step 1-adding source Step2-adding destination Step3-try to setup connection source (postresql)to destination(clickhouse) on airbyte replication page so source and destination connector option is not visible and also try to search stream name in search bar nothing is there and also try to defined source name , sync mode, destination name is not add on that option .how can I solve this problem and which is airbyte latest version ?below attached step wise images and log file .kindly Help Me.

Hello, I am using stripe as source connector but sync attempts fails everytime with error like

io.airbyte.workers.DefaultReplicationWorker$SourceException: Source cannot be stopped!
------------------------------------------------------------------

[0m > 09:19:17.259653 [error] [MainThread]:   404 Not found: Table xxxx-realm-122345:stripe_data._airbyte_raw_disputes was not found in location US

Only some data is synced

Hi All, I can't find any documentation to setup a new custom connector from a private docker repository. There is no authentication prompt during its setup. Can someone please guide. Thank you for help in advance !

Hi, I'm currently using Fivetran to extract Google Analytics data, and would like to use Airbyte instead. I only replicate custom reports with: • dimensions • metrics • filter expression I've managed to create custom reports on Airbyte 0.36.4-alpha with dimensions and metrics but not sure how to use Filter Expression in the json field. Is this currently supported? Thanks

Hi everyone. I'm currently evaluating airbyte and comparing it to pipelinewise. I use airbyte

0.38.4-alpha

on k8s and trying to sync one table from PG to Snowflake. Table has 32 mln records. It takes airbyte anywhere from 2h30m to 3h30 min to do initial sync on this table. Pipelinewise takes 37min. I'm not sure how to get the same numbers. Another thing that confuses me that after sync is done I see that both tables in snowflake have 32 mln records. But the size of the table created by pipelinewise is 2.6GB and the one created by airbyte is 5GB (and on top of that why does airbye UI shows that 49.25 GB worth of data were processed - those numbers don't match). Why is that? Any ideas.

Hello! I want to contribute to airbyte adding a field to a stream from the facebook marketing connector, how can I do this?

This message was deleted.

Hey everyone, got a couple of noob questions (or one twice): I want to set up incremental + append dedup for Google Analytics and Google Ads but I'm not sure what I should be using for the primary key. For Google Analytics I've put

view_id, ga_date

so a composite one (which I hope is right, it seems to be that the

ga_date

shouldn't be replicated in a given

view_id

? But I'm not sure what to use for Google Ads. Just the

segments.date

, right? Or is that dumb for some reason I'm not seeing? Thanks for any pointers!

Hi! I'm trying to follow this tutorial for automating my data scraping (https://airbyte.com/tutorials/data-scraping-with-airflow-and-beautiful-soup). Everything is working fine until the actual implementation of the dag. First, the provided python code doesn't actually compile (after substituting the required variables in the code). However, once I get the python code to compile, Airflow doesn't actually pick up on my new dag on the webserver (after I restarted the webserver and scheduler). I've been really struggling to get the new dag on airflow. The connection on airbyte seems to be up and running fine though. Any help would be greatly appreciated!

Hey, guys! I'm needing to do an integration in airbyte using a connector using api. Does anyone have an example of how to do it or a link or that could help me?

If we were to use Airbyte as part of a multi-user environment where different users have different lists of connectors and sources, is the cloud based solution the only way to accomplish that? Or does the Open source version support siloed workspaces too?

Trying to do getting started with a local docker-compose on a M1 mac. I'm running into errors from the quickstart steps, resulting in a traceback. Threading for the full error copy-paste. Failing on file not found, docker/transport/unixconn.py in connect.

HI team. I am trying to deploy and use Airbyte based on AWS EKS. Can I grant IAM Role to the POD where the Connectors are executed?

Question about logs — on an EC2 instance, I’m seeing many directories in the

airbyte_workspace

volume

/data

. According to the docs this is where logs, configs, and I assume other things are stored. Is it safe to delete these after a period of time? I need to cleanup logs on a very large Airbyte instance (millions of directories in

/data

) and would like to remove all directories older than 2 weeks, but do not want to remove data that will not be regenerated.

Hey guys - Trying to use aribyte to replicate a WordPress MySQL database (WPEngine) to Postgres - Have been working with WPEngine support for hours and we cant get the thing to connect. Anyone available who can help?

Hi All, I am looking for anyone who has used an external db for their Airbyte deployment on K8s this is possible but airbyte has not updated there documentation to explain how

Hello!! Coming from custom python written extract. Hoping to be able to move away from that to a more manageable (1 person team) option!!

Hi Team, I am in process of building singer based connector. I am trying to run below command ./gradlew airbyte integrationsconnectorssource oracle py singerbuild and getting the below error. Plugin [id: 'com.gradle.enterprise', version: '3.4.1'] was not found in any of the following sources: - Gradle Core Plugins (plugin is not in 'org.gradle' namespace) - Plugin Repositories (could not resolve plugin artifact 'com.gradle.enterprisecom.gradle.enterprise.gradle.plugin3.4.1') Searched in the following repositories: Gradle Central Plugin Repository

Hello All , I wanted to limit number of worker pulling data from a connector to save API calls but I couldn't find where to change this config.Airbyte dashboard has an option to import config but not clear on how to upload (what format and filename) How do i do it ?

Is there anyone that know how to configure partitioning of columns of files when using parquet for S3 destination? It seems like Airbyte only can dump files given a hardcoded prefix, rather than having a partitioning strategy based on a column value of the data?

hi all, Looking for someone who can share with me at a high level how they source controlled airbyte across versions. Any help is great thanks.

Hello Airbyte community. I have question, why Airbyte deletes all the data after editing connection streams? For example if I add new stream to connection I’m getting such message:

WARNING! Updating the schema will delete all the data for this connection in your destination and start syncing from scratch

. Why is that? Why the data is deleted from all streams, not only from new one, but also from currently existing. So after this I need to collect all data from from scratch? Or how I could prevent from this?

Hello! I am trying to add the Snapchat Marketing source on Airbyte cloud and it doesn't show up in the drop down list to add it. However, the documentation says it is available for cloud. Is there something I need to do to add it to the selectable sources? Thanks in advance

Hi Team, Not sure if this is the right group, I had to post this here as I am getting error when I try to build a connection from S3 to PostgreSQL. Error says - Failed to fetch schema. Please try again Can you help me understand what is causing this error. As I tried the same with the other S3 to PostgresSQL connection and it did not had any issue

Hi everyone, We recently started using Airbyte (a great product btw). We followed this documentation to create the environment with 30GB of disk space, which we thought would be enough and never grow since we are moving data to different locations. However, we are observing storage growth over time, reaching 30GB. Does anyone know why this is happening? My guess is logs file, but 30GB of log text sounds crazy. If that's the case, does anyone here know if we can customize log settings?

With regards to https://github.com/airbytehq/airbyte/issues/2670 issue, I am having the same issue. I am not clear with solution. Can somebody help with this?

Hey all, just deployes Airbyte to GCP but I'm having problems, the main one being I can't see how to add a new/custom connector - I need to be able do custom transformations on data before final load. The docs keep saying to go to the Admin section, or add one to the normalization stage, but neither of those are present. I'm using 0.39.2-alpha. Any advice?