Hello all, I recently stumbled upon Airbyte and was exploring it to use it for a big data application. As per the documentation, data between the Airbyte source and destination containers is moved through unix pipes (STDOUT --> STDIN). My question is around this implementation: • How reliable is this approach of moving data through unix pipes? • Do we have any documented benchmarks or performance/throughput stats of Airbyte? • What is the maximum limit to the data being moved from source to destination? How does Airbyte perform if the data is very large (in GBs lets say)? Any help would be greatly appreciated! 🙂

For those of you who run Airbyte together with dbt: how do you run dbt? on dbt cloud or on e.g. GCP Cloud Run, or it's own GCE instance, or same instance as Airbyte?

Hi Team, I want to use Airbyte in one of my projects. I was referring the API document ( https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#auth ). I have two questions for now. 1) From where I can get API key? 2) Does Airbyte provide Hosted services so that I do not have to deploy my own Aitbyte instance on my own server? Thank you & Best Regards

Hi Team, How can I scale my self hosted Airbyte instance?

Hello everyone! 🙋‍♂️ I'm a developer and currently developing a data pipeline that uses Airbyte as the main source. In our platform, we have a multi-tenant scenario, where people belong to different workspaces (like in airbyte), and have a

workspace_uuid

associated to them. When we create a new connection using, for example, Hubspot as source and PostgreSQL as destination, we would like the stream to return airbyte's

workspace_uuid

, in order to know, later on in Postgres, where these values come from (all workspaces have a different

api_key

to connect to Hubspot and therefore their data is different). Could anyone help me find the best solution to the multi-tenant problem? I can't insert airbyte's

workspace_uuid

in the output stream because Hubspot's connector is already built, and I am not comfortable with creating one table for each workspace in Postgres. I don't know if there are any other solutions. Thanks in advance and sorry for the long post! 😄

Once we had stored data at our own postgres-db using pg as destination, how can we take advantage of tables and __airbyte__ab_id_ field in order to get _airbyte__data information? Our next step is to have available this data at the client as a data table, but I'm hesitant on how I must to solve some obstacles. How could we relate created-tables at postgres to created-connections? Thank You

Hi all.. Looking to get some info here as we are exploring using Airbyte to be part of our data framework, and trying to see if it can solve multiple problems.. Would Airbyte support a use case where a list of integrations are provided, embedded in a product, and users are allowed to configure their credentials and trigger an execution of said integrations published integrations? For example.. I have an HR cloud product, our customers need to integrate their systems to ours and vice versa. We'd create an area in our product called integrations, where our users could select a logo for whatever app they want to connect with us, they would provide their auth credentials, and then provide the execution schedule for the integration

Hi, anyone integrated open source airbyte with Okta/SSO Or enabled LDAP user authentication ? I am looking make connectors user specific only ?

hello guys, can I uninstall connectors which I do not want to use now? There are so many of them make me hard to find what I want. So can I uninstall them now and install them when I want?

when applying for Google Ads API access, do you upload a design spec? if so, which one?

Hi Airbyte Team, we are using argo CD to manage Airbyte in k8s, After upgrading Airbyte to a newer version by changing the docker images, the Web UI would still show the older version. (My guess is this information is stored in the DB) Is there an easy way to let the web app reflect the newer version?

Hi guys. I have followed the install instructions however I don’t get a banner and I get errors when doing the docker-compose up. Looks like a pg error. Saying a 172. IP can’t access the db.

How to load data from Mongo DB to postgres in normalized form?

Hi guys, i’m interested in any anonymization plugins/discussions

Tried to setup the Facebook Marketing API connector today with a developer token that I created last Thursday. Getting this error:

The connection tests failed.

"FacebookAPIException('Error: 2635, (#2635) You are calling a deprecated version of the Ads API. Please update to the latest version: v12.0.')"

Do you believe this is related to my token and can I then upgrade it in my FB Developer Account, or does the issue reside on the connector side?

Hello. how can I connect to

scheduler store

(internal postgres) for accessing the JOBS table?

Has anyone had any experience working with AWS STS as an alternative means of AWS authentication to access tokens for Airbyte connectors? We're looking to use the Snowflake connector with S3 staging but restrictions in place mean we can't create explicit IAM users and therefore permanent credentials.

Hi! Can we set the initial loads from the user(like we need to intially load the last 14 days source data)? If no, could we achieve this with any other alternative solution?

Hello, Successfully setup Airbyte on kubernetes but how can I provide the setup to someone as I dont find some authentication mechanism. The plan for now is setting up ingress and making it accessible from the vpn only. How are people setting this up?

Is there any way to gracefull end a manual run which will write the data to the

_airbyte_raw_*

tables? Or can i just copy the data from the tmp table to the raw table?

Hello Everyone. I have a question. Is there a way to dynamically set start_date when retrieving calls from Zendesk? How do we set pagination?

Hi All, I am getting stared on airbyte planning to deploy on GCP GKE. Unable to fund any recourses related to this can someone help on this

Is there anyway to set the refresh frequency based on date and time? Can we set the initial sync on customised date& time and has to trigger only once?

Hello all! We are exploring Airbyte for our data integration use cases and a question came up. In our existing ingest, we have the need to exclude some columns from a database source (Postgres). As an example, we would like to ingest user records but exclude PII in the table. Is that possible to do with existing connectors or would we need to create a custom one?

(access to airbyte itself)

hi, i have setup a docker-compose installation of airbyte and hooked to port up to a domain. but now everyone can access the instance without having to authenticate. how can i enable email+pw auth?

hey, I am trying to make a ETL pipeline from postgres to postgres, I want to use log based replication from the slave of my master db, currently postgres slave DBs do not support replication slots, is thereany wat I can do this?

👋 Hello, team! I am a beginner and just finished reading documentation from the website including that of apache air. Are there any open datasets or tasks I can practice to grow my skills or are there any open tasks with data available that a beginner can try to grow their skills?

Hi! 👋 How can I send a notification to slack channel when a sync succed or fail? It's possible?

Hi, i want to know how airbyte scheduling works, especially for 24 hour schedule. was it run from 24 hours after first run or following like aftermidnight in vm's timezone? (edit) if it was triggered 24h after first run, could we change it to more spesific time?