Hi all, not sure if this is the correct place to ask. For source I enter

File > JSONL

, it says all is succesful, but when it runs the connection it says 'Failed to fetch schema. Please try again` (and again...). When I enter the JSONL file to a random online site, it can determine the schema immediately. Anyone want to help with the issue?

Hi 👋, not sure if this is the right forum. I'm looking to use the Powered by Airbyte approach, but am thinking of going the hosted route at first. I saw this on the docs page, but it seems like it might conflict with other messages I've seen in Slack since joining today:

For Airbyte Open-Source you don't need the API Token for Authentication! All endpoints are possible to access using the API without it. Note: Airbyte Cloud does not currently support API access.

So, I suppose I have two questions. 1. Is this still the case? Only the open-source version supports API access? 2. If so, is there any sense of when API support for cloud would be added to the roadmap?

Hi - can someone give me an API response for the following endpoint?

POST /v1/source_definitions/list_latest

Hello I would like to start with Airbyte and I have some questions. I want to integrate some data sources to BigQuery, however I couldnt understand what the cost is exactly, Could I do it only with the open source plan? Thank you in advance!

is there a way to store connector configuration As code in Git or does it only live in the database?

Hello Team. I have configured VM to connect to GCP as the documentation mentioned but there's the following Error when connecting to local host. eduardo@dataline:~/airbyte$ gcloud --project=named-perigee-343716 beta compute ssh dataline -- -L 8000localhost80 00 -N -f Did you mean zone [us-central1-a] for instance: [dataline] (Y/n)? Y bind [127.0.0.1]8000 Address already in use channel_setup_fwd_listener_tcpip: cannot listen to port: 8000 Could not request local forwarding. How can I replace what's located in that terminal to successfully connect it. It is the only active instance we have. We would appreciate your support.

Is there any way to add another table to an existing database (source = Redshift) to database (destination = Snowflake) connection without reconfiguring the entire stream and wiping all the destination data?

Hey, this might be a simple question but I didn’t find a clear answer anywhere. I’d like to use Airbyte to load data on a daily basis, incrementally. Do I need to have the Airbyte syncing running all the time or if I kick off a new sync it would pick up from the latest synced record ? Considering the source is massive I wouldn’t want to scan the whole thing every time I restart Airbyte or kick off a new sync. I understand the source must be able to do

Incremental Sync

but is this “state” only kept through a single run of Airbyte ? Also, does an Incremental Sync read always scan the table?

Hello, hope there is an answer: I ran the code on my local terminal to create an ssh tunnel, however the following message is being displayed. bind (...) Address already in use channel_setup_fwd_listener_tcpip: cannot listen to port: 8000 Could not request local forwarding. How can I fix it in order to access localhost properly...

I need to use AWS DynamoDB as a source

how can I do that i don’t see any connector for that

What is the right way to add a private docker registry (google artifact registry in this case) to the airbyte environment? So I can pull my own custom connector images. (I got Airbyte running in GCP k8s using the kustomize set-up) EDIT: Nevermind, after some digging I found this part of the documentation (https://docs.airbyte.com/operator-guides/using-custom-connectors#for-kubernetes-airbyte-deployments)

hello to everyone,

I am new in Airbyte. I have deployed the airbyte through GCP and it is working. I have created also my first connection, but i cant see the option to check which tables from te source i want to transfer to the destination. Can you please advise

Hi there 🙂 I am struggling with our Hubspot data replication and I’m exploring if Airbyte can do the job here We have more than 7M records in Hubspot, do you know if using Oauth is enough to bypass the API limit? In the documentation, it’s mentioned that: “This connector supports only authentication with API Key”*,* but there is an OAuth method, will it work?

I couldn’t find any information, but maybe I didn’t look hard enough. Is it possible to do a hybrid deployement of airbyte? Like have some of the workers deployed on Ec2 for just CDC of databases and the rest on Airbyte cloud?

I’m interested in using the Postgres CDC source. I have a use case where row changes have a User ID, and I want: • records relating to user A to get synced to destination1, and • records relating to user B to get synced to destination2 I know that I can run a transformation after the load, but that’s too late. destination1 would see records for both user A and B. Is there a way to apply a filter beforehand?

Ok, having trouble since I’m very new to data engineering… I tested fivetran and got dbt set up no problem, but that's because it was just enough plug-n-play for my level of development knowledge. Could somebody reeaaally break it down for me as if i were 5 yrs old? HOW can I make my own dbt models to use with airbyte open-sourced? Where can I access the pre-built dbt project to manipulate and connect through github? I have it running on a EC2 instance with an ssh tunnel from my Mac and testing with an Instagram>BigQuery connection up and running. Please don't share the airbyte-dbt help doc, as it doesn't seem to dumb it down enough for me 😂_… or the demo hour youtube video_ 🙏

Is there a way to setup incremental sync deduped + history to recognize hard deletes?

Hi,

Hi , I am trying to connect to salesforce as source and redshift as destination. As part of transformation, is there a way to add a new field into destination table - For example, Accounts table is having Col A, Col B, Col C. I would like to add a Calculated Col D getting added to Accounts [ Col D could be a result of set of computations from external source like a python/java ] . I was exploring with DBT option, but couldn't find any good example with the same. ! Any assistance you can provide would be greatly appreciated!

Hi! There is a way to connect Airbyte to slack webhook in order to get notified if something fails or succeed?

AWS deployment question: Where can we found in Airbyte docs a relationship between EC2 instance type and number of jobs that will run in order to know when to scale the machine? Doc says:

For long-running Airbyte installations, we recommend a t2.large instance

What is the concept of long-running? That same doc says t3.medium is enough for testing purposes. Currently we have a t3.medium instance and RDS db.t3.micro. So I would like to know which setup should be used regarding EC2 and RDS in order to avoid issues due to the instance type.

Hi guys, I have attached logs for an airbyte connection between postgres to snowflake. The sync succeeded but has resulted in 0 bytes and 0 records. I am sure that the info exists inside of the schema in postgres. Any ideas why? Any help is appreciated! Perhaps cc @Harshith (Airbyte) @John (Airbyte) and anyone else on the airbyte team

Is there any user authentication yet planned for open source airbyte?

Hi, Are there any ways airbyte can notify us upon every scheduled successful sync ? Basically a notifier to invoke a configurable external API endpoint. Any assistance you can provide would be greatly appreciated!

Hi there! Is there a maximum number of connections you can define in Airbyte? We make business intelligence software for bridal stores. We need to ELT each of our clients’ data into our database for analysis. The data comes from disparate systems (accounting, point-of-sale) using client specific credentials (api/oauth tokens). The way I’m thinking of using Airbyte for this is by dynamically creating a new source and connection for each client via the Airbyte API. This would result in many sources and connections. I’d like to know if my thinking is on the right track or if there are practical limitations for this approach.

Hi everyone ! I'm new to Airbyte (Open source version) and I had a question about its workings. When credentials for a connector are entered, how does Airbyte store them internally ? Curious to know if any internal DB is used and which one ?

If I Use Airbyte Opensource with EÇ2 instance, then I only pay for EC2 compute and storage Right ? And do not pay anything for Airbyte ?

Hi There! I'm using airbyte opensource, is there any way to export some sync data from the syncs. Basically, wanted to get the num of rows, size, time etc. Thinking of moving it to DWH or BI tool to create a report.