Hi team! Not sure if this is the right place to ask this.... If I wanted to sync data from Instagram Ads, would I use the Facebook Marketing source connector? Or would I need to create a new connector to do that? I am new to using Facebook's APIs and am finding them mildly confusing 👀 💦

Just wondering is it possible to commit an airbyte configuration (all the connections) to version control?

This message was deleted.

hi guys, is there any authentication on the WEB UI ? Internal or any auth config to integrate with external OAuth2 / Others ? And with auth, some sort of RBAC, with roles like Admin / Viewer / User would be great. Anything in the roadmap regarding this?

For the Facebook Marketing API, how do I know if I have a lot of data? Put another way, how could I spot if I have too much data for a given time period and thus calibrate my time period to suit the size of my data?

Is there any way to set up an airbyte instance on Google Compute Engine entirely using terraform? I’m new to terraform and I’ve been using airbyte for a bit but I wasn’t sure if there’s a way to automate some of the steps in this setup guide: https://docs.airbyte.io/deploying-airbyte/on-gcp-compute-engine

Hey guys, is there any way to create a destination database in postgres while setting up connectors?

Hello, is there a best practice for full refresh vs. incremental? I am using Harvest connector -> postgres full sync works, but incremental doesn't seem to do anything. Should I be running a full sync then switching to incremental?

So we have this very old reservation system sitting in SQL Server, and we want to extract this data into JSON documents / events. However, one of our JSON models gets data from 130 tables, which are all joined together with various keys and shapes. If we set up a CDC change listener on SQL Server, then whenever a table value changes on one of the 130 tables, we need to be able to pull together a complete new JSON record, containing data from these 130 tables, and therefore we need to JOIN all this data using something like dbt or airbyte SQL syntax. For this to work, tho, you need to keep the entire state of the SQL server available in a store local to Airbyte, so a JOIN can always be performed. Is this use case relevant to Airbyte? Am I barking up the wrong tree here?

Hey, is there a way to create connections using code instead of GUI? I have tons of CSV files to be synced to Postgres DB daily. Thanks!

Hello, First of all I want to thank the Airbyte team to create a wonderful easy to use tool. I have been tinkering with Airbyte for a while and finally decided to implement it for my small data stack. My use case is simple and straightforward, I have a MySQL database in production which I want to replicate to a PostgreSQL instance in the local network. I had no issues (just had to make sure I read the documentation correctly) setting up Airbyte, setting source and destination. Though I'm facing a problem with the sync itself, I'm trying to sync 10 odd tables, only 2 of the tables have a lot of records in them (about 3 million in one and 300k in other), total database size should be around 2GB. I have tried 9-10 times so far but sync fails at the Normalization step for the table with maximum records. I suspect that's happening maybe because of the number of records or size of the table, Is there any work around for this or something obvious that I'm missing? Here's the log snippet of the error

normalization - 2021-11-18 23:08:42 INFO () LineGobbler(voidCall):82 - Database Error in model recipient (models/generated/airbyte_tables/cms_production/recipient.sql)
normalization - 2021-11-18 23:08:42 INFO () LineGobbler(voidCall):82 -   server closed the connection unexpectedly
normalization - 2021-11-18 23:08:42 INFO () LineGobbler(voidCall):82 -   	This probably means the server terminated abnormally
normalization - 2021-11-18 23:08:42 INFO () LineGobbler(voidCall):82 -   	before or while processing the request.

Hi all, new to Airbyte. I been trying to sync a large table (30GB, 300M records) from Snowflake in to Google BiqQuery but kept failing due to several issues such as memory issues, part limits, partition limits etc. After several attempts, I gave up trying several things and wasting cloud resources. I would like to know how can I load data from a large table in batches ? For example, extract and load 1-10000000 records and so on. Do I have to do it as part of custom transformation in DBT ? I have a feeling this is a very common issue and wondering how others are tackling.

hey I am trying to connect google-sheet as source. I am getting

The connection test failed

with the following logs:

Is there a way to version control the tables you've brought in from a source? For example if someone on my team wants to add in a new table I'd like for it to have to be approved first via some git versioning

Hey Everyone, I have tried using airbyte and I am loving it so far. Can someone point me to the API documentation? I am not able to find the docs on this link : https://docs.airbyte.io/api-documentation

can i upgrade to the latest airbyte version without the latest incremental normalization changes introduced in

0.30.24-alpha

? or am i able to disable it (cc’ing @Chris (deprecated profile))

hello, has anyone been successful in sending Jira data to Redshift using the airbyte? can you help me?

Hey, recently started using airbyte and was a little concerned regarding the Usage Data collection before deploying it to Production. Is there any documentation I can refer to which details what data is being gathered ?

Is the AirByte Cloud offering SOC2 compliant?

Hi All, Bit of a stupid question I’m sure - we’re looking to move data in both directions (from our mySQL server into our tools and also from our tool into this server), Airbyte looks like it would only work in one of these directions? Is this correct?

Hey guys, I would like to sync my data from different data sources to PostgreSQL database: • How to do my data structure flattens? Now I got 3 columns (Airbyte AB ID, Airbyte Data, Airbyte Emitted At). I would like to have a columns from Airbyte Data as well • How to have a name for tables without "airbyte" prefix? Now I got: "airbyte tmp uru amplitude active users", "airbyte tmp rgr amplitude average session lengths", ... Looking forward

Hi, I wonder if someone could advise , Just followed the documentation to do DBT normalisation with some small customisations. Pretty much exactly as laid out in the docs, However when I try to use the normalisation, loading from git hub, The extract fails. Logs --- io.airbyte.workers.WorkerException: Dbt Transformation Failed. at io.airbyte.workers.DbtTransformationWorker.run(DbtTransformationWorker.java:57) ~[io.airbyte-airbyte-workers-0.30.25-alpha.jar:?] at io.airbyte.workers.DbtTransformationWorker.run(DbtTransformationWorker.java:16) ~[io.airbyte-airbyte-workers-0.30.25-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$1(TemporalAttemptExecution.java:145) ~[io.airbyte-airbyte-workers-0.30.25-alpha.jar:?] at java.lang.Thread.run(Thread.java:832) [?:?] Caused by: io.airbyte.workers.WorkerException: DBT Transformation Failed. at io.airbyte.workers.DbtTransformationWorker.run(DbtTransformationWorker.java:54) ~[io.airbyte-airbyte-workers-0.30.25-alpha.jar:?] ... 3 more Suppressed: io.airbyte.workers.WorkerException: Normalization process wasn’t successful

All, can I get some links to integrate source(say mysql) to destination(say file system) programmaticly only to airbyte and NOT through UI?

Hi everyone! It looks like I dont have access to "Custom Transforms" in Airbyte cloud, is that right? or am I doing something wrong on my side?

In airflow, i write my dags in python code, and in dbt i write my data transformations in code. is there a way to define connections in airbyte as code? so that we can version control, integrate into ci/cd pipelines, etc

I just start to had a look into airbyte, looks really nice. I still have some question: • Is it GDPR compliant / if I host it myself? • Does the tool integrate some real-time data feature?

Hi Airbyte team, is there a way to sync from salesforce without creating an app there? Just using credentials of a user.

What's the best way to change one variable in .env file? Clone the Airbyte GitHub repo, change the variable and update Airbyte instance from a cloned repo? And do it again when the new version is rolled out?

Hi Airbyte team. How do I go about getting permission to create a remote branch off master? I have an issue open here. I am about to open a PR for it but realised when pushing to remote that I don't have permissions to do that. I couldn't find anything relevant in the docs around this

Hello! How's everyone? How do I change the batch number default so it can be bigger?